ConQUR: Mitigating Delusional Bias in Deep Q-Learning

1 INTRODUCTION . Q-learning ( Watkins & Dayan , 1992 ; Sutton & Barto , 2018 ) lies at the heart of many of the recent successes of deep reinforcement learning ( RL ) ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , with recent advancements ( e.g. , van Hasselt ( 2010 ) ; Bellemare et al . ( 2017 ) ; Wang et al . ( 2016 ) ; Hessel et al . ( 2017 ) ) helping to make it among the most widely used methods in applied RL . Despite these successes , many properties of Q-learning are poorly understood , and it is challenging to successfully apply deep Q-learning in practice . When combined with function approximation , Q-learning can become unstable ( Baird , 1995 ; Boyan & Moore , 1995 ; Tsitsiklis & Roy , 1996 ; Sutton & Barto , 2018 ) . Various modifications have been proposed to improve convergence or approximation error ( Gordon , 1995 ; 1999 ; Szepesvári & Smart , 2004 ; Melo & Ribeiro , 2007 ; Maei et al. , 2010 ; Munos et al. , 2016 ) ; but it remains difficult to reliably attain both robustness and scalability . Recently , Lu et al . ( 2018 ) identified a source of error in Q-learning with function approximation known as delusional bias . It arises because Q-learning updates the value of state-action pairs using estimates of ( sampled ) successor-state values that can be mutually inconsistent given the policy class induced by the approximator . This can result in unbounded approximation error , divergence , policy cycling , and other undesirable behavior . To handle delusion , the authors propose a policy-consistent backup operator that maintains multiple Q-value estimates organized into information sets . Each information set has its own backed-up Q-values and corresponding “ policy commitments ” responsible for inducing these values . Systematic management of these sets ensures that only consistent choices of maximizing actions are used to update Q-values . All potential solutions are tracked to prevent premature convergence on any specific policy commitments . Unfortunately , the proposed algorithms use tabular representations of Q-functions , so while this establishes foundations for delusional bias , the function approximator is used neither for generalization nor to manage the size of the state/action space . Consequently , this approach is not scalable to RL problems of practical size . In this work , we propose CONQUR ( CONsistent Q-Update Regression ) , a general framework for integrating policy-consistent backups with regression-based function approximation for Q-learning and for managing the search through the space of possible regressors ( i.e. , information sets ) . With suitable search heuristics , our framework provides a computationally effective means for minimizing the effects of delusional bias in Q-learning , while admitting scaling to practical problems . Our main contributions are as follows . First we define novel augmentations of standard Q-regression to increase the degree of policy consistency across training batches . While testing exact consistency is expensive , we introduce an efficient soft-consistency penalty that promotes consistency of new labels with earlier policy commitments . Second , drawing on the information-set structure of Lu et al . ( 2018 ) , we define a search space over Q-regressors to allow consideration of multiple sets of policy commitments . Third , we introduce heuristics for guiding the search over regressors , which is critical given the combinatorial nature of information sets . Finally , we provide experimental results on the Atari suite ( Bellemare et al. , 2013 ) demonstrating that CONQUR can offer ( sometimes dramatic ) improvements over Q-learning . We also show that ( easy-to-implement ) consistency penalization on its own ( i.e. , without search ) can improve over both standard and double Q-learning . 2 BACKGROUND . We assume a discounted , infinite horizon Markov decision process ( MDP ) , M = ( S , A , P , p0 , R , γ ) . The state space S can reflect both discrete and continuous features , but we take the action space A to be finite ( and practically enumerable ) . We consider Q-learning with a function approximator Qθ to learn an ( approximately ) optimal Q-function ( Watkins , 1989 ; Sutton & Barto , 2018 ) , drawn from some approximation class parameterized by Θ ( e.g. , the weights of a neural network ) . When the approximator is a deep network , we generically refer to the algorithm as DQN , the method at the heart of many recent RL successes ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . For online Q-learning , at a transition s , a , r , s′ , the Q-update is given by : θ ← θ + α ( r + γmax a′∈A Qθ ( s ′ , a′ ) −Qθ ( s , a ) ) ∇θQθ ( s , a ) . ( 1 ) Batch versions of Q-learning , including DQN , are similar , but fit a regressor repeatedly to batches of training examples ( Ernst et al. , 2005 ; Riedmiller , 2005 ) . Batch methods are usually more data efficient and stable than online Q-learning . Abstractly , batch Q-learning works through a sequence of ( possibly randomized ) data batches D1 , · · ·DT to produce a sequence of regressors Qθ1 , . . . , QθT = Qθ , estimating the Q-function.1 For each ( s , a , r , s′ ) ∈ Dk , we use a prior estimator Qθk−1 to bootstrap the Q-label q = r + γmaxa′ Qθk−1 ( s ′ , a′ ) . We then fit Qθk to this training data using a suitable regression procedure with an appropriate loss function . Once trained , the ( implicit ) induced policy πθ is the greedy policy w.r.t . Qθ , i.e. , πθ ( s ) = arg maxa∈AQθ ( s , a ) . Let F ( Θ ) , resp . G ( Θ ) , be the corresponding class of expressible Q-functions , resp . greedy policies . Intuitively , delusional bias occurs whenever a backed-up value estimate is derived from action choices that are not ( jointly ) realizable in G ( Θ ) ( Lu et al. , 2018 ) . Standard Q-updates back up values for each ( s , a ) pair by independently choosing maximizing actions at the corresponding next states s′ . However , such updates may be “ inconsistent ” under approximation : if no policy in G ( Θ ) can jointly express all past action choices , backed up values may not be realizable by any expressible policy . Lu et al . ( 2018 ) show that delusion can manifest itself with several undesirable consequences . Most critically , it can prevent Q-learning from learning the optimal representable policy in G ( Θ ) ; it can also cause divergence . To address this , they propose a non-delusional policy consistent Q-learning ( PCQL ) algorithm that provably eliminates delusion . We refer to the original paper for details , but review the main concepts we need to consider below . The first key concept is that of policy consistency . For any S ⊆ S , an action assignment σS : S → A associates an action σ ( s ) with each s ∈ S. We say σ is policy consistent if there is a greedy policy π ∈ G ( Θ ) s.t . π ( s ) = σ ( s ) for all s ∈ S. We sometimes equate a set SA of state-action pairs with an implied assignment π ( s ) = a for all ( s , a ) ∈ SA . If SA contains multiple pairs with the same state s , but different actions a , it is a multi-assignment ( though we loosely use the term “ assignment ” in both cases when there is no risk of confusion ) . In ( batch ) Q-learning , each successive regressor uses training labels generated by assuming maximizing actions ( under the prior regressor ) are taken at its successor states . Let σk reflect the collection of states and corresponding maximizing actions taken to generate labels for regressor Qθk ( assume it is policy consistent ) . Suppose we train Qθk by bootstrapping on Qθk−1 and consider a training sample ( s , a , r , s′ ) . Q-learning generates label r + γmaxa′ Qθk−1 ( s ′ , a′ ) for input ( s , a ) . Notice , 1We describe our approach using a straightforward form of batch Q-learning , but it can accommodate many variants , e.g. , where the regressor used for bootstrapping is some earlier Q-estimator , or the estimators generating the max-actions and the value estimates are different as in double Q-learning ( van Hasselt , 2010 ; Hasselt et al. , 2016 ) ; indeed , we experiment with such variants . however , that taking action a∗ = argmaxa′ Qθk ( s ′ , a′ ) at s′ may not be policy consistent with σk . Thus Q-learning will estimate a value for ( s , a ) assuming the execution of a policy that can not be realized given the limitations of the approximator . The PCQL algorithm ( Lu et al. , 2018 ) prevents this by insisting that any action assignment σ used to generate bootstrapped labels is consistent with earlier assignments . Notice that this means Q-labels will often not be generated using maximizing actions relative to the prior regressor . The second key concept is that of information sets . One will generally not be able to use maximizing actions to generate labels , so tradeoffs can be made when deciding which actions to assign to different states . Indeed , even if it is feasible to assign a maximizing action a to state s early in training , say at batch k , since it may prevent assigning a maximizing a′ to s′ later , say batch k + ` , we may want to consider a different assignment to s to give more flexibility to maximize at other states later . PCQL doesn ’ t try to anticipate the tradeoffs—rather it maintains multiple information sets , each corresponding to a different assignment to the states seen in the training data so far . Each gives rise to a different Q-function estimate , resulting in multiple hypotheses . At the end of training , the best hypothesis is the one maximizing expected value w.r.t . an initial state distribution . PCQL provides strong convergence guarantees , but it is a tabular algorithm : the function approximator retricts the policy class , but is not used to generalize Q-values . Furthermore , its theoretical guarantees come at a cost : it uses exact policy consistency tests—tractable for linear approximators , but not practical for large problems ; and it maintains all consistent assignments . As a result , PCQL can not be used for large RL problems of the type tackled by DQN . 3 THE CONQUR FRAMEWORK . We develop the CONQUR framework to provide a practical approach to reducing delusion in Qlearning , specifically addressing the limitations of PCQL identified above . CONQUR consists of three main components : a practical soft-constraint penalty that promotes policy consistency ; a search space to structure the search over multiple regressors ( information sets , action assignments ) ; and heuristic search schemes ( expansion , scoring ) to find good Q-regressors . 3.1 PRELIMINARIES . We assume a set of training data consisting of quadruples ( s , a , r , s′ ) , divided into ( possibly nondisjoint ) batches D1 , . . . DT for training . This perspective is quite general : online RL corresponds to |Di| = 1 ; off-line batch training ( with sufficiently exploratory data ) corresponds to a single batch ( i.e. , T = 1 ) ; and online or batch methods with replay are realized when the Di are generated by sampling some data source with replacement . For any data batch D , let χ ( D ) = { s′ : ( s , a , r , s′ ) ∈ D } denote the collection of successor states of D. An action assignment σD for D is an assignment ( or multi-assignment ) from χ ( D ) to A : this dictates which action σD ( s′ ) is considered “ maximum ” for the purpose of generating a Q-label for pair ( s , a ) ; i.e. , ( s , a ) will be assigned training label r + γQ ( s′ , σ ( s′ ) ) rather than r + γmaxa′∈AQ ( s ′ , a′ ) . The set of all such assignments is Σ ( D ) = Aχ ( D ) ; note that it grows exponentially with |D| . Given Q-function parameterization Θ , we say σD is Θ-consistent ( w.r.t . D ) if there is some θ ∈ Θ s.t . πθ ( s′ ) = σ ( s′ ) for all s′ ∈ χ ( D ) .2 This is simple policy consistency , but with notation that emphasizes the policy class . Let ΣΘ ( D ) denote the set of all Θ-consistent assignments over D. The union σ1 ∪ σ2 of two assignments ( over D1 , D2 , resp . ) is defined in the usual way .

Summary: This paper presents a solution to tackling the problem of delusional bias in Deep Q-learning, building upon Lu et.al. (NeuRIPS 2018).  Delusional bias arises because independently choosing maximizing actions at a state may be inconsistent as the backed-up values may not be realizable by any policy. They encourage non-delusional Q-functions by adding a penalty term that enforces that the max_a in Q-learning chooses actions that do not give rise to actions outside the realizable policy class. Further, in order to keep track of all consistent assignments, they pose a search problem and propose heuristics to approximately perform this search. The heuristics are based on sampling using exponentiated Q-values and scoring possible children using scores like Bellman error, and returns of the greedy policy. Their final algorithm is evaluated on a DQN and DDQN, where they observe some improvement from both components (consistency penalty and approximate search).