ConQUR: Mitigating Delusional Bias in Deep Q-Learning

1 INTRODUCTION . Q-learning ( Watkins & Dayan , 1992 ; Sutton & Barto , 2018 ) lies at the heart of many of the recent successes of deep reinforcement learning ( RL ) ( Mnih et al. , 2015 ; Silver et al. , 2016 ) , with recent advancements ( e.g. , van Hasselt ( 2010 ) ; Bellemare et al . ( 2017 ) ; Wang et al . ( 2016 ) ; Hessel et al . ( 2017 ) ) helping to make it among the most widely used methods in applied RL . Despite these successes , many properties of Q-learning are poorly understood , and it is challenging to successfully apply deep Q-learning in practice . When combined with function approximation , Q-learning can become unstable ( Baird , 1995 ; Boyan & Moore , 1995 ; Tsitsiklis & Roy , 1996 ; Sutton & Barto , 2018 ) . Various modifications have been proposed to improve convergence or approximation error ( Gordon , 1995 ; 1999 ; Szepesvári & Smart , 2004 ; Melo & Ribeiro , 2007 ; Maei et al. , 2010 ; Munos et al. , 2016 ) ; but it remains difficult to reliably attain both robustness and scalability . Recently , Lu et al . ( 2018 ) identified a source of error in Q-learning with function approximation known as delusional bias . It arises because Q-learning updates the value of state-action pairs using estimates of ( sampled ) successor-state values that can be mutually inconsistent given the policy class induced by the approximator . This can result in unbounded approximation error , divergence , policy cycling , and other undesirable behavior . To handle delusion , the authors propose a policy-consistent backup operator that maintains multiple Q-value estimates organized into information sets . Each information set has its own backed-up Q-values and corresponding “ policy commitments ” responsible for inducing these values . Systematic management of these sets ensures that only consistent choices of maximizing actions are used to update Q-values . All potential solutions are tracked to prevent premature convergence on any specific policy commitments . Unfortunately , the proposed algorithms use tabular representations of Q-functions , so while this establishes foundations for delusional bias , the function approximator is used neither for generalization nor to manage the size of the state/action space . Consequently , this approach is not scalable to RL problems of practical size . In this work , we propose CONQUR ( CONsistent Q-Update Regression ) , a general framework for integrating policy-consistent backups with regression-based function approximation for Q-learning and for managing the search through the space of possible regressors ( i.e. , information sets ) . With suitable search heuristics , our framework provides a computationally effective means for minimizing the effects of delusional bias in Q-learning , while admitting scaling to practical problems . Our main contributions are as follows . First we define novel augmentations of standard Q-regression to increase the degree of policy consistency across training batches . While testing exact consistency is expensive , we introduce an efficient soft-consistency penalty that promotes consistency of new labels with earlier policy commitments . Second , drawing on the information-set structure of Lu et al . ( 2018 ) , we define a search space over Q-regressors to allow consideration of multiple sets of policy commitments . Third , we introduce heuristics for guiding the search over regressors , which is critical given the combinatorial nature of information sets . Finally , we provide experimental results on the Atari suite ( Bellemare et al. , 2013 ) demonstrating that CONQUR can offer ( sometimes dramatic ) improvements over Q-learning . We also show that ( easy-to-implement ) consistency penalization on its own ( i.e. , without search ) can improve over both standard and double Q-learning . 2 BACKGROUND . We assume a discounted , infinite horizon Markov decision process ( MDP ) , M = ( S , A , P , p0 , R , γ ) . The state space S can reflect both discrete and continuous features , but we take the action space A to be finite ( and practically enumerable ) . We consider Q-learning with a function approximator Qθ to learn an ( approximately ) optimal Q-function ( Watkins , 1989 ; Sutton & Barto , 2018 ) , drawn from some approximation class parameterized by Θ ( e.g. , the weights of a neural network ) . When the approximator is a deep network , we generically refer to the algorithm as DQN , the method at the heart of many recent RL successes ( Mnih et al. , 2015 ; Silver et al. , 2016 ) . For online Q-learning , at a transition s , a , r , s′ , the Q-update is given by : θ ← θ + α ( r + γmax a′∈A Qθ ( s ′ , a′ ) −Qθ ( s , a ) ) ∇θQθ ( s , a ) . ( 1 ) Batch versions of Q-learning , including DQN , are similar , but fit a regressor repeatedly to batches of training examples ( Ernst et al. , 2005 ; Riedmiller , 2005 ) . Batch methods are usually more data efficient and stable than online Q-learning . Abstractly , batch Q-learning works through a sequence of ( possibly randomized ) data batches D1 , · · ·DT to produce a sequence of regressors Qθ1 , . . . , QθT = Qθ , estimating the Q-function.1 For each ( s , a , r , s′ ) ∈ Dk , we use a prior estimator Qθk−1 to bootstrap the Q-label q = r + γmaxa′ Qθk−1 ( s ′ , a′ ) . We then fit Qθk to this training data using a suitable regression procedure with an appropriate loss function . Once trained , the ( implicit ) induced policy πθ is the greedy policy w.r.t . Qθ , i.e. , πθ ( s ) = arg maxa∈AQθ ( s , a ) . Let F ( Θ ) , resp . G ( Θ ) , be the corresponding class of expressible Q-functions , resp . greedy policies . Intuitively , delusional bias occurs whenever a backed-up value estimate is derived from action choices that are not ( jointly ) realizable in G ( Θ ) ( Lu et al. , 2018 ) . Standard Q-updates back up values for each ( s , a ) pair by independently choosing maximizing actions at the corresponding next states s′ . However , such updates may be “ inconsistent ” under approximation : if no policy in G ( Θ ) can jointly express all past action choices , backed up values may not be realizable by any expressible policy . Lu et al . ( 2018 ) show that delusion can manifest itself with several undesirable consequences . Most critically , it can prevent Q-learning from learning the optimal representable policy in G ( Θ ) ; it can also cause divergence . To address this , they propose a non-delusional policy consistent Q-learning ( PCQL ) algorithm that provably eliminates delusion . We refer to the original paper for details , but review the main concepts we need to consider below . The first key concept is that of policy consistency . For any S ⊆ S , an action assignment σS : S → A associates an action σ ( s ) with each s ∈ S. We say σ is policy consistent if there is a greedy policy π ∈ G ( Θ ) s.t . π ( s ) = σ ( s ) for all s ∈ S. We sometimes equate a set SA of state-action pairs with an implied assignment π ( s ) = a for all ( s , a ) ∈ SA . If SA contains multiple pairs with the same state s , but different actions a , it is a multi-assignment ( though we loosely use the term “ assignment ” in both cases when there is no risk of confusion ) . In ( batch ) Q-learning , each successive regressor uses training labels generated by assuming maximizing actions ( under the prior regressor ) are taken at its successor states . Let σk reflect the collection of states and corresponding maximizing actions taken to generate labels for regressor Qθk ( assume it is policy consistent ) . Suppose we train Qθk by bootstrapping on Qθk−1 and consider a training sample ( s , a , r , s′ ) . Q-learning generates label r + γmaxa′ Qθk−1 ( s ′ , a′ ) for input ( s , a ) . Notice , 1We describe our approach using a straightforward form of batch Q-learning , but it can accommodate many variants , e.g. , where the regressor used for bootstrapping is some earlier Q-estimator , or the estimators generating the max-actions and the value estimates are different as in double Q-learning ( van Hasselt , 2010 ; Hasselt et al. , 2016 ) ; indeed , we experiment with such variants . however , that taking action a∗ = argmaxa′ Qθk ( s ′ , a′ ) at s′ may not be policy consistent with σk . Thus Q-learning will estimate a value for ( s , a ) assuming the execution of a policy that can not be realized given the limitations of the approximator . The PCQL algorithm ( Lu et al. , 2018 ) prevents this by insisting that any action assignment σ used to generate bootstrapped labels is consistent with earlier assignments . Notice that this means Q-labels will often not be generated using maximizing actions relative to the prior regressor . The second key concept is that of information sets . One will generally not be able to use maximizing actions to generate labels , so tradeoffs can be made when deciding which actions to assign to different states . Indeed , even if it is feasible to assign a maximizing action a to state s early in training , say at batch k , since it may prevent assigning a maximizing a′ to s′ later , say batch k + ` , we may want to consider a different assignment to s to give more flexibility to maximize at other states later . PCQL doesn ’ t try to anticipate the tradeoffs—rather it maintains multiple information sets , each corresponding to a different assignment to the states seen in the training data so far . Each gives rise to a different Q-function estimate , resulting in multiple hypotheses . At the end of training , the best hypothesis is the one maximizing expected value w.r.t . an initial state distribution . PCQL provides strong convergence guarantees , but it is a tabular algorithm : the function approximator retricts the policy class , but is not used to generalize Q-values . Furthermore , its theoretical guarantees come at a cost : it uses exact policy consistency tests—tractable for linear approximators , but not practical for large problems ; and it maintains all consistent assignments . As a result , PCQL can not be used for large RL problems of the type tackled by DQN . 3 THE CONQUR FRAMEWORK . We develop the CONQUR framework to provide a practical approach to reducing delusion in Qlearning , specifically addressing the limitations of PCQL identified above . CONQUR consists of three main components : a practical soft-constraint penalty that promotes policy consistency ; a search space to structure the search over multiple regressors ( information sets , action assignments ) ; and heuristic search schemes ( expansion , scoring ) to find good Q-regressors . 3.1 PRELIMINARIES . We assume a set of training data consisting of quadruples ( s , a , r , s′ ) , divided into ( possibly nondisjoint ) batches D1 , . . . DT for training . This perspective is quite general : online RL corresponds to |Di| = 1 ; off-line batch training ( with sufficiently exploratory data ) corresponds to a single batch ( i.e. , T = 1 ) ; and online or batch methods with replay are realized when the Di are generated by sampling some data source with replacement . For any data batch D , let χ ( D ) = { s′ : ( s , a , r , s′ ) ∈ D } denote the collection of successor states of D. An action assignment σD for D is an assignment ( or multi-assignment ) from χ ( D ) to A : this dictates which action σD ( s′ ) is considered “ maximum ” for the purpose of generating a Q-label for pair ( s , a ) ; i.e. , ( s , a ) will be assigned training label r + γQ ( s′ , σ ( s′ ) ) rather than r + γmaxa′∈AQ ( s ′ , a′ ) . The set of all such assignments is Σ ( D ) = Aχ ( D ) ; note that it grows exponentially with |D| . Given Q-function parameterization Θ , we say σD is Θ-consistent ( w.r.t . D ) if there is some θ ∈ Θ s.t . πθ ( s′ ) = σ ( s′ ) for all s′ ∈ χ ( D ) .2 This is simple policy consistency , but with notation that emphasizes the policy class . Let ΣΘ ( D ) denote the set of all Θ-consistent assignments over D. The union σ1 ∪ σ2 of two assignments ( over D1 , D2 , resp . ) is defined in the usual way .

Summary: A recent paper by Lu et al introduced delusional bias in Q-learning, an error due to the max in the Bellman backup not being consistent with the policy representation implied by the greedy operator applied to the approximated value function. That work proposed a consistent algorithm for small and finite state spaces, which essentially enumerates over realizable policies. This paper proposes an algorithm for overcoming delusional bias in large state spaces. The idea is to add to the Q-learning objective a smooth penalty term that induces approximate consistency, and search over possible Q-function approximators. Several heuristic methods are proposed for this search, and results are demonstrated in Atari domains.