The Difficulty of Passive Learning in Deep Reinforcement Learning

1 Introduction . Learning to act in an environment purely from observational data ( i.e . with no environment interaction ) , usually referred to as offline reinforcement learning , has great practical as well as theoretical importance ( see [ Levine et al. , 2020 ] for a recent survey ) . In real-world settings like robotics and healthcare , it is motivated by the ambition to learn from existing datasets and the high cost of environment interaction . Its theoretical appeal is that stationarity of the data distribution allows for more straightforward convergence analysis of learning algorithms . Moreover , decoupling learning from data generation alleviates one of the major difficulties in the empirical analysis of common reinforcement learning agents , allowing the targeted study of learning dynamics in isolation from their effects on behavior . Recent work has identified extrapolation error as a major challenge for offline ( deep ) reinforcement learning [ Achiam et al. , 2019 , Buckman et al. , 2021 , Fujimoto et al. , 2019b , Fakoor et al. , 2021 , Liu et al. , 2020 , Nair et al. , 2020 ] , with bootstrapping often highlighted as either the cause or an amplifier of the effect : The value of missing or under-represented state-action pairs in the dataset can be over-estimated , either transiently ( due to insufficient training or data ) or even asymptotically ( due to modelling or dataset bias ) , resulting in a potentially severely under-performing acquired policy . The corrective feedback-loop [ Kumar et al. , 2020b ] , whereby value over-estimation is self-correcting via exploitation during interaction with the environment ( while under-estimation is corrected by exploration ) , is critically missing in the offline setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To mitigate this , typically one of a few related strategies are proposed : policy or learning update constraints preventing deviations from states and actions well-covered by the dataset or satisfying certain uncertainty bounds [ Fujimoto et al. , 2019a , b , Kumar et al. , 2019 , 2020c , Achiam et al. , 2019 , Wang et al. , 2020b , Wu et al. , 2021 , Nair et al. , 2020 , Wu et al. , 2019 , Yu et al. , 2020 ] , pessimism bias to battle value over-estimation [ Buckman et al. , 2021 , Kidambi et al. , 2020 ] , large and diverse datasets to improve state space coverage [ Agarwal et al. , 2020 ] , or learned models to fill in gaps with synthesized data [ Schrittwieser et al. , 2021 , Matsushima et al. , 2020 ] . While many of these enjoy theoretical justification in the tabular or linear cases [ Thomas et al. , 2015 ] , guarantees for the practically relevant non-linear case are mostly lacking . In this paper we draw inspiration from the experimental paradigm introduced in the classic Held and Hein [ 1963 ] experiment in psychology . The experiment involved coupling two young animal subjects ’ movements and visual perceptions to ensure that both receive the same stream of visual inputs , while only one can actively shape that stream by directing the pair ’ s movements ( Fig . 1 , top-left ) . By showing that , despite identical visual experiences , only the actively moving subject acquired adequate visual acuity , the experiment established the importance of active locomotion in learning vision . Analogously , we introduce the ‘ Tandem RL ’ setup , pairing an ‘ active ’ and a ‘ passive ’ agent in a training loop where only the active agent drives data generation , while both perform identical learning updates from the generated data1 . By decoupling learning dynamics from its impact on data generation , while preserving the non-stationarity of the online learning setting , this experimental paradigm promises to be a valuable analytic tool for the precise empirical study of RL algorithms . Holding architectures , losses , and crucially data distribution equal across the active and passive agents , or varying them in a controlled manner , we perform a detailed empirical analysis of the failure modes of passive ( i.e . non-interactive , offline ) learning , and pinpoint the contributing factors in properties of the data distribution , function approximation and learning algorithm . Our study confirms some past intuitions for the failure modes of offline learning , while refining and extending the findings in the deep RL case . In particular , our results indicate an empirically less critical role for bootstrapping than previously hypothesized , while foregrounding erroneous extrapolation or over-generalization by a function approximator trained on an inadequate data distribution as the crucial challenge . Among other things , our experiments draw a sharp boundary between the mostly well-behaved ( and analytically well-studied ) case of linear function approximation , and the non-linear case for which theoretical guarantees are lacking . Moreover , we delineate different , more and less effective , ways of enhancing the training data distribution to support successful offline learning , e.g . by analysing the impact of dataset size and diversity , the stochasticity of the data generating policy , or small amounts of self-generated data . Our results provide hints towards a hypothesis relevant in both offline and online RL : robust learning of control with function approximation may require interactivity not merely as a data gathering mechanism , but as a counterbalance to a ( sufficiently expressive ) approximator ’ s tendency to ‘ exploit gaps ’ in an arbitrary fixed data distribution by excessive extrapolation . 1The ‘ tandem ’ analogy is of course that of two riders , both of whom experience the same route , while only the front rider gets to decide on direction . 2 The Experimental Paradigm of Tandem Reinforcement Learning . The Tandem RL setting , extending a similar analytic setup in [ Fujimoto et al. , 2019b ] , consists of two learning agents , one of which ( the ‘ active agent ’ ) performs the usual online training loop of interacting with an environment and learning from the generated data , while the other ( the ‘ passive agent ’ ) learns solely from data generated by the active agent , while only interacting with the environment for evaluation . We distinguish two experimental paradigms ( see Fig . 1 , top-right ) : Tandem : Active and passive agents start with independently initialized networks , and train on an identical sequence of training batches in the exact same order . Forked Tandem : An agent is trained for a fraction of its total training budget . It is then ‘ forked ’ into active and passive agents , which start out with identical network weights . The active agent is ‘ frozen ’ , i.e . receives no further training , but continues to generate data from its policy . The passive agent is trained on this generated data for the remainder of the training budget . 2.1 Implementation . Our basic experimental agent is ‘ Tandem DQN ’ , an active/passive pair of Double-DQN agents2 [ van Hasselt et al. , 2016 ] . Following the usual training protocol [ Mnih et al. , 2015 ] , the total training budget is 200 iterations , each of which consists of 1M steps taken on the environment by the active agent , interspersed with regular learning updates ( on one , or concurrently on both agents , depending on the paradigm ) , on batches of transitions sampled from the active agent ’ s replay buffer . Both agents are independently evaluated on the environment for 500K steps after each training iteration . Most of our experiments are performed on the Atari domain [ Bellemare et al. , 2013 ] , using the exact algorithm and hyperparameters from [ van Hasselt et al. , 2016 ] . We use a fixed set of four representative games to demonstrate most of our empirical results , two of which ( BREAKOUT , PONG ) can be thought of as easy and largely solved by baseline agents , while the others ( SEAQUEST , SPACE INVADERS ) have non-trivial learning curves and remain challenging . Unless stated otherwise , all results show averages over at least 5 seeds , with confidence intervals indicating variation over seeds . In comparative plots , boldface entries indicate the default Tandem DQN configuration , and gray lines always correspond to the active agent ’ s performance . 2.2 The Tandem Effect . We begin by reproducing the striking observation in [ Fujimoto et al. , 2019b ] that the passive learner generally fails to adequately learn from the very data stream that is demonstrably sufficient for its architecturally identical active counterpart ; we refer to this phenomenon as the ‘ tandem effect ’ ( Fig . 1 , bottom ) . We ascertain the generality of this finding by replicating it across a broad suite of environments and agent architectures : Double-DQN on 57 Atari environments ( Appendix Figs . 10 & 11 ) , adapted agent variants on four Classic Control domains from the OpenAI Gym library [ Brockman et al. , 2016 ] and the MinAtar domain [ Young and Tian , 2019 ] ( Appendix Figs . 12 & 15 ) , and the distributed R2D2 agent [ Kapturowski et al. , 2019 ] ( Appendix Fig . 14 ) . Details on agents and environments are provided in the Appendix3 . Empirically , we make the informal observation that while active and passive Q-networks tend to produce similar values for typical state-action pairs under the active policy ( where the action is the active Q-value function ’ s argmax for a given state ) , their values are less correlated for other ( non-argmax ) actions , and in fact the active and passive greedy policies of a Tandem DQN tend to disagree in a large fraction of states under the behavior distribution ( on average > 75 % of states , after 100M steps of training , across 57 Atari games ; see Appendix Fig . 13 ) . Moreover , in a fraction ( ≈ 12/57 ) of Atari games , we observe the passive agent ’ s network to strongly over-estimate a fraction of state-action values , with the over-estimation growing as training progresses . 2Our choice of Double-DQN as a baseline is motivated by its relatively strong performance and robustness compared to vanilla DQN [ Mnih et al. , 2015 ] , paired with its simplicity compared to later variants like Rainbow [ Hessel et al. , 2018 ] , which allows for more easily controlled experiments with fewer moving parts . 3We provide two Tandem RL implementations : https : //github.com/deepmind/deepmind-research/tree/ master/tandem_dqn based on the DQN Zoo [ Quan and Ostrovski , 2020 ] , and https : //github.com/google/ dopamine/tree/master/dopamine/labs/tandem_dqn based on the Dopamine library [ Castro et al. , 2018 ] . 3 Analysis of the Tandem Effect . In line with existing explanations [ Levine et al. , 2020 ] , we propose that the tandem effect is primarily caused by extrapolation error when certain state-action pairs are under-represented in the active agent ’ s behavior data . Specifically with ε-greedy policies , even small over-estimation of the values of rarely seen actions can lead to sufficient behavior deviations to cause catastrophic under-performance . We further extend this hypothesis : in the context of deep reinforcement learning ( i.e . with non-linear function approximation ) , an inadequate data distribution can drive over-generalization [ Bengio et al. , 2020 ] , making such erroneous extrapolation likely . While the tandem effect can show up as learning inefficiency even in the tabular case [ Xiao et al. , 2021 ] , it proves especially pernicious in the case of non-linear function approximation , where erroneous extrapolation can lead to errors not just on completely unseen , but also rarely seen data , and can persist in the infinite-sample limit . Coalescing this view and past analyses of challenges in offline RL ( e.g . [ Levine et al. , 2020 , Fujimoto et al. , 2019b , Liu et al. , 2020 ] ) into the following three potential contributing factors in the tandem effect provides a natural structure to our analysis : Bootstrapping ( B ) The passive agent ’ s bootstrapping from poorly estimated ( in particular , overestimated ) values causes any initially small mis-estimation to get amplified . Data Distribution ( D ) Insufficient coverage of sub-optimal actions under the active agent ’ s policy may lead to their mis-estimation by the passive agent . In the case of over-estimation , this may lead to the passive agent ’ s under-performance . Function Approximation ( F ) A non-linear function approximator used as a Q-value function may tend to wrongly extrapolate the values of state-action pairs underrepresented in the active agent ’ s behavior distribution . This tendency can be inherent and persistent , in the sense of being independent of initialization and not being reduced with increased training on the same data distribution . These proposed contributing factors are not at all mutually exclusive ; they may interact in causing or exacerbating the tandem effect . Insufficient coverage of sub-optimal actions under the active agent ’ s behavior distribution ( D ) may lead to insufficient constraint on the respective values , which allows for effects of erroneous extrapolation by a function approximator ( F ) . Where this results in over-estimation , the use of bootstrapping ( B ) carries the potential to ‘ pollute ’ even well-covered state-action pairs by propagating over-estimated values ( especially via the max operator in the case of Q-learning ) . In the next sections we empirically study these three factors in isolation , to establish their actual roles and relative contributions to the overall difficulty of passive learning .

Summary: The paper addresses the difficulties that standard RL methods have in the offline setting. It presents a new analysis technique designed to better understand the causes of the difficulties. The technique, called Tandem RL setting, and forked tandem, extend the setup, introduced by Fujimoto, Meger, and Precup in the BCQ-paper "Off-Policy Deep Reinforcement Learning without Exploration" and enable systematic studies of causes for some kind of problems with offline RL. The technique is used for extensive studies, in several benchmarks, and cautious conclusions are drawn.