The Difficulty of Passive Learning in Deep Reinforcement Learning

1 Introduction . Learning to act in an environment purely from observational data ( i.e . with no environment interaction ) , usually referred to as offline reinforcement learning , has great practical as well as theoretical importance ( see [ Levine et al. , 2020 ] for a recent survey ) . In real-world settings like robotics and healthcare , it is motivated by the ambition to learn from existing datasets and the high cost of environment interaction . Its theoretical appeal is that stationarity of the data distribution allows for more straightforward convergence analysis of learning algorithms . Moreover , decoupling learning from data generation alleviates one of the major difficulties in the empirical analysis of common reinforcement learning agents , allowing the targeted study of learning dynamics in isolation from their effects on behavior . Recent work has identified extrapolation error as a major challenge for offline ( deep ) reinforcement learning [ Achiam et al. , 2019 , Buckman et al. , 2021 , Fujimoto et al. , 2019b , Fakoor et al. , 2021 , Liu et al. , 2020 , Nair et al. , 2020 ] , with bootstrapping often highlighted as either the cause or an amplifier of the effect : The value of missing or under-represented state-action pairs in the dataset can be over-estimated , either transiently ( due to insufficient training or data ) or even asymptotically ( due to modelling or dataset bias ) , resulting in a potentially severely under-performing acquired policy . The corrective feedback-loop [ Kumar et al. , 2020b ] , whereby value over-estimation is self-correcting via exploitation during interaction with the environment ( while under-estimation is corrected by exploration ) , is critically missing in the offline setting . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . To mitigate this , typically one of a few related strategies are proposed : policy or learning update constraints preventing deviations from states and actions well-covered by the dataset or satisfying certain uncertainty bounds [ Fujimoto et al. , 2019a , b , Kumar et al. , 2019 , 2020c , Achiam et al. , 2019 , Wang et al. , 2020b , Wu et al. , 2021 , Nair et al. , 2020 , Wu et al. , 2019 , Yu et al. , 2020 ] , pessimism bias to battle value over-estimation [ Buckman et al. , 2021 , Kidambi et al. , 2020 ] , large and diverse datasets to improve state space coverage [ Agarwal et al. , 2020 ] , or learned models to fill in gaps with synthesized data [ Schrittwieser et al. , 2021 , Matsushima et al. , 2020 ] . While many of these enjoy theoretical justification in the tabular or linear cases [ Thomas et al. , 2015 ] , guarantees for the practically relevant non-linear case are mostly lacking . In this paper we draw inspiration from the experimental paradigm introduced in the classic Held and Hein [ 1963 ] experiment in psychology . The experiment involved coupling two young animal subjects ’ movements and visual perceptions to ensure that both receive the same stream of visual inputs , while only one can actively shape that stream by directing the pair ’ s movements ( Fig . 1 , top-left ) . By showing that , despite identical visual experiences , only the actively moving subject acquired adequate visual acuity , the experiment established the importance of active locomotion in learning vision . Analogously , we introduce the ‘ Tandem RL ’ setup , pairing an ‘ active ’ and a ‘ passive ’ agent in a training loop where only the active agent drives data generation , while both perform identical learning updates from the generated data1 . By decoupling learning dynamics from its impact on data generation , while preserving the non-stationarity of the online learning setting , this experimental paradigm promises to be a valuable analytic tool for the precise empirical study of RL algorithms . Holding architectures , losses , and crucially data distribution equal across the active and passive agents , or varying them in a controlled manner , we perform a detailed empirical analysis of the failure modes of passive ( i.e . non-interactive , offline ) learning , and pinpoint the contributing factors in properties of the data distribution , function approximation and learning algorithm . Our study confirms some past intuitions for the failure modes of offline learning , while refining and extending the findings in the deep RL case . In particular , our results indicate an empirically less critical role for bootstrapping than previously hypothesized , while foregrounding erroneous extrapolation or over-generalization by a function approximator trained on an inadequate data distribution as the crucial challenge . Among other things , our experiments draw a sharp boundary between the mostly well-behaved ( and analytically well-studied ) case of linear function approximation , and the non-linear case for which theoretical guarantees are lacking . Moreover , we delineate different , more and less effective , ways of enhancing the training data distribution to support successful offline learning , e.g . by analysing the impact of dataset size and diversity , the stochasticity of the data generating policy , or small amounts of self-generated data . Our results provide hints towards a hypothesis relevant in both offline and online RL : robust learning of control with function approximation may require interactivity not merely as a data gathering mechanism , but as a counterbalance to a ( sufficiently expressive ) approximator ’ s tendency to ‘ exploit gaps ’ in an arbitrary fixed data distribution by excessive extrapolation . 1The ‘ tandem ’ analogy is of course that of two riders , both of whom experience the same route , while only the front rider gets to decide on direction . 2 The Experimental Paradigm of Tandem Reinforcement Learning . The Tandem RL setting , extending a similar analytic setup in [ Fujimoto et al. , 2019b ] , consists of two learning agents , one of which ( the ‘ active agent ’ ) performs the usual online training loop of interacting with an environment and learning from the generated data , while the other ( the ‘ passive agent ’ ) learns solely from data generated by the active agent , while only interacting with the environment for evaluation . We distinguish two experimental paradigms ( see Fig . 1 , top-right ) : Tandem : Active and passive agents start with independently initialized networks , and train on an identical sequence of training batches in the exact same order . Forked Tandem : An agent is trained for a fraction of its total training budget . It is then ‘ forked ’ into active and passive agents , which start out with identical network weights . The active agent is ‘ frozen ’ , i.e . receives no further training , but continues to generate data from its policy . The passive agent is trained on this generated data for the remainder of the training budget . 2.1 Implementation . Our basic experimental agent is ‘ Tandem DQN ’ , an active/passive pair of Double-DQN agents2 [ van Hasselt et al. , 2016 ] . Following the usual training protocol [ Mnih et al. , 2015 ] , the total training budget is 200 iterations , each of which consists of 1M steps taken on the environment by the active agent , interspersed with regular learning updates ( on one , or concurrently on both agents , depending on the paradigm ) , on batches of transitions sampled from the active agent ’ s replay buffer . Both agents are independently evaluated on the environment for 500K steps after each training iteration . Most of our experiments are performed on the Atari domain [ Bellemare et al. , 2013 ] , using the exact algorithm and hyperparameters from [ van Hasselt et al. , 2016 ] . We use a fixed set of four representative games to demonstrate most of our empirical results , two of which ( BREAKOUT , PONG ) can be thought of as easy and largely solved by baseline agents , while the others ( SEAQUEST , SPACE INVADERS ) have non-trivial learning curves and remain challenging . Unless stated otherwise , all results show averages over at least 5 seeds , with confidence intervals indicating variation over seeds . In comparative plots , boldface entries indicate the default Tandem DQN configuration , and gray lines always correspond to the active agent ’ s performance . 2.2 The Tandem Effect . We begin by reproducing the striking observation in [ Fujimoto et al. , 2019b ] that the passive learner generally fails to adequately learn from the very data stream that is demonstrably sufficient for its architecturally identical active counterpart ; we refer to this phenomenon as the ‘ tandem effect ’ ( Fig . 1 , bottom ) . We ascertain the generality of this finding by replicating it across a broad suite of environments and agent architectures : Double-DQN on 57 Atari environments ( Appendix Figs . 10 & 11 ) , adapted agent variants on four Classic Control domains from the OpenAI Gym library [ Brockman et al. , 2016 ] and the MinAtar domain [ Young and Tian , 2019 ] ( Appendix Figs . 12 & 15 ) , and the distributed R2D2 agent [ Kapturowski et al. , 2019 ] ( Appendix Fig . 14 ) . Details on agents and environments are provided in the Appendix3 . Empirically , we make the informal observation that while active and passive Q-networks tend to produce similar values for typical state-action pairs under the active policy ( where the action is the active Q-value function ’ s argmax for a given state ) , their values are less correlated for other ( non-argmax ) actions , and in fact the active and passive greedy policies of a Tandem DQN tend to disagree in a large fraction of states under the behavior distribution ( on average > 75 % of states , after 100M steps of training , across 57 Atari games ; see Appendix Fig . 13 ) . Moreover , in a fraction ( ≈ 12/57 ) of Atari games , we observe the passive agent ’ s network to strongly over-estimate a fraction of state-action values , with the over-estimation growing as training progresses . 2Our choice of Double-DQN as a baseline is motivated by its relatively strong performance and robustness compared to vanilla DQN [ Mnih et al. , 2015 ] , paired with its simplicity compared to later variants like Rainbow [ Hessel et al. , 2018 ] , which allows for more easily controlled experiments with fewer moving parts . 3We provide two Tandem RL implementations : https : //github.com/deepmind/deepmind-research/tree/ master/tandem_dqn based on the DQN Zoo [ Quan and Ostrovski , 2020 ] , and https : //github.com/google/ dopamine/tree/master/dopamine/labs/tandem_dqn based on the Dopamine library [ Castro et al. , 2018 ] . 3 Analysis of the Tandem Effect . In line with existing explanations [ Levine et al. , 2020 ] , we propose that the tandem effect is primarily caused by extrapolation error when certain state-action pairs are under-represented in the active agent ’ s behavior data . Specifically with ε-greedy policies , even small over-estimation of the values of rarely seen actions can lead to sufficient behavior deviations to cause catastrophic under-performance . We further extend this hypothesis : in the context of deep reinforcement learning ( i.e . with non-linear function approximation ) , an inadequate data distribution can drive over-generalization [ Bengio et al. , 2020 ] , making such erroneous extrapolation likely . While the tandem effect can show up as learning inefficiency even in the tabular case [ Xiao et al. , 2021 ] , it proves especially pernicious in the case of non-linear function approximation , where erroneous extrapolation can lead to errors not just on completely unseen , but also rarely seen data , and can persist in the infinite-sample limit . Coalescing this view and past analyses of challenges in offline RL ( e.g . [ Levine et al. , 2020 , Fujimoto et al. , 2019b , Liu et al. , 2020 ] ) into the following three potential contributing factors in the tandem effect provides a natural structure to our analysis : Bootstrapping ( B ) The passive agent ’ s bootstrapping from poorly estimated ( in particular , overestimated ) values causes any initially small mis-estimation to get amplified . Data Distribution ( D ) Insufficient coverage of sub-optimal actions under the active agent ’ s policy may lead to their mis-estimation by the passive agent . In the case of over-estimation , this may lead to the passive agent ’ s under-performance . Function Approximation ( F ) A non-linear function approximator used as a Q-value function may tend to wrongly extrapolate the values of state-action pairs underrepresented in the active agent ’ s behavior distribution . This tendency can be inherent and persistent , in the sense of being independent of initialization and not being reduced with increased training on the same data distribution . These proposed contributing factors are not at all mutually exclusive ; they may interact in causing or exacerbating the tandem effect . Insufficient coverage of sub-optimal actions under the active agent ’ s behavior distribution ( D ) may lead to insufficient constraint on the respective values , which allows for effects of erroneous extrapolation by a function approximator ( F ) . Where this results in over-estimation , the use of bootstrapping ( B ) carries the potential to ‘ pollute ’ even well-covered state-action pairs by propagating over-estimated values ( especially via the max operator in the case of Q-learning ) . In the next sections we empirically study these three factors in isolation , to establish their actual roles and relative contributions to the overall difficulty of passive learning .

Summary: This paper presents an empirical study on offline deep reinforcement learning in atari environments. The authors use tandem learning -- where an agent B learns from experience collected from another agent A. The authors consider 3 hypotheses as to why agent B tends to perform worse in this setting, and provide many ablation-type experiments to support/reject these hypotheses.