The Difficulty of Passive Learning in Deep Reinforcement Learning
Authors: Georg Ostrovski
Pablo Samuel Castro
Will Dabney
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep reinforcement learning
deep RL
offline reinforcement learning
offline RL
DQN
reinforcement learning
off-policy
policy improvement
policy optimization
distributional shift
empirical analysis
empirical method

Abstract: Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for a need to more clearly understand the challenges in this setting.

In the vein of Held & Hein's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation in conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light on phenomena observed in the online case of learning control.

The Difficulty of Passive Learning in Deep Reinforcement Learning
Georg Ostrovski DeepMind ostrovskii@deepmind.d com
Pablo Samuel Castro Google Research, Brain Team pscâgoogle.com com
Will Dabney
DeepMind wdabney@deepmiindco com
Abstract
Learning to act from observational data without active environmental interaction is a well-known challenge in Reinforcement Learning (RL). Recent approaches involve constraints on the learned policy or conservative updates, preventing strong deviations from the state-action distribution of the dataset. Although these methods are evaluated using non-linear function approximation, theoretical justifications are mostly limited to the tabular or linear cases. Given the impressive results of deep reinforcement learning, we argue for need more clearly understand the challenges in this setting. In the vein of Held & Hein's 's classic 1963 experiment, we propose the "tandem learning" experimental paradigm which facilitates our empirical analysis of the difficulties in offline reinforcement learning. We identify function approximation conjunction with fixed data distributions as the strongest factors, thereby extending but also challenging hypotheses stated in past work. Our results provide relevant insights for offline deep reinforcement learning, while also shedding new light phenomena observed in the online case of learning control.
Introduction
Learning to act in environment purely from observational data (i.e. with environment interac- tion), usually referred to as offline reinforcement learning, has great practical as well as theoretical importance (see [Levine al., 2020] for recent survey). In real-world settings like robotics and healthcare, is motivated by the ambition to learn from existing datasets and the high cost en- vironment interaction. Its theoretical appeal is that stationarity of the data distribution allows for more straightforward convergence analysis of learning algorithms. Moreover, decoupling learning from data generation alleviates of the major difficulties in the empirical analysis of common reinforcement learning agents, allowing the targeted study of learning dynamics in isolation from their effects on behavior.
Recent work has identified extrapolation error as a major challenge for offline (deep) reinforcement learning [Achiam et al., 2019, Buckman et al., 2021, Fujimoto et al., 2019b, Fakoor et al., 2021, Liu et al., 2020, Nair et al., 2020], with bootstrapping often highlighted as either the cause or an amplifier of the effect: The value of missing under-represented state- action pairs in the dataset can be over-estimated, either transiently (due to insufficient training or data) or even asymptotically (due to modelling dataset bias), resulting in a potentially severely under- performing acquired policy. The corrective feedback-loop [Kumar et al., 2020b), whereby value over-estimation is self celf-cooreeting via exploitation during interaction with the environment (while under-estimation is corrected by exploration), is critically missing in the offline setting.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Tandem
Forked Tandem
Environment Observation
Action
Active
Actve
Passive
Breaknut
Seaquest
Space invaders
15030
3000
310
10
2000
190
-10
pasoles
actins
5020
1000
250
Etwicrrmunt frames (millorss)
Encrrnmmnt Frames (rillinn)
Etwiormmnn frmms Im (ntliors)
Envronmment frames (rilliema)
(top-left) Held and Hein [1963] experiment setup. (top-right) Iilustrations of the Tandem and Forked Tandem experiment setups. (hottom) Tandem (active and passive) performance on selected Atari domains. In all figures, active agent performance is shown in gray.
Figure 1:
To mitigate this, typically one of a few related strategies are proposed: policy or learning update constraints preventing deviations from states and actions well-covered by the dataset satisfying certain uncertainty bounds [Fujimoto et al., 20,9a,b, Kumar et al.. 2019, 2020c, Achiam et al., 2019, Wang et al., 2020b, Wu et al., 2021, Nair et al., 2020, Wu et al., 2019, Yu al., 2020], pessimism bias to battle value over-estimation [Buckman et al., 2021, Kidambi et al., 2020]. large and diverse datasets to improve state space coverage [Agarwal al., 2020], or learned models to fill in gaps with synthesized data [Schrittwieser et al., 2021, Matsushima et al., 2020]. While many of these enjoy theoretical justification in the tabular or linear cases [Thomas al., 2015], guarantees for the practically relevant non-linear case are mostly lacking.
In this paper we draw inspiration from the experimental paradigm introduced in the classic Held and Hein [1963] experiment in psychology. The experiment involved coupling two young animal subjects' movements and visual perceptions to ensure that both receive the same stream of visual inputs, while only one can actively shape that stream by directing the pair's ' movements (Fig. 1, top-left). By showing that, despite identical visual experiences, only the actively moving subject acquired adequate visual acuity, the experiment established the importance of active locomotion in learning vision. Analogously, we introduce the 'Tandem RL' setup, pairing an 'active' and 'passive' agent in training loop where only the active agent drives data generation, while both perform identical learning updates from the generated data By decoupling learning dynamics from its impact on data generation, while preserving the non-stationarity of the online learning setting, this experimental paradigm promises to be a valuable analytic tool for the precise empirical study of RL algorithms.
Holding architectures, losses, and crucially data distribution equal across the active and passive agents, or varying them in a controlled manner, we perform a detailed empirical analysis of the failure modes of passive (i.e. non-interactive, offline) learning. and pinpoint the contributing factors in properties of the data distribution, function approximation and learning algorithm. Our study confirms some past intuitions for the failure modes of offline learning, while refining and extending the findings in the deep RL case. particular, our results indicate an empirically less critical role for bootstrapping than previously hypothesized, while foregrounding erroneous extrapolation or sear-ennrraliztionn by a function approximator trained on an inadequate data distribution as the crucial challenge. Among other things, our experiments draw a sharp boundary between the mostly well-behaved (and analytically well-studied) case of linear function approximation, and the non-linear case for which theoretical guarantecs are lacking. Morcover, enhancing the training data distribution to support successful offline learning, c.g. by analysing the impact of dataset size and diversity, the stochasticity of the data generating policy, or small amounts of self-generated data. Our results provide hints towards a hypothesis relevant in both offline and online RL: robust learning of control with function approximation may require interactivity not merely as a data gathering mechanism, but as a counterbalance to a (sufficienlly expressive) approximator's tendency to 'exploit gaps' in arbitrary fixed data distribution by excessive extrapolation.
delincate different, more and less effective, ways of
'The 'tandem' analogy is of course that of two riders, both of whom experience the same route, while only the front rider gets to decide direction.

---

2 The Experimental Paradigm of Tandem Reinforcement Learning
The Tandem RL setting, extending a similar analytic setup in [Fujimoto et al., 2019b], consists of two learning agents, one of which (the 'active agent") performs the usual online training loop of interacting with an environment and learning from the generated data, while the other (the 'passive agent') learns solely from data generated by the active agent, while only interacting with the environment for evaluation. We distinguish two experimental paradigms (see Fig. 1, top-right):
Tandem: Active and passive agents start with independently initialized networks, and train an identical sequence of training batches in the exact same order.
An agent is trained for fraction of its total training budget. It is then 'forked' into active and passive agents, which start out with identical network weights. The active agent is 'frozen', iee. receives no further training, but continues to generate data from its policy. The passive agent is trained on this generated data for the remainder of the training budget.
Forked Tandem:
2.1 Implementation
Our basic experimental agent is 'Tandem DQN', an active/passive pair of Double-Do DQQ agents² [van Hasselt et al., 2016]. Following the usual training protocol [Mnih et al., 2015], the total training budget is 200 iterations, each of which consists of IM steps taken on the environment by the active agent, interspersed with regular learning updates (on one, or concurrently
both agents, depending on the paradigm), on hatches of transitions sampled from the active agent's replay buffer. Both agents independently evaluated the environment for 500K steps after each training iteration.
Most of our experiments are performed on the Atari domain [Bellemare et al., 2013], using the exact algorithm and hyperparameters from [van Hasselt et al., 2016]. We use fixed set of four representative games to demonstrate most of our empirical results, two of which (BREAKOUT, PONG) can be thought of easy and largely solved by baseline agents, while the others (SEAQUEST, SPACE INVADERS) have non-trivial learning curves and remain challenging. Unless stated otherwise, all results show averages over least 5 seeds, with confidence intervals indicating variation seeds. In comparative plots, boldface entries indicate the default Tandem DQN configuration, and gray lines always correspond to the active agent's performance.
2.2 The Tandem Effect
We begin by reproducing the striking observation in [Fujimoto al., 2019b] that the passive learner generally fails to adequately learn from the very data stream that is demonstrably sufficient for its architecturally identical active counterpart; we refer this phenomenon as the 'tandem effect' (Fig. 1, bottom). We ascertain the generality of this finding by replicating it across a broad suite of environ- ments and agent architectures: Double-DQN on 57 Atari environments (Appendix Figs. 10 11), adapted agent variants on four Classic Control domains from the OpenAl Gym library [Brockman et al., 2016] and the MinAtar domain [Young and Tian, 2019] (Appendix Figs. 12 & 15), and the distributed R2D2 agent [Kapturowski et al., 2019] (Appendix Fig. 14). Details on agents and environments are provided in the Appendix
Empirically, we make the informal observation that while active and passive Q-networks tend to produce similar values for typical state-action pairs under the active policy (where the action is the active Q-value function's 's argmax for given state), their values are less correlated for other (non-argmax) actions, and in fact the active and passive greedy policies of a Tandem DQN tend to disagree in large fraction of states under the behavior distribution (on average > 75% of states, after 100M steps of training, across Atari games; see Appendix Fig. 13). Moreover, in a fraction (<s 12/57) of Atari games, we observe the passive agent's network to strongly over-estimate fraction of state-action values, with the over-estimation growing as training progresses.
"Our choice of Double-DQN as a baseline is motivated by its relatively strong performance and robustness compared to vanilla DQN [Mnih et al., 2015], paired with its simplicity compared to later variants like Rainbow [Hessel et al., 2018), which allows for more easily controlled experiments with fewer moving parts.
https:/giihul.oo/depmninaaa 'We provide two Tandem RL implementations: https://githubb. com/teepmidd/deepmine hetpteeeemiiind-eereeeeeeeeeeeee research/tredd
master/tandem_dde dqn based on the DQN Zoo [Quan and Ostrovski. 2020]. and https://ithhub.co https:/github.com/gogggg com//gogle//
oppamiiefrecmmstteddppannilliiie -dgeeeemmgnggggge based on the Dopamine library [Castro et al., 2018).
3

---

Brea cout
Pong
Seiguesst
Space invaders
20
4/1%
Toloat returs
320
10:00
210
Vanilla Tandom DQN
Same terget 0
5003
3000
Seme target
20
Surri Earget
150 Enxtrenment Trames (mill om))
50
103
200
53
100
1.5
250
100
355
200
50
100
150
203
Enaicemmnnt trames (rillonn)
Ehv nowmmnmtttt trames uniiionn)
EmV ronmert frames (millnna)
Figure 2: Active vs. passive performance when using the active agent's target policy and/or value function for constructing passive bootstrapping targets.
3 Analysis of the Tandem Effect
In line with existing explanations [Levine et al., 2020), we propose that the tandem effect is primarily caused by extrapolation error when certain state-action pairs under-represented in the active agent's behavior data. Specifically with E-greedy policies, even small over-estimation of the values of rarely seen actions can lead to sufficient behavior deviations to cause catastrophic under-performance.
We further extend this hypothesis: in the context of deep reinforcement learning (i.e. with non-linear function approximation), inadequate data distribution can drive over-generalization [Bengio et al., 2020], making such erroneous extrapolation likely. While the tandem effect can show up as learning inefficiency even in the tabular case [Xiao et al., 2021]. it proves especially pernicious in the case of non-linear function approximation, where erroneous extrapolation can lead to errors not just on completely unseen, but also rarely seen data, and can persist in the infinite-sample limit.
Coalescing this view and past analyses of challenges in offline RL (e.g. [Levine al., 2020, Fujimoto et al., 2019b, Liu et al., 2020]) into the following three potential contributing factors in the tandem effect provides a natural structure to our analysis:
Bootstrapping (B) The passive agent's 's bootstrapping from poorly estimated (in particular, over- estimated) values causes uny initially small mis-estimation to get amplified.
Data Distribution (D) Insufficient coverage of sub-optinal actions under the active agent's 's policy may lead to their mis-estimation by the passive agent. the case of over-estimation, this may lead to the passive agent's 's under-perte under-performance.
Function Approximation (F) A non-linear function approximator used as a Q-value function may tend to wrongly extrapolate the values of state- taatencction pairs underrepresented in the active agent's behavior distribution. This tendency can be inherent and persistent, in the sense of being independent of initialization and not being reduced with increased training on the same data distribution.
These proposed contributing factors are not at all mutually exclusive; they may interact in causing or exacerbating the tandem effect. Insufficient coverage of sub-optimal actions under the active agent's behavior distribution (D) may lead to insufficient constraint on the respective values, which allows for effects of erroneous extrapolation by a function approximator (F). Where this results in over-estimation, the use of bootstrapping (B) carries the potential to 'pollute' even well-covered state-action pairs by propagating over-estimated values (especially via the max operator in the case of Q-learning). In the next sections we empirically study these three factors in isolation, to establish their actual roles and relative contributions to the overall difficulty of passive learning.
3.1 The Role of Bootstrapping
One distinguishing feature of reinforcement learning as opposed to supervised learning is its frequent use of learned quantities preliminary optimization targets, most prominently in what is referred to as bootstrapping in the widely used TD algorithms [Sutton, 1988], where preliminary estimates of the value function form Q(s, a) + r + (s(', arg max,, Q(s', ((s',a')), a')). where Q denotes the parametric Q-value function, and Q is the target network Q-value function, i.e. time-delayed copy of Q.
used update targets. In the Double-DQN algorithm these updates take the
Four value functions are involved in the active and passive updates of Tandem DQN: Q QA, Qr and Qp. where the A/ subscripts refer to the Q-value functions of the active and passive agents,

---

respectively. The use of its own target network by the passive agent makes bootstrapping plausible strong contributor to the tandem effect. To test this, we replace the target values and/or policies in the update equation for the passive agent, with the values provided by the active agent's ss value functions:
r - Qp(s' arg a Qp(s',
Vanilla Tandem DQN
r Qal(', arg max, Qp(6','))
Same Target Q Same Target
Qp(s,a) +
r Qr(s', arg max Qa(s',
Qa(s', arg max Q (ss''a'))
Same Target T&Q
As shown in Fig. 2, the use of the active value functions as targets reduces the active-passive gap by only small amount. Note that when both active target values and policy are used, both networks are receiving an identical sequence of targets for their update computations, sequence that suffices for the active agent to learn successful policy. Strikingly, despite this the tandem effect appears largely preserved: in all but the easiest games (e.g. PONG*) the passive agent fails to learn effectively.
To more precisely understand the effect of bootstrapping with respect to a potential value over- estimation by the passive agent, in Appendix Fig. 16 we also show the values of the passive networks in the above experiment compared to those of the respective active networks. As hypothesised, we observe that the vanilla tandem setting leads to significant value over-estimation, and that indeed bootstrapping plays a substantial role in amplifying the effect: passive networks trained using the active network's 's bootstrap targets do not over-estimate compared to the active network at all.
These findings indicate that a simple notion of value over-estimation itself is not the fundamental cause of the tandem effect, and that (B) plays amplifying, rather than causal role. Additional evidence for this is provided below, where the tandem effect occurs in a purely supervised setting without bootstrapping.
3.2 The Role of the Data Distribution
The critical role of the data distribution for offline learning is well established [Fujimoto et al., 20190 9b,, Jacq et al., 2019, Liu et al., 2020, Wang et al., 2021]. In particular, Wang et al. [2020a] showed that simpler notions of state-space coverage may not suffice for efficient offline learning with function approximation (even in the linear case and under strong realizability assumption); much stronger assumptions on the data distribution, typically not satisfied practical scenarios, may actually be required. Here we extend past analysis empirically, by investigating how properties of the data distribution (e-g. stochasticity, stationarity, the size and diversity of the dataset, and its proximity to the passive agent's 's own behavior distribution) affect its suitability for passive learning.
simple way to affect the data distribution's 's state-action coverage (albeit in a blunt and uniform way) is by varying the exploration parameter of the active agent's e-greedy behavior policy (for training, not for evaluation). Note that a higher E parameter affects the active agent's 's own training performance, as its ability to navigate environments requiring precise control is reduced. In Fig. (top) we therefore report the relative passive performance (i.e. a fraction of the active agent's performance, which itself also varies across parameters), with absolute performance plots included in the Appendix for completeness (Fig. 17). We observe that the relative passive performance is indeed substantially improved when the active behavior policy's 's stochasticity (and as a consequence its coverage of non-argmax actions along trajectories) is increased, and conversely it reduces with greedier behavior policy, providing evidence for the role of (D).
The exploration parameter
Sticky actions An alternative potential source of stochasticity is the environment itself, e.g. the of "sticky actions' in Atari [Machado et al., 2018]: with fixed probability. an agent action is ignored (and the previous action repeated instead). This type of environment-side stochasticity should not be expected to cause new actions to appear in the behavior data, and indeed Fig. 3 (bottom) shows no substantial impact on the tandem effect.
Replay size hypothesize that the difference is due to the vastly different dataset size (full training of 200M
Our results contrast with the strong offline RL results in [Agarwal et al., 2020]. We
'PONG is an outlier in that it only has 3 actions, and in a large fraction of states actions have no (irreversible) consequences, making greedy policies somewhat robust to errors in the underlying value function.
5

---

Breakout
Pang
Seaquest
Space invaders
1.0
2.0
1.0
04
0.4
0.4
02
52
0.2
0%
50
100
150
200
50
100
150
200
0.0
50
100
150
200
50
100
150
209
Envronment frames millions)
Enwformmnt fimms (ilrr))
Erwfronment frames Eonst
Enwiorment frames Ond land)
Resetee paette pertommete
1.0
Breakout
1.0
Pong
1.0
Seaquest
3.0
Spa Invadars
0.8
0.8
0.5
Stick Action
Regular Aarrt
022
0.2
0.2
0.
1:0
150
200
0.0g
50
105
150
200
0.0g
50
100
150
200
5.0
50
103
150
200
Breiurment Bornes Iniiloonst
Envromment Famees in (llms))
Enviormment flmiis (milions)
Endironmunt: flrms (mill lon))
Figure 3: Passive as fraction of active performance for varying active e-greedy behavior policies (top); regular Atari vs sticky-action. Atari (bottom). We report relative passive performance, as active performance varies across configurations. See Appendix Figs. 17 & 18 for absolute performance.
Seaquest
M
veture
320
10
10300
230
120
-10
replay con- 2e6
replay bipe Led
5000
3000
replay s2e 4nd
Environment frares
(milkorss
Ern
En Imillonss
Figure 4: Active vs. passive performance for different replay sizes (for passive agent).
transitions vs. replay buffer of 1M). Interpolating between the tandem and the offline RL setting from [Agarwal et al., 2020], we increase the replay buffer size, thereby giving the passive agent access to somewhat larger data diversity and state-action coverage (this does not affect the active agent's training as the active agent is constrained to only sample from the most recent 1M replay samples, as in the baseline variant). A larger replay buffer somewhat mitigates the passive agent's 's under-pertee pedee--prffrmance (Fig. 4), though it appears to mostly slow down rather than prevent the passive agent from eventually under-performinn sufficient replay buffer size may depend on the effective state-space size of an environment, we also perform analogous experiments the (much smaller) classic control domains; results (Appendix Fig. 22) remain qualitatively the same.
its active counterpart substantially. As we suspect that
Note that for fixed dataset size, sample diversity can take different forms. Many samples from single policy may provide better coverage of states on, or near, policy-typical trajectories. Meanwhile, larger collection of policies, with fewer samples per policy, provides better coverage of many trajectories at the expense of lesser coverage of small deviations from each. To disentangle the impact of these modalities, while also shedding light the role of stationarity of the distribution, we next switch to the Forked Tandem' variation of the experimental paradigm.
Fixed policy Upon forking, the frozen active policy is executed to produce training data for the passive agent, which begins its training initialized with the active network's weights. Note that this constitutes a stronger variant of the tandem experiment. At the time of forking, the agents do not merely share analogous architectures and equal 'data history', but also identical network weights (whereas in the simple tandem setting, the agents were distinguished by independently initialized networks). Moreover, the data used for passive training can be thought of as "best-case scenario': generated by single fixed policy, identical to the passive policy at the beginning of passive training. Strikingly, the tandem effect is not only preserved but even exacerbated this setting (Fig. 5, top): after forking, passive performance decays rapidly in all but the easiest games, despite starting from a near-optimally performing initialization. This re-frames the tandem effect as not merely the difficulty of passively learning to act, but even to passively maintain performance. Instability appears to be inherent the very learning process itself, providing strong support to the hypothesis that an interplay between (D) and (F) is critical to the tandem effect.

---

Breakcatt
Pong
Seaquest
Space invaders
20
15020
Tolott marm
310
10010
230
Firked 29
2000
Foried 9
Forked 100
100
15c
200
53
103
150
269
50
300
150
200
50
100
150
203
Envicommnnt Tramns prilkona)
rosment frames (millon))
Ensiorment Trames (rilkors)
Envretment trames (rilliana)
Breaknut
Pong
Seaquest
Space invaders
420
15020
w nm
W
330
10020
Eslore
5030
120
-20
Etwicrmmnnt framess (millonss)
Encrromment Frames (rillinn)
Etrd Etwiicement framis (millors)
Envrrnmment framess (rilliema)
Figure 5: Performance of a Forked Tandem DQN, training passively after freezing its data generating policy (top) or its replay buffer (bottom). Vertical lines indicate forking time points.
In Appendix Fig. 23 we additionally show that similarly to the regular tandem setting, stochasticity of the active policy after forking influences the passive agent's 's ability to maintain performance.
Fixed replay A variation on the above experiments is to freeze the replay buffer while continuing to train the passive policy from this fixed dataset. Instead of stream of samples from a single policy, this fixed data distribution now contains fixed number of samples from a training process of the length of the replay buffer, i.e. from number of different policies. The collapse of passive performance here (Fig. 5, bottom) is less rapid, yet qualitatively similar. In Appendix Fig. 24 we present yet another variant of this experiment with similar results, showing that the effect is robust to minor variations in the exact way of fixing the data distribution of a learning agent.
These experiments provide strong evidence for the importance of (D): a larger replay buffer, containing samples from more diverse policies, can be expected to provide an improved coverage of (currently) non-greedy actions, reducing the tandem effect. While the forked tandem begins passive learning with the seemingly advantageous critically limited in this case. In the frozen-policy case, a large number of samples from the very same e-greedy policy can be expected to provide very little coverage of non-greedy actions, while in the frozen-replay case, smaller number of samples from multiple policies can be expected to only do somewhat better this regard. In both cases the tandem effect is highly pronounced.
initialization, state-action coverage is
On-policy evaluation The strength of the last two experiments lies in the observation that, since active and passive networks have identical parameter values at the beginning of passive training, their divergence cannot be attributed to small initial differences getting amplified by training on an inadequate data distribution. With so many factors held fixed, the collapse of passive performance when trained on the very data distribution produced by its own initial policy begs the question whether off-policy Q-learning itself is to blame for this failure mode, e.g. via statistical werressimmtion bias introduced by the max operator [van Hasselt, 2010]. Here we provide a negative answer, by performing on-policy evaluation with SARSA [Rummery and Niranjan, 1994] (Fig. 6). and even purely supervised regression on the Monte-Carlo returns (Appendix Fig. 25), in the forked tandem setup. While evaluation succeeds, in the sense of minimizing evaluation error the given behavior distribution, atypical action values under the behavior policy suffer substantial estimation error, resulting in occasional over-estimation. The resulting e-greedy control policy under-performs the initial policy at forking time as catastrophically as in the other forked tandem experiments (more details in Appendix A.3). Strengthening the roles of (D) and (F) while further weakening that of (B), these observations point to an inherent instability of offline learning, different from that of Baird's famous example [Baird, 1995] the 'Deadly Triad' [Sutton and Barto, 2018, van Hasselt et al., 2018]; an instability that results purely from erroneous extrapolation by the function approximator, when the utilized data distribution does not provide adequate coverage of relevant state-action pairs.
Self-generated data by the passive agent is needed to correct for the tandem effect?'. While full investigation of this question exceeds the scope of this paper and is left for future work, the tandem setup lends itself to
Our final empirical question in this section is 'How much data generated
7

---

Breakout
Pong
Seaquest
Space invaders
420
mm
15020
3000
Myn
Estoce
230
Foried (275
5030
Forted (550
Filled 103
100
15.
200
53
103
150
300
53
300
15.0
200
50
100
150
203
Enaierment mamee (oonn)
rosmant trames irillong
Enaifomment trmmee (rilionn)
Envrntment frames (rillion))
Figure 6: Passive performance in Forked Tandem DQN after policy evaluation with SARSA.
Seaquest
Space
330
10
15000
3000
200
2000
10%
3000
1000
50
50
Figure 7: Passive performance for different amounts of self seff--gneerted data in the passive agent's replay batches.
simple experiment: both agents interact with the environment and fill individual replay buffers, one of them (for simplicity still referred to as 'passive") however learns from data stochastically mixed from both replay buffers. Fig. 7 shows that even a moderate amount (10%-20%) of 'own' data yields substantial reduction of the tandem effect, while 50/50 mixture completely eliminates it.
3.3 The Role of Function Approximation
We structure our investigation of the role of function approximation in the tandem effect into two categories: the optimization process and the function class used.
Optimization Agarwal et al. [2020] and Obando-Cerov and Castro [2021] demonstrated that the Adam optimization algorithm [Kingma and Ba, 2015] outperforms RMSProp [Tieleman and Hinton, 2012] used in experiments. In Appendix Fig. 20 we show that while both active and passive agents perform better with Adam, the tandem effect itself is unaffected by the choice of optimizer.
Another plausible hypothesis is that the passive network suffers from under-fitting and requires more updates on the same data attain comparable performance the active learner. Varying the number of passive agent updates per active agent update step, find that more updates worsen the performance the passive agent (Appendix Fig. 21). This rejects insufficient training as a possible cause, and further supports the role of (D). We also note that, together with the forked tandem experiments in the previous section, this finding distinguishes the tandem effect from the issue of estimation eиror in the offline learning setting [Xiao et al., 2021]: while in the tabular setting estimation error dominates the learning challenge and a sufficient training duration (assuming full state-space coverage) guarantees convergence good solution, this is not necessarily the case with function approximation trained on a given data distribution.
Function class Given that the active and passive agents share an identical network architecture, the passive agent's under-performance cannot be explained by an insufficiently expressive function approximator. Performing the tandem experiment with pure regression the passive network's outputs towards the active network's (a variant of network distillation [Hinton et al., 2015]), instead of TD based training, we observe that the performance gap indeed vastly reduced and in some games closed entirely (see Appendix Fig. 19); however, strikingly, it remains in some.
Next, we vary the function class of both networks by varying the depth and width of the utilized Q-networks on a set of Classic Control tasks. As can be seen in Fig. 8 (and Appendix Fig. 28), magnitude of the cetive-passive performance gap appears negatively correlated with network width, which is in line with (F): an increase in network capacity results in less pressure towards over-generalizing to infrequently seen action values and an ultimately smaller tandem effect. On the other hand, the gap seems to correlate positively with network depth. We speculate that this may

---

CartPole hidden layers
CartPole- 1 hidden loyers
CartPdee hidden layers
10
06
0.5
0.5
0.
Re
04
04
04
522 in ho
02
022
100
202
302
0%
220
200
403
0%0
320
thousanss
dhousando
Figure 8: Passive performance as fraction of active performance in CartPole: varying number of hidden layers and units.
Breaknut
Pong
Seaquest
Space invaders
15020
1000
-300
10
3000
200
layera00
10020
1000
-10
5020
2000
-20
100
150
200
52
102
130
200
50
200
25.0
200
120
150
Ereicoment framss (milkors)
rosment frames (rilion))
Enwiorment hamms Im (ilor)))
Envronmennt framms (rilliona)
Figure 9: Active vs. passive performance, with first 1s of layers of active/passive networks shared.
relate to the finding that deeper networks tend to be biased towards simpler (e.g. lower rank) solutions, which may suffer from increased over-generalization [Huh et al., 2021, Kumar al., 2020a].
Finally, we investigate varying the function class of only the passive network by sharing the weights of the first le (out of 5) layers of active and passive networks, while constraining the passive network to only update the remaining top 5 - & layers, and using the 'representation' at layer & acquired through active learning only. This reduces the 'degrees of freedom' of the passive agent, which we hypothesize reduces its potential for divergence. Indeed, Fig. 9 illustrates that passive performance correlates strongly with the number of tied layers, with the variant for which only the linear output layer is trained passively performing on par with the active agent. A similar result is obtained in the forked tandem setting, see Appendix Fig. 27. This finding provides strong indirect hint towards (F); with only part of the network's 's layers being trained passively, much of its 'generalization capacity' is shared between active and passive agents. States that are not aggregated by the shared bottom layers (only affected by active training) have to be 'erroneously" aggregated by the remaining top layers of the network for over-generalization to occur. A more thorough investigation of this, exceeding the scope of this paper, may involve attempting to measure (over-)generalization more directly, e.g. via gradient interference [Bengio et al., 2020).
4 Applications of the Tandem Setting
In addition to being valuable for studying the challenges in offline RL, we propose that the Tandem RL setting provides analytic capabilities that make it useful tool in the empirical analysis of general (online) reinforcement learning algorithms. At its core, the tandem setting aims decouple learning dynamics from its impact on behavior and the data distribution, which are inseparably intertwined in the online setting. While classic offline RL achieves a similar effect, as an analytic tool has the potential downside of typically using a stationary distribution. Tandem RL, on the other hand, presents the passive agent with a data distribution which realistically represents the type of non -tatttttionartty encountered in online lcarning process, while still holding that distribution independent from the learning dynamics being studied. This allows Tandem RL to be used to study, e.g., the impact of variations in the learning algorithm on the quality of learned representation, without having to control for the indirect confounding effect of a different behavior causing a different data distribution.
While extensive examples of this exceed the scope of this paper, Appendix A.5 contains a single such experiment, testing QR-DQN [Dabney et al., 2018] as a passive learning algorithm (the active agent being a Double-DQN). This is motivated by the observation of Agarwal et al. [2020], that QR-DQN outperforms DQN in the offline setting. QR-DQN indeed appears to be a nontrivially different passive learning algorithm, significantly better in some games, while curiously worse in others (Fig. 29).

---

5 Discussion and Conclusion
At high level, our work can be viewed as investigating the issue of (in cimppatttibyt between the data distribution used to train function approximator and the data distribution relevant in its evaluation. While in supervised learning, generalization can be viewed as the problem of transfer from a training to (given) test distribution, the fundamental challenge for control in reinforcement learning is that the test distribution is created by the very outcome of learning itself, the learned policy. The various difficulties of learning to act from offline data alone throw into focus the role of interactivity in the learning process: only by continuously interacting with the environment does an agent gradually 'unroll' the very data on which its performance will be evaluated.
This need not be an obstacle in the case of exact (i.e. tabular) functions: with sufficient data, extrapolation error can be avoided entirely. In the case of function approximation however, as small errors compound rapidly into a difference in the underlying state distribution, significant divergence and, as this and past work demonstrates, ultimately catastrophic under-performe occur. Function approximation plays a two-fold role here: (1) being an approximation, it allows deviations in the outputs; (2) as the learned quantity, it is (especially in the non-lineat case) highly sensitive to variations in the imput distribution. When evaluated for control after offline training, these two roles combine in a way that is 'unexplored' by the training process: minor outpul errors cause drift in behavior, and thereby a drift in the test distribution.
can
While related, this challenge is subtly different from the well-known divergence issues of off-policy learning with function approximation, demonstrated by Baird's famous counterexample [Baird, 1995] (see also [TTsisikkis and Van Roy, 1996)) and conceptualized as the Deadly Triad [Sutton and Barto, 2018, van Hasselt et al.. 2018]. While these depend on bootstrapping as a mechanism to cause a feedback-loop resulting in value divergence, our results show that the offline learning challenge persists even without bootstrapping, as small differences in behavior cause a drift the 'test distribution' itself. Instead of a training- time output drift caused by bootstrapping, the central role is taken by a test-time drift of the state distribution caused by the interplay of function approximation and fixed data distribution (as opposed to dynamically self-generated data).
Our empirical work highlights the importance of interactivity and 'learning from your own mistakes' in learning control. Starting out as an investigation of the challenges in offline reinforcement learning, it also provides a particular viewpoint on the classical online reinforcement learning case. Heuristic explanations for highly successful deep RL algorithms like DQN, based on intuitions from (c.g.) approximate policy iteration, need to be viewed with caution in light of the apparent hardness of a policy improvement step based on approximate policy evaluation with function approximator.
Finally, the forked tandem experiments show that even high- serforerforing initializations are not robust to collapse of control performance, when trained under their own (but fixed!) behavior distribution. Not just learning to act, but even maintaining performance appears hard in this setting. This provides an intuition that we distill into the following working conjecture: The dynamics of deep reinforcement learning for control are unstable on (almost) any fixed data distribution.
Expanding on the classical on- vs. off-policy dichotomy, we propose that indefinitely training on any fixed data distribution, without strong explicit regularization or additional inductive bias, gives rise to "exploitation of gaps in the data' by a function approximator, akin to the over-fitting occurring when over-training on fixed dataset in supervised learning. Interaction, i.e. generating at least moderate amounts of one's own experience, appears to be a powerful, and for the most part necessary, regularizer and stabilizer for learning act, by creating dynamic equilibrium between optimization of a function approximator and its own data-generation process.
Broader impact statement This work lies in the realm of foundational RL, contributing to the fundamental understanding and development of RL algorithms, and as such is far removed from ethical issues and direct societal consequences. On the other hand, it highlights the empirical difficulty and limitations of offline deep RL for control increasingly important for practical applications, e.g. robotics, where interactive data is costly, and learning from offline datasets is desirable. In this way complements existing theoretical hardness results this area and provides additional context to existing empirical techniques which aim to overcome or circumvent those limitations. We believe that an improved understanding of these challenges can play an important role in creating robust and stable offline learning algorithms whose outputs can be more safcly deployed in the real world.
10

---

Acknowledgements
We would like to thank Hado Hasselt and Joshua Greaves for feedback early draft of this paper, and Zhongwen Xu for an unpublished related piece of work at DeepMind that inspired some of our experiments. We also thank Clare Lyle, David Abel, Diana Borsa, Doina Precup, John Quan, Marc G. Bellemare, Mark Rowland, Michal Valko, Remi Munos, Rishabh Agarwal, Tom Schaul and Yaroslav Ganin, and many other colleagues at DeepMind and Google Brain for the numerous discussions that helped shape this research.
References
Joshua Achiam, Ethan Knight, and Picter Abbeel. Towards characterizing divergence in deep Q-learning arXiv preprint arXiv:1903.c 08894, 2019.
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In Proceedings of the 37th International Conference Machine Learning, volume 119, pages 104-114. PMLR, 2020.
Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995, pages 30-37. Elsevier, 1995.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The Arcade Learning Environment: evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning, pages 767-777, 2020.
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://githut: com/ google/jax.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 1606.001:4 2016.
Jacob Buckman, Carles Gelada, and Mare G. Bellemare. The importance of pessimism in dhedddataset policy optimization. In International Conference on Learning Representations, 2021.
David Budden, Matteo Hessel, John Quan, Steven Kapturowski, Kate Baumli, Surya Bhupatiraju, Aurelia Guy, and Michael King. RLax: Reinforcement Learning JAX, 2020. URL http:
//github. com com//eepmindd/lax.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G. Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint
10, 2018.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Rasool Fakoor, Jonas Mueller, Pratik Chaudhari, and Alexander J. Smola. Continuous doubly constrained batch reinforcement learning. arXiv preprint 10X1::2102.09255, 2021.
Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708 2019a.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Proceedings of the 36th International Conference on Machine Learning, volume 97,
pages 2052-2062. PMLR, 2019b.
Richard Held and Alan Hein. Movement-producee stimulation in the development of visually guided behavior, Journal of Comparative and Physiological Psychology, 5675):872-876, 1963.

---

Tom Hennigan, Trevor Cai, Tamara Norman, and Igor Babuschkin. Haiku: Sonnet for JAX, 2020.
URL http://github http:///hhuuu..com///epeppppt/ com/deepmind/ ttm-aaiu.
Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski. Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: combining improvements in deep reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, 2018.
Matteo Hessel. David Budden, Fabio Viola, Mihacla Rosca, Eren Sezener, and Tom Hennigan. Optax: Composable gradient transformation and optimisation, in JAX!, 2020. URL atttp://gihhn optax.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung. Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2 10427, 2021.
Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pages 1990-2999, 2019.
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. MOReL: Model- based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 21810 21810-21823, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the International Conference on Learning Representations, 2015.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy Q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, volume 32, 2019.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit unde-praaaraamtttiiitiooo inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. 14498, 2020a.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. DisCor: Corrective feedback in reinforcement learning via distribution correction. In Advances in Neural Information Processing Systems, volume 33, pages 18560-18572, 2020b.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pages 1179-1191, 2020c.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005. 004.0,, 2020.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch reinforce- ment learning without great exploration. arXiv preprint arXiv:2007 008022, 2020.
Marlos C Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the Arcade Learning Environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Reseurch, 61:523-562, 2018.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. arXiv preprint (0l::20006.06647,, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
12

---

Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint Stt::20000093599 2020.
Johan S. Obaadoo-oCoo Ceroon and Pablo Samuel Castro. Revisiting Rainbow: Promoting more insightful and inclusive deep reinforcement learning research. In Proceedings of the 38th International Conference Machine Learning. PMLR, 2021.
John Quan and Georg Ostrovski. DQN 7o0: Reference implementations of DQN-based agents, 2020.
URL http:://gt wtt./oooom//aeaa/ee comm/deeppinnd/ddm..ee
Gavin A. Rummery and Mahesan Niranjan. On-line Q-leaning using connectionist systems. Techni- cal report, University of Cambridge, Department of Engineering Cambridge, UK, 1994.
Julian Schritwiessrr Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. arXiv preprint arXiv:2104.06294 2021.
Richard S. Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3 (1):9-44, 1988.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High confidence policy improvement. In International Conference on Machine Learning, pages 2380-2388. PMLR, 2015.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5: RMSProp. COURSERA: Neural Networks for Machine Learning, 2012.
John N. Tsitsiklis and Benjamin Van Roy. Analysis of temporal-differenc learning with function ap- proximation. In Proceedings of the 9th International Conference on Neural Information Processing Systems, page 1075-1081, Cambridge, MA, USA, 1996. MIT Press.
Hado van Hasselt. Double Q-learning. In Advances in Neural Information Processing Systems, volume 23, 2010.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 18/2. 2018
Ruosong Wang, Dean P. Foster, and Sham M. Kakade. What the statistical limits of offline RI. with linear function approximation? arXiv preprint arXiv:2010.1 1895, 2020a.
Ruosong Wang. Yifan Wu, Ruslan Salakhutdinov, and Sham M. Kakade. Instabilities of offline RL with pre-trained neural representation. arXiv preprint arXiv:2/03.049// 2021.
Ziyu Wang, Alexander Novikov, Konrad Zoina, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. arXiv preprint arXiv:2006. 15 134, 2020b.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprini arXiv: 1911. 1361, 2019.
Yue Wu, Shuangfei Zhou, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021.
Chenjun Xiao, Ilbin Lee, Bo Dai, Dale Schuurmans, and Csaba Szepesvari. On the sample complexity of batch reinforcement learning with policy-induced data. arXiv preprint arXiv:2 222.000009977 2021.
Kenny Young and Tian Tian. MinAtar: An Atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint vrXiv:1903.03177 2019.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. arXiv preprint arXiv:2005. 13239, 2020.
13


Reviews:
[{"review_id": "vj79aJOrewo", "reviewer": "Reviewer_kz5A", "summary": "This paper provides detailed empirical analysis to study the challenge of offline reinforcement learning. The authors propose to use the Tandem RL setup as the analytic tool, where one active agent performs usual online training loop, while the passive agent can only learn from the data generated by the action learner. The results provide some insights for revealing the difficulty of offline RL. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Contributions:\n\nThe paper is very well written. The definitions and experiment setups are clearly defined. I really enjoy reading this paper. \n\nThe main message of this paper is that, as function approximators for value functions, deep neural networks are extrapolating in an uncontrolled fashion especially under insufficient coverage of the fixed data set, and as such blow up errors in an iterative training scenario. I think this is well expected as even in the linear case such extrapolation errors occur as suggested by previous theoretical works. \n\nIt is really good to see that the authors actually try to provide detailed analysis to reveal the hardness of offline RL. Most previous works in deep offline RL literatures just try to demonstrate their proposed algorithms are good in some cases, but it’s rarely to see offline RL algorithm can fail very easily, and what are the major factors that cause the trouble. \n\nWeakness:\n\nI have the following comments and questions that I hope the authors to address:\n\nWhat does the over-generalization exactly mean in the paper? Does that mean over estimate the unseen data or something even stronger? Is it possible to visualize such over-generalization?\n\nI think the data distribution assumption (Hypothesis 2) as well as the supporting experiments can be strengthened such that it can distinguish between \"enough data coverage\"  and \"enough on-current-policy data coverage\". It has been shown that the \"enough data coverage\" is not enough to guarantee efficient offline RL (e.g. see [2,3]). Maybe the experiments shown in the paper connects the failure of passive learning to the later one?  \n\nAlmost all experiments are in the policy optimization problem. I am wondering if the hardness of offline RL already exists in the simpler policy evaluation problem since the Hypothesis 2 and 3 seem also exist there (although I understand it's hard to design experiments for policy evaluation in this tandem setups). \n\nThere is also some recent theoretical paper [1] trying to reveal the difficulty of offline RL. But I guess the major factor there is the estimation error that comes from the insufficient coverage of the data (as they consider the tabular setting so there is no extrapolation error while the problem still exists). I am wondering if the authors distinguished the estimation error and extrapolation error in this paper? \n\n[1]  Xiao, C., Lee, I., Dai, B., Schuurmans, D. and Szepesvari, C., 2021. On the Sample Complexity of Batch Reinforcement Learning with Policy-Induced Data. arXiv preprint arXiv:2106.09973.\n\n[2] Wang, R., Foster, D.P. and Kakade, S.M., 2020. What are the Statistical Limits of Offline RL with Linear Function Approximation?. arXiv preprint arXiv:2010.11895.\n\n[3] Wang, R., Wu, Y., Salakhutdinov, R. and Kakade, S.M., 2021. Instabilities of offline rl with pre-trained neural representation. arXiv preprint arXiv:2103.04947.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal issue has been found. ", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issue has been found. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "t4dmZWNxwrG", "reviewer": "Reviewer_LigT", "summary": "This paper presents an empirical study on offline deep reinforcement learning in atari environments. The authors use tandem learning -- where an agent B learns from experience collected from another agent A. The authors consider 3 hypotheses as to why agent B tends to perform worse in this setting, and provide many ablation-type experiments to support/reject these hypotheses.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main comments\n---------------\n\nOverall the paper is well written and relatively easy to follow. Offline RL is an important domain, and the experimental setup enables the authors to probe many important questions. The authors specify the hypotheses they are testing and interpret what the experiments say about these. This is a good step towards good empirical practices. \n\nCompared to previous work, there are some novel and interesting experiments: 1) the authors consider using different bootstrap targets for the tandem learning, and shows that the tandem-gap persists even if using the bootstrap targets of \"online\" agent. 2) The authors study the effects of \\epsilon and replay buffer size on the tandem gap. The question here is interesting, but the results vary among environments which makes it hard to draw conclusions. 3) A method of \"forking\" a tandem learning, which demonstrates that the tandem gap appears quickly in this case. \n\nThere are two main issues with the paper 1) the novelty of the setup is not clear and 2) the clarity of the hypotheses and what the experiments say could be improved. \n\n\n**Novelty**\n\nThe experimental setup of \"tandem learning\" seems to already have been proposed by [13](Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration.). Indeed, the authors state that \"We begin by reproducing the striking observation in [13]\". However, in the abstract, the authors state that \"we propose the “tandem learning” experimental paradigm\". Could the authors clarify their contribution of proposing tandem learning vis-a-vis [13]?\n\n\n**Clarity**\n\nIt would be preferable to state the 3 hypotheses of Section 3 in a mathematical language to make them more precise. It is unclear what is meant by e.g. \"the stationarity of the data distribution [...] play only a small role.\". What is a \"small\" role? Since the hypotheses are stated in a relatively \"hand-wavy\" way, it becomes hard to see what the experiments say about them. Since the hypotheses drive the experiments, it should be crystal clear what they are and how they differ.\n\n\nIn general, only graphs are presented and used to draw conclusions. When making precise statements, it would be preferable to use some specific statistics and then perform statistical significance testing to make sure the results are not random noise. E.g. the authors state that \"As can be seen in Fig. 3 (top), relative passive performance is indeed substantially improved\".  Fig 3 shows improvements for some games, but differences for seaquest or pong might as well be due to chance. The author states \"Unless stated otherwise, all results show averages over at least 5 seeds, with confidence intervals indicating variation over seeds.\" This appears to be the ONLY place in the text where seeds are mentions. What experiments use more than 5 seeds? And if there are any, how many seeds are used? At any rate, 5 seeds are very low for making statistically significant observations. \n\n\n\n\nMinor Comments\n-----------\n\nI doubt most reviewers are familiar with \"Held & Hein’s classic 1963 experiment\" as stated in the abstract. Referring to it so early will likely confuse most readers.\n\nMany figures lack labeled axes, see e.g. Fig 3,4, and 5.\n\nIs CUDA set to deterministic mode? How are the seeds selected?\n\nDouble-DQN, the used architecture, is a relatively simple model that isn't used in practice. I think it would have been better to use rainbow, and simply use hyperparameters from reference implementations. By not changing such parameters, the authors can keep the \"moving parts\" to a minimum.\n\nSome experiments seem to use only 4 environments. Whenever results for more environments are available, I think they should be presented in the main paper.\n\n## post rebuttal comment\n\nI have read the rebuttal and thank the authors for their response. \n\nMy two main objections to the paper remains. Firstly, I don't think the novelty is particularly high. The authors state that a novel aspect is \"explicit framing of the Tandem as an analytic tool\". The tool has clearly already been used implicitly as an analytic tool in a previous paper. Framing a known method as an explicit analytic tool doesn't count as novel to me. Secondly, I think the clarity is lacking due to the lack of statistical rigor.  The authors state that \"given the inner complexity of any modern deep RL algorithm, we have little hope of being able to pin these down with full mathematical rigour\". Full rigor might be unattainable, nonetheless I think the paper could be improved significantly along this direction. I believe that the paper would be better served by using more seeds on fewer environments instead of using a few seeds across the whole 57 task Atari suite. However, the paper also has many strong points and to me this is a borderline paper. I am clearly in the minority regarding these opinions, and given the other scores I expect to see the paper in the final program and I'm sure it would bring about many interesting discussions despite the issues given above.\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "dJAmu3Tv0C", "reviewer": "Reviewer_2Fyh", "summary": "This paper studies the situation of passive reinforcement learning, simulating the famous Held and Hein twin cats paper where an active agent collects data to learn while the passive agent experiences the same data (but does not choose actions). The result, running Double-DQN on Atari, is similar - that the passive learner is often much worse. The paper then attempts to study why by evaluating three hypotheses: bootstrapping, data distribution, and function approximation, through a sequence of experiments. The conclusion is mixed, but mostly supports that data distribution is the key problem, with good evidence that bootstrapping is relatively less important than previously emphasized. Overall, the paper is a clever and thorough study of passive learning beyond the existing literature on offline RL.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea of the paper is to study “tandem” reinforcement learning: one agent (“active”) runs online RL as usual, another agent (“passive”) trains on the data of the active agent. Thus, the learning for the passive agent is different than offline RL because it is continuously receiving new data, but that data is not collected based on the agent’s own policy. Across Atari tasks, the passive agent performs significantly worse, unable to solve 3/4 of them (Fig 1 - all Atari envs are further included in the appendix and seem to have the same trend). This suggests similar issues as offline RL also exists in passive learning - it is not just additional data that is important but that the data comes from the agent’s own policy. The rest of the paper is devoted to studying why this happens and evaluating three hypotheses: (1) bootstrapping, (2) data distribution, and (3) function approximation as the major issue that causes the gap.\n\nSection 3.1: this experiment was very interesting! For bootstrapping, the passive agent’s target is replaced by the active agent. The results are still poor for the passive agent, even when the passive agent is using the same exact targets as the active agent. This is quite surprising, as bootstrapping is often identified as one of the difficulties of off-policy RL (eg. the deadly triad). I believe it would be just as informative to perform the same study with the active agent, and I hope this can be included in the rebuttal.\n\nSection 3.2: these sequence of experiments study data distribution. 1. The exploration parameter epsilon being higher reduces the gap between active and passive. This is somewhat unsurprising and the presentation of relative performance makes it a bit hard to understand if the issue is that active performance deteriorates or passive performance improves. 2. Sticky-actions does not change much; I did not really understand the point of this experiment and it seems like it could left out or moved to the appendix. 3. Larger replay buffers improve the passive agent and close the gap substantially to active. One question is whether it also improves the active agent. This is quite interesting though, suggesting that if you are limited to a small buffer, then the data distribution of that buffer being drawn from your own actions is very important. But perhaps with enough data it matters less (this would be consistent with standard deep learning). 4. If the passive agent is forked from an active agent in the middle of training, the performance deteriorates. This is also interesting and surprising, that the wrong data distribution can actually cause you to forget how to solve the task (even though the new data is solving the task!). One thought here is that perhaps bootstrapping is the culprit, and experiments similar to those in section 3.1 might disentangle why this deterioration happens - although this is partially addressed by the “on-policy evaluation” experiment. 5. Instead of freezing the policy, the replay buffer is frozen; there is still deterioration but slightly slower - again surprising. It is especially surprising this works better than the previous experiment, because I would have thought overfitting is a major issue with fixed replay buffer. 6. For forked tandem, using SARSA instead of Q-learning does not solve the deterioration, so ood bootstrapping may not be the culprit. 7. Adding data generated from the passive agent closes the gap, and closes it relatively quickly from 10% making a sizable difference to 50% closing the entire gap.\n\nSection 3.3: Finally, function approximation is evaluated as the issue. First, ADAM and RMSProp is compared; no major difference. Adding more gradient updates per step makes the passive agent worse. Larger width networks seem to close the passive-active gap somewhat, but larger depth networks do the opposite. These findings are interesting but basically inconclusive. In Figure 8, a varying amount of the first k layers is shared between the active and passive agent, and the results are that sharing more layers helps, with sharing 3/5 closing about half of the performance gap and sharing 4/5 closing the entire performance gap (at that point it is linear so if the active one works, the passive one should too). This result, that sharing the early layers improves performance this much, is also surprising to me - the lower layers being trained on the same objective but better/different data significantly changes the outcome (although perhaps this is due to bootstrapping in the end?).\n\nThere is no related work; the paper instead covers prior work in the intro and throughout. This could be okay, but it is missing discussion of some modern offline RL work that studies similar problems:\n[1] Conservative Q-Learning for Offline Reinforcement Learning. Kumar 2020\n[2] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization. Matsushima 2020.\n[3] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. Nair 2020.\n[4] Behavior Regularized Offline Reinforcement Learning. Wu 2019.\n[5] Model-based Offline Policy Optimization. Yu 2020.\n\nThe concluding conjecture - deep RL is unstable on any fixed data distribution - seems too strong, and then vacuous when hedged with - “without explicit regularization or additional inductive bias.” All of offline RL is basically trying to find the right balance of expressivity and regularization, and while this paper provides interesting experiments in this direction towards the data distribution depending on the current policy being important, it doesn’t preclude some form of regularization that alleviates the issue.\n\nGenerally the paper is well organized and very clear and enjoyable to read.\n\nMinor comments\n\nMaybe I missed - in forked tandem, where does the stochasticity come from? Is it from sampling in epsilon greedy exploration, environment stochasticity, or something else?\n\nIt is considered bad style to use a citation as a noun, eg. line 175 “offline results in [3]” instead of writing a method or author name - it makes it harder to read.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Generally, yes. However, the final broader impact paragraph as-is does not add too much value, you could use it to discuss limitations or consequences of understanding the important of active vs. passive in different application domains.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GTIFipJJNGJ", "reviewer": "Reviewer_cED7", "summary": "The paper addresses the difficulties that standard RL methods have in the offline setting. It presents a new analysis technique designed to better understand the causes of the difficulties. The technique, called Tandem RL setting, and forked tandem, extend the setup, introduced by Fujimoto, Meger, and Precup in the BCQ-paper \"Off-Policy Deep Reinforcement Learning without Exploration\" and enable systematic studies of causes for some kind of problems with offline RL. The technique is used for extensive studies, in several benchmarks, and cautious conclusions are drawn.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nIt seems to me a valuable direction to strengthen the analysis techniques for understanding the root cause of the problems. This can be used to support the development of methods.\n\nWeaknesses:\nNone. \nOne would like to see more. In particular, studies on how the individual techniques developed to make offline RL more robust (batch regularization, batch constraining, use of models, use of uncertainty) perform in this analysis.\nBut, of course, this is the task for future work, as it is way beyond the scope of this paper.\n\nOriginality: \nThe conception of the paper, as well as the presented method are original.\n\nQuality: \nVery good.\n\nClarity:\nVery clear.\n\nSignificance:\nVery high. I can well imagine that the presented approach to analysis will be used in the future and help to further increase the performance of offline RL methods.\n\nFurther comments:\nIn the paragraph at the end of the 1st page, starting with \"To mitigate this,\" I think the mention of uncertainty consideration as an approach to make offline RL more robust is missing. This technique is used frequently in model-based offline RL, e.g. [1,2,3], but also in model-free offline RL [4].\n[1] Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, ICML 2018\n[2] MOPO: Model-based Offline Policy Optimization, NeurIPS 2020\n[3] MOReL: Model-Based Offline Reinforcement Learning, NeurIPS 2020\n[4] An Optimistic Perspective on Offline Reinforcement Learning, ICML2020\n\nIn page 5, line 161, the epsilon in \"parameter \\epsilon\" should be bold,  \\boldmath{\\epsilon}\n\nPlease check the bibliography for accidental lower case letters, like „q-learning“\n\nPlease check consistency: In [14] it’s „Hado Hasselt“, in [16] and [38] it’s „Hado Van Hasselt“, in [39] it’s „Hado van Hasselt“, which I believe to be correct.\n\n**After rebuttal** \\\nI have read the other reviews and the feedback from the authors and think that the authors have responded well to the suggestions. I raised the score to 8.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyzes the reason for the poor performance of learning control policies from offline datasets. The main conclusion is that
- Insufficient coverage of suboptimal actions to poor performance when the policy is evaluated is online. 
- This problem is further aggravated by deep networks that extrapolate value function in the regions of the space that are infrequently explored by the agent. 

Three reviewers vote for accepting the paper, while reviewer LigT dissents. The concerns about (i) more seeds; (ii) increased mathematical rigor will definitely make the claims of the paper stronger. However, they are not a ground for rejection. I encourage the authors to incorporate these suggestions. 
Decision: Accept (Poster)