Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper provides detailed empirical analysis to study the challenge of offline reinforcement learning. The authors propose to use the Tandem RL setup as the analytic tool, where one active agent performs usual online training loop, while the passive agent can only learn from the data generated by the action learner. The results provide some insights for revealing the difficulty of offline RL. 

main_review: Contributions:

The paper is very well written. The definitions and experiment setups are clearly defined. I really enjoy reading this paper. 

The main message of this paper is that, as function approximators for value functions, deep neural networks are extrapolating in an uncontrolled fashion especially under insufficient coverage of the fixed data set, and as such blow up errors in an iterative training scenario. I think this is well expected as even in the linear case such extrapolation errors occur as suggested by previous theoretical works. 

It is really good to see that the authors actually try to provide detailed analysis to reveal the hardness of offline RL. Most previous works in deep offline RL literatures just try to demonstrate their proposed algorithms are good in some cases, but it’s rarely to see offline RL algorithm can fail very easily, and what are the major factors that cause the trouble. 

Weakness:

I have the following comments and questions that I hope the authors to address:

What does the over-generalization exactly mean in the paper? Does that mean over estimate the unseen data or something even stronger? Is it possible to visualize such over-generalization?

I think the data distribution assumption (Hypothesis 2) as well as the supporting experiments can be strengthened such that it can distinguish between "enough data coverage"  and "enough on-current-policy data coverage". It has been shown that the "enough data coverage" is not enough to guarantee efficient offline RL (e.g. see [2,3]). Maybe the experiments shown in the paper connects the failure of passive learning to the later one?  

Almost all experiments are in the policy optimization problem. I am wondering if the hardness of offline RL already exists in the simpler policy evaluation problem since the Hypothesis 2 and 3 seem also exist there (although I understand it's hard to design experiments for policy evaluation in this tandem setups). 

There is also some recent theoretical paper [1] trying to reveal the difficulty of offline RL. But I guess the major factor there is the estimation error that comes from the insufficient coverage of the data (as they consider the tabular setting so there is no extrapolation error while the problem still exists). I am wondering if the authors distinguished the estimation error and extrapolation error in this paper? 

[1]  Xiao, C., Lee, I., Dai, B., Schuurmans, D. and Szepesvari, C., 2021. On the Sample Complexity of Batch Reinforcement Learning with Policy-Induced Data. arXiv preprint arXiv:2106.09973.

[2] Wang, R., Foster, D.P. and Kakade, S.M., 2020. What are the Statistical Limits of Offline RL with Linear Function Approximation?. arXiv preprint arXiv:2010.11895.

[3] Wang, R., Wu, Y., Salakhutdinov, R. and Kakade, S.M., 2021. Instabilities of offline rl with pre-trained neural representation. arXiv preprint arXiv:2103.04947.
limitations_and_societal_impact: No societal issue has been found.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the situation of passive reinforcement learning, simulating the famous Held and Hein twin cats paper where an active agent collects data to learn while the passive agent experiences the same data (but does not choose actions). The result, running Double-DQN on Atari, is similar - that the passive learner is often much worse. The paper then attempts to study why by evaluating three hypotheses: bootstrapping, data distribution, and function approximation, through a sequence of experiments. The conclusion is mixed, but mostly supports that data distribution is the key problem, with good evidence that bootstrapping is relatively less important than previously emphasized. Overall, the paper is a clever and thorough study of passive learning beyond the existing literature on offline RL.
main_review: The main idea of the paper is to study “tandem” reinforcement learning: one agent (“active”) runs online RL as usual, another agent (“passive”) trains on the data of the active agent. Thus, the learning for the passive agent is different than offline RL because it is continuously receiving new data, but that data is not collected based on the agent’s own policy. Across Atari tasks, the passive agent performs significantly worse, unable to solve 3/4 of them (Fig 1 - all Atari envs are further included in the appendix and seem to have the same trend). This suggests similar issues as offline RL also exists in passive learning - it is not just additional data that is important but that the data comes from the agent’s own policy. The rest of the paper is devoted to studying why this happens and evaluating three hypotheses: (1) bootstrapping, (2) data distribution, and (3) function approximation as the major issue that causes the gap.

Section 3.1: this experiment was very interesting! For bootstrapping, the passive agent’s target is replaced by the active agent. The results are still poor for the passive agent, even when the passive agent is using the same exact targets as the active agent. This is quite surprising, as bootstrapping is often identified as one of the difficulties of off-policy RL (eg. the deadly triad). I believe it would be just as informative to perform the same study with the active agent, and I hope this can be included in the rebuttal.

Section 3.2: these sequence of experiments study data distribution. 1. The exploration parameter epsilon being higher reduces the gap between active and passive. This is somewhat unsurprising and the presentation of relative performance makes it a bit hard to understand if the issue is that active performance deteriorates or passive performance improves. 2. Sticky-actions does not change much; I did not really understand the point of this experiment and it seems like it could left out or moved to the appendix. 3. Larger replay buffers improve the passive agent and close the gap substantially to active. One question is whether it also improves the active agent. This is quite interesting though, suggesting that if you are limited to a small buffer, then the data distribution of that buffer being drawn from your own actions is very important. But perhaps with enough data it matters less (this would be consistent with standard deep learning). 4. If the passive agent is forked from an active agent in the middle of training, the performance deteriorates. This is also interesting and surprising, that the wrong data distribution can actually cause you to forget how to solve the task (even though the new data is solving the task!). One thought here is that perhaps bootstrapping is the culprit, and experiments similar to those in section 3.1 might disentangle why this deterioration happens - although this is partially addressed by the “on-policy evaluation” experiment. 5. Instead of freezing the policy, the replay buffer is frozen; there is still deterioration but slightly slower - again surprising. It is especially surprising this works better than the previous experiment, because I would have thought overfitting is a major issue with fixed replay buffer. 6. For forked tandem, using SARSA instead of Q-learning does not solve the deterioration, so ood bootstrapping may not be the culprit. 7. Adding data generated from the passive agent closes the gap, and closes it relatively quickly from 10% making a sizable difference to 50% closing the entire gap.

Section 3.3: Finally, function approximation is evaluated as the issue. First, ADAM and RMSProp is compared; no major difference. Adding more gradient updates per step makes the passive agent worse. Larger width networks seem to close the passive-active gap somewhat, but larger depth networks do the opposite. These findings are interesting but basically inconclusive. In Figure 8, a varying amount of the first k layers is shared between the active and passive agent, and the results are that sharing more layers helps, with sharing 3/5 closing about half of the performance gap and sharing 4/5 closing the entire performance gap (at that point it is linear so if the active one works, the passive one should too). This result, that sharing the early layers improves performance this much, is also surprising to me - the lower layers being trained on the same objective but better/different data significantly changes the outcome (although perhaps this is due to bootstrapping in the end?).

There is no related work; the paper instead covers prior work in the intro and throughout. This could be okay, but it is missing discussion of some modern offline RL work that studies similar problems:
[1] Conservative Q-Learning for Offline Reinforcement Learning. Kumar 2020
[2] Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization. Matsushima 2020.
[3] AWAC: Accelerating Online Reinforcement Learning with Offline Datasets. Nair 2020.
[4] Behavior Regularized Offline Reinforcement Learning. Wu 2019.
[5] Model-based Offline Policy Optimization. Yu 2020.

The concluding conjecture - deep RL is unstable on any fixed data distribution - seems too strong, and then vacuous when hedged with - “without explicit regularization or additional inductive bias.” All of offline RL is basically trying to find the right balance of expressivity and regularization, and while this paper provides interesting experiments in this direction towards the data distribution depending on the current policy being important, it doesn’t preclude some form of regularization that alleviates the issue.

Generally the paper is well organized and very clear and enjoyable to read.

Minor comments

Maybe I missed - in forked tandem, where does the stochasticity come from? Is it from sampling in epsilon greedy exploration, environment stochasticity, or something else?

It is considered bad style to use a citation as a noun, eg. line 175 “offline results in [3]” instead of writing a method or author name - it makes it harder to read.
limitations_and_societal_impact: Generally, yes. However, the final broader impact paragraph as-is does not add too much value, you could use it to discuss limitations or consequences of understanding the important of active vs. passive in different application domains.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper addresses the difficulties that standard RL methods have in the offline setting. It presents a new analysis technique designed to better understand the causes of the difficulties. The technique, called Tandem RL setting, and forked tandem, extend the setup, introduced by Fujimoto, Meger, and Precup in the BCQ-paper "Off-Policy Deep Reinforcement Learning without Exploration" and enable systematic studies of causes for some kind of problems with offline RL. The technique is used for extensive studies, in several benchmarks, and cautious conclusions are drawn.
main_review: Strengths:
It seems to me a valuable direction to strengthen the analysis techniques for understanding the root cause of the problems. This can be used to support the development of methods.

Weaknesses:
None. 
One would like to see more. In particular, studies on how the individual techniques developed to make offline RL more robust (batch regularization, batch constraining, use of models, use of uncertainty) perform in this analysis.
But, of course, this is the task for future work, as it is way beyond the scope of this paper.

Originality: 
The conception of the paper, as well as the presented method are original.

Quality: 
Very good.

Clarity:
Very clear.

Significance:
Very high. I can well imagine that the presented approach to analysis will be used in the future and help to further increase the performance of offline RL methods.

Further comments:
In the paragraph at the end of the 1st page, starting with "To mitigate this," I think the mention of uncertainty consideration as an approach to make offline RL more robust is missing. This technique is used frequently in model-based offline RL, e.g. [1,2,3], but also in model-free offline RL [4].
[1] Decomposition of uncertainty in Bayesian deep learning for efficient and risk-sensitive learning, ICML 2018
[2] MOPO: Model-based Offline Policy Optimization, NeurIPS 2020
[3] MOReL: Model-Based Offline Reinforcement Learning, NeurIPS 2020
[4] An Optimistic Perspective on Offline Reinforcement Learning, ICML2020

In page 5, line 161, the epsilon in "parameter \epsilon" should be bold,  \boldmath{\epsilon}

Please check the bibliography for accidental lower case letters, like „q-learning“

Please check consistency: In [14] it’s „Hado Hasselt“, in [16] and [38] it’s „Hado Van Hasselt“, in [39] it’s „Hado van Hasselt“, which I believe to be correct.

**After rebuttal** \
I have read the other reviews and the feedback from the authors and think that the authors have responded well to the suggestions. I raised the score to 8.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presents an empirical study on offline deep reinforcement learning in atari environments. The authors use tandem learning -- where an agent B learns from experience collected from another agent A. The authors consider 3 hypotheses as to why agent B tends to perform worse in this setting, and provide many ablation-type experiments to support/reject these hypotheses.
main_review: Main comments
---------------

Overall the paper is well written and relatively easy to follow. Offline RL is an important domain, and the experimental setup enables the authors to probe many important questions. The authors specify the hypotheses they are testing and interpret what the experiments say about these. This is a good step towards good empirical practices. 

Compared to previous work, there are some novel and interesting experiments: 1) the authors consider using different bootstrap targets for the tandem learning, and shows that the tandem-gap persists even if using the bootstrap targets of "online" agent. 2) The authors study the effects of \epsilon and replay buffer size on the tandem gap. The question here is interesting, but the results vary among environments which makes it hard to draw conclusions. 3) A method of "forking" a tandem learning, which demonstrates that the tandem gap appears quickly in this case. 

There are two main issues with the paper 1) the novelty of the setup is not clear and 2) the clarity of the hypotheses and what the experiments say could be improved. 


**Novelty**

The experimental setup of "tandem learning" seems to already have been proposed by [13](Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration.). Indeed, the authors state that "We begin by reproducing the striking observation in [13]". However, in the abstract, the authors state that "we propose the “tandem learning” experimental paradigm". Could the authors clarify their contribution of proposing tandem learning vis-a-vis [13]?


**Clarity**

It would be preferable to state the 3 hypotheses of Section 3 in a mathematical language to make them more precise. It is unclear what is meant by e.g. "the stationarity of the data distribution [...] play only a small role.". What is a "small" role? Since the hypotheses are stated in a relatively "hand-wavy" way, it becomes hard to see what the experiments say about them. Since the hypotheses drive the experiments, it should be crystal clear what they are and how they differ.


In general, only graphs are presented and used to draw conclusions. When making precise statements, it would be preferable to use some specific statistics and then perform statistical significance testing to make sure the results are not random noise. E.g. the authors state that "As can be seen in Fig. 3 (top), relative passive performance is indeed substantially improved".  Fig 3 shows improvements for some games, but differences for seaquest or pong might as well be due to chance. The author states "Unless stated otherwise, all results show averages over at least 5 seeds, with confidence intervals indicating variation over seeds." This appears to be the ONLY place in the text where seeds are mentions. What experiments use more than 5 seeds? And if there are any, how many seeds are used? At any rate, 5 seeds are very low for making statistically significant observations. 




Minor Comments
-----------

I doubt most reviewers are familiar with "Held & Hein’s classic 1963 experiment" as stated in the abstract. Referring to it so early will likely confuse most readers.

Many figures lack labeled axes, see e.g. Fig 3,4, and 5.

Is CUDA set to deterministic mode? How are the seeds selected?

Double-DQN, the used architecture, is a relatively simple model that isn't used in practice. I think it would have been better to use rainbow, and simply use hyperparameters from reference implementations. By not changing such parameters, the authors can keep the "moving parts" to a minimum.

Some experiments seem to use only 4 environments. Whenever results for more environments are available, I think they should be presented in the main paper.

## post rebuttal comment

I have read the rebuttal and thank the authors for their response. 

My two main objections to the paper remains. Firstly, I don't think the novelty is particularly high. The authors state that a novel aspect is "explicit framing of the Tandem as an analytic tool". The tool has clearly already been used implicitly as an analytic tool in a previous paper. Framing a known method as an explicit analytic tool doesn't count as novel to me. Secondly, I think the clarity is lacking due to the lack of statistical rigor.  The authors state that "given the inner complexity of any modern deep RL algorithm, we have little hope of being able to pin these down with full mathematical rigour". Full rigor might be unattainable, nonetheless I think the paper could be improved significantly along this direction. I believe that the paper would be better served by using more seeds on fewer environments instead of using a few seeds across the whole 57 task Atari suite. However, the paper also has many strong points and to me this is a borderline paper. I am clearly in the minority regarding these opinions, and given the other scores I expect to see the paper in the final program and I'm sure it would bring about many interesting discussions despite the issues given above.






limitations_and_societal_impact: n/a
review_title: null

Meta-review: The paper analyzes the reason for the poor performance of learning control policies from offline datasets. The main conclusion is that
- Insufficient coverage of suboptimal actions to poor performance when the policy is evaluated is online. 
- This problem is further aggravated by deep networks that extrapolate value function in the regions of the space that are infrequently explored by the agent. 

Three reviewers vote for accepting the paper, while reviewer LigT dissents. The concerns about (i) more seeds; (ii) increased mathematical rigor will definitely make the claims of the paper stronger. However, they are not a ground for rejection. I encourage the authors to incorporate these suggestions. 
Decision: Accept (Poster)