Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: #rebuttal response
Thanks for your response and the additional experiments. I do think it is a very interesting paper but I keep my score. First, the improvement of their model is not very significant both in Table 1 and the standard RL setting. Second, I am still not clear about how off-policy learning benefits the study on standard RL. Since exploration and exploitation interact with each other, disentanglement between them cannot reflect the original difficulty of RL problems. Different exploration strategies or balancing methods between exploration and exploitation will produce different data. I guess this is why the data generation process is a common concern raised by both Reviewers 1 and 2. I think the response of the authors is not enough to clarify this. In addition, I cannot find Figures 4a and 4b. I think this paper is not ready for publication right now but I encourage the authors to improve this work and resubmit it in the future.


#review

This paper collects a replay dataset of Atari to propose a benchmark for offline (batch) reinforcement learning. Experiments on this dataset show that offline RL algorithms can outperform online DQN. Then, this paper presents a new off-policy RL algorithm, which uses random convex combinations of multiple networks as the Q-value estimator. The experimental results show that it outperforms distributional RL algorithms (i.e., C51 and QR-DQN) in the offline setting and performs comparably in the online setting. 

The idea of randomly combining Q networks is interesting and paper is well written to demonstrate their key ideas. However, I have some concerns.

Why are off-line RL algorithms necessary? I am not sure this paper is relevant to the community of standard RL. This paper does not show the usage of their method to a standard RL agent. I am happy to change my score if the authors can clarify the motivation of studying the offline performance of conventional RL algorithms and clearly show the improvement over baseline algorithms in a standard RL setting rather than only show insights for potential usage in online RL.

In my opinion, the results are not significant to support their claims. Such a small gap (4.9% as shown in Table 1) over baseline models may result from considerable hyper-parameter tunning. In addition, the figures and tables in this paper do not show deviations and confidence intervals.

Since REM has similar architectures with attention networks, it will be better to include an attention-based Q network as a baseline model.

In the right sub-table of Table 1, I wonder why the performance of online QR-DQN in this paper is much lower than that reported in the original papers (media is about 200%).


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: *Summary*

Paper tackles an important issue, which is benchmarking off-policy deep RL algorithms. The motivation is important, because 1) assessing the performance of off-policy RL algorithm is hard due to the mixed effect of exploration and exploitation while collecting data; hence the paper suggest to fix the data collection process by a standard algorithm (DQN) and then compare the off-policy methods. 2) A fixed dataset will help the reproducibility of the results. 

Additionally the paper assess the performance of some off-policy RL methods (mainly DQN and QR-DQN) and showed that given offline data, these algorithms can obtain a superhuman performance. 

Besides the mentioned contribution, authors strived for a simple algorithm (easy to implement), and suggests a new simple algorithm, Random Ensemble Mixture (REM) by training multiple Q-functions and randomly sampling a subset for ensemble estimate. The mina motivation come from supervised learning literature. 

*Decision*

I enjoyed reading the paper and I think authors tackled a very important problem. Although a very important step, I vote for a weak reject for this paper as I believe the contribution of the current paper is limited. I made my decision mainly based on the following points:

[Limited Task Diversity]: I agree with the authors that having a fix offline data for testing different algorithms is important; however the suggestion (DQN logged data on Atari2600) is a very limited dataset, in terms of 1) task diversity 2) Data collection strategy. 
For a benchmark, I would like the dataset to have different strategies for data collection, so that the effect of data collection can be stripped off. In the current assessment, we compare the performance of algorithms condition on the DQN e-greedy data collection procedure. (For example, the right claim to make is: QR-DQN showed a better performance condition on the current data generation process). And it is not clear to me the same observation will hold given a different data generation process (for example a random policy, or a human playing data, or …)
Also, I think only focusing on Atari tasks is limited in nature, as many of these games share a similar underlying mechanism. 


[Overfitting] I believe having a fixed dataset for testing off-policy RL is great, however the current suggestion is very prone to overfitting. I can foresee that if the community start to use this benchmark as a test case, soon we will overfit to this task, in a sense that I try algorithm X on offline data, test on Atari see the results, tune algorithm X test on Atari again, … This way I finally have an algorithm that only learned from offline data, but is performing great on online Atari. But I am overfitting!


[Simplicity Trade-Off] Authors focused on getting a simple algorithm, but to me it is unclear why should we optimize for simple algorithm, and my impression of simple as authors have in mind is a “easy to implement” algorithm. 
Simplicity is great, but usually I believe we seek a simple algorithm to be able to get more insight into the theory and build more intuition. However, in this work there is a lot emphasis on simple algorithm, but no extra significant insight into off-policy learning has been gained. Additionally, I found the suggested algorithm ad-hoc and of limited contribution. 


Minor Comments/ Question that did not significantly contribute to my decision: 

[Valid Q-function]: authors mentioned a convex combination of Q-values are a valid Q-value, but never defined a “valid” Q-value. Do authors mean, if Q1 is generated by \pi_1 and Q2 by \pi_2 then convex combination of Q1 and Q2, Q^* can be also generated by some policy \pi^*? 
If so, this needs clarification. 
And if so, I’m not sure why this is important, since in Q-learning the Q-values that we obtain before convergence might be “invalid” (by the definition above, so that no real policy can generate those values exactly), so what is the point of paying attention to a valid Q values in the first place?

*Improvement*

I think some clear improvement avenues for the future are
1. More diverse set of tasks: I would like to see some non-Atari tasks included in the dataset.
2. More diverse data collection strategy: Example can be different exploration strategies than e-greedy. Or even random exploration, or maybe data of human players. Since we don’t want to have an algorithm that is good only for state distribution induced by e-greedy exploration.
3. More diverse task will help with overfitting by itself. But a good benchmark needs a test case that cannot be exploited. 
4. More insight into why REM is outperforming QR-DQN or other algorithms. Currently I am not convinced that we understand why REM is outperforming, or if it’s only because we designed it to be good for this specific task and dataset? 


Generally, I believe authors made a very important step, and I really enjoyed reading the paper; however, I think the current contribution is not sufficient to merit a publication. This makes a good workshop paper at this point.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper investigated the offline settings of deep Q-networks, and provided relevant comparison among different agents. The authors also proposed two DQN variants by using ensembles, and showed the improvement (mainly from the random ensemble mixture) on Atari games, when compared with baselines. 

In general, this paper explored an important problem, i.e., offline vs. online settings in DQNs. The experimental comparison also looked interesting, as it showed the empirical evidence on the influence of such settings. A lot of the experiments were performed to support the claims, which I really appreciate. On the other hand, it's a bit unclear to me whether offline should be preferred over its online counterpart, given the results presented and potential computational cost. For the proposed REM method, further explanation isolating from the offline vs. online setting could be helpful to understand why it works. Furthermore, some settings in the experiments need to be clarified. If the authors can address my concern, I would be willing to raise my score.

1. To my knowledge, using the logged experience to train an offline DQN agent is an interesting exploration. This approach can also achieve competitive performance, when compared with the online agent. My concern is whether the authors had considered the cost due to this offline setting, as additional time needs to be spent on training another agent to obtain such logged data, which is not required in the online methods. Moreover, Table 1 suggests that the online agents usually outperformed their offline counterparts, from which I am somewhat in doubt about whether the offline approaches should be preferred.
2. I am also wondering how the performance depends on the agent used to generate the logged data. In the current version, it is from the Nature DQN, but what will happen if we use a different (can be either better or worse) agent? For example, how about using C51 to generate the logged data?  I hope the authors can provide more evidence and analysis on why the offline settings should work. 
3. When training the agents with the logged experience, how did you handle the new data generated? I understand the offline setting may not consider this issue, but am just thinking if such data can be better used.  
4. The ensemble based DQNs proposed in Section 4 are very interesting, which can even be a separate contribution, as they do not depend on the online or offline settings. From this perspective, I am a bit curious if the authors have more explanation on the benefits of such methods, e.g., reducing the overestimation bias and gradient noise, as shown in the averaged-DQN and the following softmax-DQN work, 
Song et al., Revisiting the Softmax Bellman Operator: New Benefits and New Perspective, ICML 2019, 
which the authors may refer to in their paper.
5. For the curves in Figures 1 and 4, how about the corresponding running time? Such information could be helpful for others trying to reproduce the results.
review_title: Official Blind Review #1

Meta-review: All the reviewers recommend rejecting the submission. There is no basis for acceptance.
Decision: Reject