Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: In this work, R2D3 (Recurrent Replay Distributed DQN from Demonstration), which combines R2D2 [1] with imitation learning (IL), is proposed. Similar to the existing works on “reinforcement learning (RL) with demonstration” such as DQfD, DDPGfD, policy optimization with demonstration (POfD) [2], hard exploration conditions (sparse reward, partial observability, high variance in initial states) are assumed, which is difficult to achieve good performance with RL without demonstration in general. Eight tasks in such conditions were devised and used to test the performance of R2D3.

I like the fact that the authors of this work have chosen quite challenging scenarios, but I think the novelty of this submission is a bit weak to be accepted to the conference. I believe “RL with demonstration” becomes meaningful when it beats both RL and IL in some reasonable setting. For example, POfD [2] assumes sparse-reward tasks with *imperfect* demonstrations, which is difficult to achieve good performance by using RL or IL. From such a perspective, I have the following concerns:

- Imitation learning baselines: There has been recent advancement in imitation learning. In the submission, it was mentioned that “GAIL has never been successfully applied to complex partially observable environments that require memory”, but there’s [3] that successfully uses GAIL in such a setting. Also, off-policy imitation learning such as DAC [4] is shown to be highly sample-efficient compared to GAIL in MuJoCo domain. However, the submission only considers behavioral cloning (BC) (which shows poor performance at unseen states due to the covariate shift problem) as a baseline among imitation learning method

- Reinforcement learning baselines: The submission adopted R2D2 as an RL baseline, and it seems to me that the R2D2 agent starts from random initialization. For a fair comparison, however, I believe R2D2 with BC (or Batch RL) initialization should be considered.

In addition to the above concerns, it seems to me that most of the features in R2D3 simply combines those in either DQfD or R2D2, and I couldn’t find out its own algorithmic novelty except “demo ratio” parameter. 

I’ll increase my score if I made wrong comments or misunderstood the contribution.

References
[1] Kapturowski, Ostrovski, Quan, Munos. and Dabney, “Recurrent experience replay in distributed reinforcement learning,” ICLR 2019.
[2] Kang, Jie, Feng, “Policy optimization with demonstrations,” ICML 2018
[3] Gangwani, Lehman, Liu, Peng, “Learning Belief Representations for Imitation Learning in POMDPs,” UAI 2019
[4] Kostrikov, Agrawal, Dwibedi, Levine, Jonathan, Tompson, “Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning,” ICLR 2019
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: The paper addresses the problem of exploiting human demonstrations in hard exploration (RL) problems. A new set of challenge tasks is introduced that destroys the performance of very strong baseline systems while highlighting the strength of the new system.

The approach (rarely but consistently training on separately prioritized human experience replays) is well motivated by the shortcomings of past agents (either in overfitting the demonstrated solution or only working in environments with not-too-hard exploration challenges). Where work by others have overspecialized on specific challenge environments (e.g. Montezuma's Revenge with weak stochasticity and observability challenges), this work intentionally dives into difficult territory.

This reviewer moves to accept this top-quality RL paper. The new agent, R2D3, is the primary contribution in combining and outperforming previous SOTA agents. These eight new environments are minor contributions with limited potential for impact on the field, but still make an independently positive contribution.

Questions for authors:
- What would be the present-day approximate retail cost for reproducing the experiments in this paper?
- At the action-rate experienced by the human demonstrators (30fps?), how much wall-clock time represented by 40B actor steps? (40 years?) Is this "making efficient use"?
- Does having highly variable initial conditions really force generalization over environmental configurations or is this wishful thinking / mysticism? To make a direct claim about this, the authors should consider an experimental design where certain classes of initial conditions (e.g. starting on the left side of the map) are withheld during training and evaluated only during testing.
- The finding of small demo ratios as being stronger is exciting, but this result seems to be tied to the specific quantity and quality of demonstrations gathered. Could a more general picture of the role of demonstrations be had by ablating the diversity of representations? The 100 demos in the full case might be degraded to 50, 25, 10, etc while holding the demo ratio fixed. This might effectively vary the weight that demonstrations take in the optimization independently of how often distinct demos are actually seen.
- Can these hard-eight scenarios be parametrically scaled up and down in terms of their exploration effort (possibly by just changing the action granularity / movement speed)? With performance on the new benchmark almost saturated in the first paper based on it, there isn't much room to grow here. In the same way that Montezuma's Revenge was found by scanning the culturally-impactful library of Atari games, perhaps more appropriate and lasting challenges can be found by looking one or more generations forward in the history of commercial console games. Can we play Star Fox? What about SimCity?


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary
-------------
The authors propose R2D3, and algorithm for learning from demonstrations in partially-observable environments with sparse rewards. The algorithm combines DQfD with recurrent networks to both leverage expert demonstrations and handle partial observability. Furthermore, the authors propose a suite of eight challenging tasks on which the proposed method is tested and compared to relevant baselines.

Comments
--------

Scaling RL agents to high-dimensional partially-observable domains with sparse rewards is a fundamental open problem and this work provides a nice contribution towards its solution. The paper is well-written and easy to read. The proposed methodology seems to be a simple combination of existing algorithms and (apologies if I am wrong) I did not see any particular challenge in its design. On the other hand, the hard-eight task suite is interesting and, if released, could be used as a benchmark by the whole community. The experiments seem quite convincing in proving the potential of the proposed method. Some comments/questions follow.

1. Section 2 presents the algorithm from a very high-level perspective. If space in the final version allows it, I would also suggest adding a more detailed pseudo-code to the main text, so that even a reader who is not completely familiar with the works this method builds upon could better understand and possibly implement the method.

2. Since the authors compare to behavioral cloning to prove the benefits over simple imitation methods, why not comparing to stronger baselines such as [1] or [2]?

3. The demo-ratio seems to be the key parameter to make this approach work (and the performance is proven very sensitive to its value). Instead of keeping it fixed across the entire learning process, have you tried to start with a high value and then decay according to a proper schedule? Intuitively, I would expect the benefits of expert demonstrations to be more valuable during the first learning episodes (where they make the agent explore better) and less during the successive phases (where the policy gets closer and closer to optimal).

4. The way recurrent states are handled with zero-initialization is probably one of the limitations and seems to play an important role in some experiments. Have you tried, at least in simpler domains, to replay whole episodes and see whether that helps?

[1] Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. In Advances in neural information processing systems (pp. 4565-4573).
[2] Finn, C., Levine, S., & Abbeel, P. (2016, June). Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning (pp. 49-58).
review_title: Official Blind Review #2

Meta-review: This paper tackles hard-exploration RL problems using learning from demonstrations. The idea is to combine the existing R2D2 algorithms with imitation learning from human demonstrations. Experiments are conducted on a new set of challenging tasks, highlighting limitations of strong current baseline while highlighting the strength of the proposed approach.

The contribution is two-folds: the proposed algorithm which clear outperforms previous SOTA agents and the set of benchmarks. All reviewers being positive about this paper, I therefore recommend acceptance.
Decision: Accept (Poster)