Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Note: the style-formatting of this paper has been heavily tweaked, and so the evaluation should be calibrated for a 9-page paper.

This paper proposes an approach for diverse self-imitation for hard exploration problems.  The idea is leverage recently proposed self-imitation approaches for learning to imitate good trajectories generated by the policy itself.  By encouraging diversity in the pool of trajectories for self-imitation, the idea is to encourage faster learner -- this basic concept is also used in approaches like prioritized experience replay, albeit at the entire trajectory level rather than individual state/action level.  

The authors view this approach as a generalization of Go-Explore, since it does not rely on having a reset mechanism.  However, I think this discussion has a lot of subtle nuances pertaining to the stochasticity of the environment (which the authors acknowledge).  For instance, if the environment is deterministic, then why not just do something like Go-Explore, since state-reset is just memorizing a deterministic action sequence? 

The empirical results are very strong, achieving state-of-the-art results for any approach not reliant on a reset mechanism.  All the primary experiments appear to be for deterministic environments.  The results on stochastic environments (in the Appendix) seem pretty weak (but please correct me if I'm mistaken here).  So one major question is whether Go-Explore is a scientifically appropriate benchmark to compare with for this setting.  

In summary, I'm willing to be convinced that this is an interesting and scientifically novel result.  I have some concerns as expressed above.


**** After Author Response ****
Thanks for the response.  I'm willing to raise my score to weak accept.  

I think the authors did a reasonable job addressing my specific questions.  Some further reflection revealed to me that there is a huge opportunity to scientifically investigate how stochasticity impacts the proposed algorithm.  For instance, one could conduct a systematic study (say of the Apple domain) where one varies the degree of stochasticity and measures how the performance the proposed algorithm changes, perhaps relative to Go-Explore on the purely deterministic version of the environment.  It seems a bit of a cop-out to say that Go-Explore is not applicable, and misses out a huge opportunity for real scientific understanding.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The authors identify and address the problem of sub-optimal and myopic behaviors of self-imitation learning in environments with sparse rewards. The authors propose DTSIL to learn a trajectory-conditioned policy to imitate diverse trajectories from the agent’s own past experience. Unlike other self-imitation learning methods, the proposed method not only leverages sub-trajectories with high rewards, but lower-reward trajectories to encourage agent exploration diversity. The authors claim the proposed method to be more likely to find a global optimal solution. 

Overall, this paper is well-written with comprehensive experimental results. The proposed trajectory-conditioned policy sounds, since rewarded trajectory carries significant information of the goal in the exploration problem. Extensive experimental results demonstrated the effectiveness of the proposed DTSIL. However, I have a few concerns below, that prevent me from giving a direct acceptance. 

1. The proposed DTSIL changes the original MDP with sparse reward to an MDP with denser reward, which allows the training process to explore more in the “space” closer to the collected high reward trajectories. Such “exploration” sounds promising. However, it would be nice to compare it with traditional reinforcement learning (e.g., with \epsilon-greedy policy for random exploration)?

2. In appendix D, the authors discussed what the parameter \delta_t controls, however, it is unclear how \delta_t should be chosen in implementation. The authors did not explain how \delta_t was selected in their experiments. Choosing the right \delta_t may be hard, but it would be nice to introduce what “heuristics” the authors used and suggest to readers. 

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The paper addresses the challenge of hard exploration tasks. The approach taken is to apply self-imitation to a diverse selection of trajectories from past experience -- practice re-doing the strangest things you've ever done. This is claimed to drive more efficient exploration in sparse-reward problems, leading to SOTA results for Montezuma's Revenge without certain common aides.

The approach is incompletely motivated. Why trajectory-conditioned policy over just goal-conditioned policy? The note in the related work section doesn't paint a clear enough picture. The trajectory buffer management strategy feels complex. Why this use strategy specifically? Could a simpler design be ruled out? In 2019 (post Go-Explore), it's not clear Montezuma's revenge poses a significant exploration challenge -- exploration doesn't even need to be interleaved with learning. Why are these three the right domains to show off these techniques?

This reviewer moves to reject the paper primarily for not balancing the high complexity of the solution to the lower difficulty of the problem. Pure-exploration algorithms (Go-Explore), not burdened by interleaving policy learning, achieve far superior scores. If the authors want to escape the shadow of this kind of technique which cheats by some framings of RL, more appropriate demonstration environments must be selected.
review_title: Official Blind Review #2

Meta-review: This paper addresses the problem of exploration in challenging RL environments using self-imitation learning. The idea behind the proposed approach is for the agent to imitate a diverse set of its own past trajectories. To achieve this, the authors introduce a policy conditioned on trajectories. The proposed approach is evaluated on various domains including Atari Montezuma's Revenge and MuJoCo.

Given that the evaluation is purely empirical, the major concern is in the design of experiments. The amount of stochasticity induced by the random initial state alone does not lead to convincing results regarding the performance of the proposed approach compared with baselines (e.g. Go-Explore). With such simple stochasticity, it is not clear why one could not use a model to recover from it and then rely on an existing technique like Go-Explore. Although this paper tackles an important problem (hard-exploration RL tasks), all reviewers agreed that this limitation is crucial and I therefore recommend to reject this paper.
Decision: Reject