Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider the Goal-Conditioned continuous control RL task, which is difficult due to reward sparsity. The authors propose a method for training deep RL agents in a more sample efficient manner through self imitation on policy traces that have been "distilled", i.e., randomly perturbed and argmax'd according to some metric like "First Hit Time", then using hindsight experience replay to relabel traces (handling reward sparsity). This improves of over the previous state of the art which did not use the policy distillation step, simply relabeling with the most recent policy, prone to error propagation. The authors demonstrate that their policy distillation step improves the state of the art on 3 RL baseline settings, and also give theoretical backing to their approach and conditions for convergence. 

main_review: - Exposition is clear and the problem is interesting. Authors clearly describe a number of specific concepts necessary to understand exactly what their contribution is, especially as compared to the state of the art.
- Authors give convincing theoretical backing to their approach, as well as conditions under which their key results hold.
- Authors clearly demonstrate an improvement over the state of the art in 3 experiment domains, and compare against several other baselines. Authors also give ablation analysis.

Weaknesses:
- My only major concern is that the state of the art (GCSL) (Ghosh et al. 2019) somehow performs essentially the worst of any baseline in all their experimental domains. The authors state that this is due to the difficulty of the domains -- perhaps then, the authors should include some experimental domains where GCSL performs at least better than trivially poor, so that it does not give the feeling that these set of experimental domains were somehow biased toward the proposed method. Showing the relative performance of the authors' method vs. SOTA in a domain where SOTA can effectively learn would also help more clearly describe exactly where the authors' contribution is having an impact as it relates to the features of different environments or training parameters.


---------------------
Update: I have read the author response. They have adequately addressed my primary concern, so I maintain my score. I recommend that the authors move some of the experiments from appendix E to the main text, since it is more convincing that domains were balanced for the competitor as well as the proposed method.
summary_of_the_review: I am rating accept as the writing is clear, motivations and contributions are clear, theoretical results are convincing, and experimental results are mostly good -- the only issue is that the state of the art somehow performs very poorly in the given domains. I hope the authors can either explain this in greater detail or include a domain in the final version where the state of the art is at least able to learn something.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose an approach for goal-conditioned reinforcement learning via self-imitation. The paper first outlines a meta-algorithm which defines conditions under which an instantiation of the algorithm constitutes a valid policy improvement operator. Following this, the authors propose a concrete instantiation of the algorithm which perturbs the policies parameters, selects policies which decrease the hitting time on randomly chosen goals and then distills the selected behavior into a new policy 

main_review: The paper is suffering from a large number of grammatical as well as spelling mistakes that make it very difficult to follow. This goes beyond inelegant use of language and includes puzzling mistakes in technical terms such as “discounted factor” or “total variance”. Based on this alone, the paper is not fit for publication in its current state.

Overall, it is unclear what insight is to be gained from this paper. The proposed concrete implementation is similar to an evolution strategy minimizing the hitting time and as such is broadly sensible; however, it is unclear in which way the proposed method has an advantage over existing approaches such as HER. I find the experimental evaluation to be unconvincing: first, HER can be significantly improved by using a more capable reinforcement learning algorithm and the sample-efficiency gains shown in the plots may be due to this alone. Similarly, it is unclear whether the method is still improving or converging on FetchSlide; final performance on this task should be significantly higher. 

summary_of_the_review: The paper is suffering from a large number of grammatical as well as spelling mistakes that make it very difficult to follow. Based on this, the paper is not fit for publication in its current state.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a self-imitation learning method for goal-conditioned continuous control tasks. The key idea is to identify a $\delta$-distilled policy that performs better than the current policy. Then a mixture policy is generated if a $\delta$-distilled policy is identified. Finally, the parameters are updated with behavior cloning to the relabeled data. Experiments on three goal-oriented tasks demonstrate the efficiency of the proposed method.
main_review: Strength:
1. The method is simple yet effective.
2. Theoretical analysis of local convergence and policy improvement is provided.
3. The paper is well written.

Weaknesses:
1. The SELECT requires resetting the environment to a given previous state, which can only be achieved in simulator but not real-world.
2. The overall learning process is similar to Go-Explore. But Go-Explore is not compared.
3. SPLID w/o SELECT does not perform well, which suggests that the method highly relies on the ability to reset the simulator.
4. It is unclear to me why it is called "meta-algorithm".
5. The experiments are weak. The role of $\delta$-distilled policy is unclear. Perhaps some case studies on the identified $\delta$-distilled policy could facilitate the understanding of the algorithm.
6. It is unclear how the algorithm improves the quality of "the timesteps taken to achieve the goal".

summary_of_the_review: The method is simple yet effective. However, it heavily relies on the ability of reseting the environment to a previous state, which is unfair to the baselines. The experiments also need improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper expands on prior works in self-imitative goal-conditioned RL by proposing to selectively filter the target policy which is used during the behavioral cloning phase. By ensuring that the target policy reaches goals more optimally than the behavior policy, the paper shows that under certain conditions, performance improvement can be guaranteed. The algorithm first collects trajectories under some exploration policy, then for fragments of these trajectories, the algorithm evaluates the current policy (starting in the first state in the fragment) to see if the fragment represents an improvement on the agent's ability to reach the state. If so, it is added to the replay buffer for behavioral cloning. In experiments, the algorithm (SPLID) performs well in deterministic, goal-conditioned environments.
main_review: Strengths:

- I appreciate the theoretical analysis of the method, as well as the step-by-step walk-through of the algorithm in section 3.1.

Weaknesses:

- Despite similarity with GCSL, GCSL performs far worse than SPLID or even SPLID without SELECT, which as far as I'm concerned is pretty much the same algorithm as GCSL with different exploration (GCSL mentions they initialize their replay buffer with random actions and they play around with different exploration strategies in section A.7, so maybe SPLID w/o SELECT could be considered the same as GCSL). Is the main difference between these methods exploration?
- The paper makes a bold claim that "resetting the environment to a given previous state ... is always tractable in simulation". This seems like a major assumption and is more than the baseline methods assume. Is there a way to implement SPLID without the ability to reset the environment to arbitrary states, perhaps using value functions?
- GCSL only optimizes for reaching the goal state by its final step, which is a slightly different problem setting than this paper. The theoretical improvement that SPLID offers is that, through the SELECT function, it will be able to learn to achieve goals in a shorter number of time-steps. However, the experiments only measure success rate, which does not take into account how optimal these polices are in terms of time taken to reach the goal.
- The paper claims that SPLID does parameter space exploration, but they simply add noise to the actions of the previous policy, which is not parameter space exploration.
- There is no discussion on the assumptions (deterministic, resettable environment) that SPLID makes in relation to the baselines.

My main questions are as follows:

- Why does GCSL perform so poorly?
- Can you implement SPLID without the assumptions of the environment?
- Does SPLID w/ select learn policies that reach the goal in fewer time-steps than SPLID w/o select?

Suggestions (not taken into account for the score):

There are several grammar mistakes as well as other mistakes which caused some confusion (for example, in theorem 2, readers are pointed to the wrong spot to find the definition of $\Delta_k$). 
summary_of_the_review: While this paper seems to get strong performance in the selected experiments, I find the experiments to be a bit suspicious and under-explored and I am unconvinced that the performance gained with this method are worth the strong assumptions (env is resettable to a selected state). I cannot recommend accepting this paper.
review_title: null

Meta-review: The paper proposes a novel meta-algorithm, called Self-Imitation Policy Learning through Iterative Distillation (SPLID) , which relies on the concept of -distilled policy to iteratively level up the quality of the target data and agent mimics from the relabeled target data.
Several aspects of the paper can be improved. The reviewers are concerned in particular about the experimental section which might not exhaust the core set of tasks, where the method should be compared with baselines. Furthermore the presentation can be significantly improved (lots of grammatical errors). Another major point is the novelty of the presented algorithm.

In the rebuttal the authors tried to address some of the remarks, in particular by adding additional experiments to the empirical section of the paper. Those experiments still do not convince some of the reviewers. Furthermore, one of the biggest concerns is still a limited novelty of the approach. The presentation of the paper still needs to be substantially improved. Thus the paper still requires nontrivial work.
Decision: Reject