Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focusses on the setting of goal-conditioned RL where subgoals are provided by graph based planners. It introduces two algorithmic contributions:
* a self imitation loss that is inspired by the "optimal substructure property" ie the fact that parts of an optimal path are themselves optimal. This idea is used to regularize the policy conditioned on a final goal towards those conditioned on the subgoal along a planner path
* subgoal skipping: instead of always attempting to reach the first subgoal the policies sometimes directly attempts more distant subgoals along a planned path to aid with exploration.
The proposed innovations are primarily evaluated in a number of maze environments. Empirically the proposed approach compares favourably to baselines, especially in more complex maze environments.
strength_and_weaknesses: Strengths:
* well motivated new objective that empirically seems to improve performance
* clearly written paper with pretty thorough evaluation

Weaknesses:
* since PIG can be applied on top of L3P it might be nice to see those results as well. 
* in some of the learning curves not all (or even no) methods seem to have converged. It would be interesting to evaluate for longer to be able to assess whether the empirical improvements are in learning speed or also in asymptotic performance.
* the specific implementation of subgoal skipping seems a little ad-hoc. Is it obvious that this is the right choice? I think mentioning the comparison to random goal sampling along the path in the main text might be good. I would also like to see an ablation over a wider range of $\alpha$ (fig 11).
* It's not clear to me that any of these approaches, even with the proposed improvements would scale to higher dimensional tasks. 
* figure 8: which environment is this?
* evaluation: are you reporting standard deviation or standard error in the mean?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and well motivated. To the best of my knowledge the proposed new objective is novel and leads to empirical performance improvements. I have no concerns about reproducibility since the method is relatively straightforward and there is code provided. 
summary_of_the_review: Nice paper with a well-motivated simple new objective that appears to lead to improved performance in goal-conditioned RL with graph-based planner guidance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work addresses goal-conditioned reinforcement learning and proposes two techniques to improve existing graph-based planning methods. The first technique is to distill the outcome of the graph-based planning procedure into the goal-conditioned policy. The second technique is to randomly skip subgoals along the planned path when the agent interacts with the environment. The authors name the combination of these two techniques PIG and evaluate PIG in a set of continuous control tasks. Empirical results show that PIG outperforms strong baselines and improve performance in all tasks. The authors also provide practical recommendations on setting the hyperparameters introduced by PIG.
strength_and_weaknesses: # Strength
* The paper is well-written and easy to follow.
* The ideas of the two techniques in PIG make sense intuitively. I especially like the idea of distilling the outcome of the planning procedure to the goal-conditioned policy by self-imitation. This technique provides a way for the agent to bootstrap its own learning.
* The empirical evaluation is thorough and solid. PIG demonstrates strong performance against baseline methods in the experiments. The ablation studies further demonstrate the efficacy of both proposed techniques.
* I appreciate that the discussion on limitations of PIG at the end of the paper.

# Weakness
* The authors mainly discuss related works in the goal-conditioned reinforcement learning literature. However, I believe the idea of distilling the planner's outcome into the goal-conditioned policy is connected to some works in the broader planning context. For example, AlphaGoZero [2] distills the outcome of the Monte-Carlo tree search (MCTS) planning procedure into a prior policy. Similarly, SAVE [1] distills the MCTS outcomes into the action-value function. I strongly suggest the authors to broaden the scope of the related work section and add a discussion on the aforementioned works.
* It would be good if the ablated versions of PIG are also compared to the baseline methods. For example, by reading the numbers approximately from Figure 4 and Figure 7, it looks like PIG w/o skipping performs worse than MSS. This observation would be intriguing if true because I would expect the self-imitation learning technique itself to be beneficial.

# Questions and discussion
* In Equation (4), all intermediate subgoals are equally weighted. But the idea behind the subgoal skipping technique essentially says the opposite. I wonder if we can further improve PIG by using a similar idea as subgoal skipping and adaptively weighting the subgoals in Equation (4). I would love to hear the authors' comment.
* Why did you choose U-shaped Ant Maze for the experiment in Figure 5 instead of the larger version of it? PIG shows the largest improvement in large U-shaped AntMaze in Figure 4. In the current Figure 5, MSS + PIG w/o planner seems to perform equally to the MSS. I am very curious to see if it can actually perform better than MSS in a more challenging environment.
* Despite providing recommendations on how to choose $\beta$ automatically, the values are fixed (by manually tuning I suppose?) in the experiments. Why is it?

# Minor suggestions
* In the second paragraph of Section 3.3, perhaps replace the edge notation ${l^{1}, l^{2}}$ by $<l^{1}, l^{2}>$.
* In the fourth line in "Evaluation" paragraph of Section 5.1, I think you mean Figure 4, not Table 4.
* Figure 6 appears before Figure 5. Please consider fixing it.

**References**
1. Hamrick _et al_ 2020, Combining Q-Learning and Search with Amortized Value Estimates
2. Silver _et al_ 2017, Mastering the Game of Go without Human Knowledge
clarity,_quality,_novelty_and_reproducibility: * Clarity: The paper is well-written and easy to follow. The methods are clearly presented.
* Quality: The provided method is solid. Experiments are well designed and results are thoroughly discussed.
* Novelty: to the best of my knowledge, the proposed two techniques in PIG are novel.
* Reproducibility: Sourcescode is provided. Implementation details are described in the appendix.
summary_of_the_review: This paper proposes two techniques to improve graph-based planning methods in goal-conditioned reinforcement learning. The proposed techniques are thoroughly evaluated and demonstrate strong empirical performance against baselines. The paper is well-written and the method is clearly presented. Thus I recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a novel and generic framework Planning-guided self-Imitation learning for Goal-conditioned policies (PIG) to improve the sample efficiency of goal-conditioned RL (GCRL). Empirical results show that PIG significantly boosts the performance of the existing GCRL methods under various goal-reaching tasks.
strength_and_weaknesses: The paper is well-written and organized, and the proposed method is simple but seems to be effective. However, the paper may not have a high impact on reinforcement learning due to some limitations of PIG.

PIG seems to be effective only for goal-reaching tasks, and all the experiments are conducted in goal-reaching tasks. Whether is PIG applied to tasks with complex reward functions? 

Whether is the proposed graph construction technique applied to complex environments with stochastic transition models, directed-graph-based, or high-dimensional state spaces? 

Subgoal skipping may induce a farther subgoal that is difficult to reach by the agent. How to guarantee that the technique can provide appropriate subgoals for the agent? 

clarity,_quality,_novelty_and_reproducibility: The paper is clear, but the novelty is limited
summary_of_the_review: The current version does not convince me to recommend acceptance


######After Rebuttal######

The additional explanation and experiments address most of my concerns
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies goal-conditioned RL, in particular the graph-based approach, where we build a graph of subgoals to improve overall performance. Compared to previous methods, this paper introduces two improvements: 1) an additional loss term to make subgoal policies compatible with end goal policies, and 2) at execution time a stochastic subgoal skipping mechanism which makes the agent sometimes aim for further ahead subgoals in the planned path. Experiments in a range of tasks show improvement on some tasks, while being on par with baselines on other environments. 
strength_and_weaknesses: Strength:
* The paper is clearly written. 
* Related work is well covered. 
* Section 3 is a very clear introduction to the overall setting and previous approaches. 
* Good ablation experiments. 

Weaknesses: 
* My biggest issue is with the evaluation of the baselines. You indicate that for the baselines you “used the best hyperparameters reported in their source codes”. However, the baseline papers do not study all the environments you use, and are often optimized for other environments. This can make your comparison (very) unfair. If you optimized you own hyperparameters, but left their hyperparameters untouched (which were optimized for other tasks), then this will strongly influence performance. I now miss details about this procedure, and in principle I think you should put as much hyperparameter optimization effort in the baselines as you put into your own method.
* Subgoal skipping gets a lot of attention in your introduction and methodology, but then in the experiments it only features in Fig 7 and a single paragraph of results.
* In the second part of your results (Fig 6 and 7) you suddenly reduce the set of environments to three. How did you choose these three (why do you show these ones, and not the other ones?). You need to tell us why these are not cherry-picked. 
* The contribution is slightly incremental. Most focus lies on the self-imitation loss. I do think this is a nice insight, but effectively it adds a single loss term to a known algorithm. It is still an interesting idea, but not totally ground-breaking. 

Minor: 
* Sec 2: You say that progress lies in “compressing the graph by removing redundant nodes”, but then you give examples like “semi-parametric topological memory”, “farthest point sampling” and “successor features”, which do not say anything about how you reduce the number of nodes in the subgoal graph. 
* Sec 3.3: You need to explain “farthest point sampling”.
* Sec 4.2: * I had to read Sec 4.2 multiple times to understand your motivation for Eq 6, I think you could write this down more clearly. I guess it says that, when the policy for the current subgoal and the final goal agree (small L_PIG), the goal-conditioned policy has propagated information quite far. Therefore, we may try to jump to further ahead subgoals, because the goal-conditioned policy seems trustworthy. I think you could phrase this more explicitly. 

clarity,_quality,_novelty_and_reproducibility: Clarity: Good
Quality: Decent
Novelty: Moderate
Reproducibility: Should be reproducible. 
summary_of_the_review: The paper is well written and clear, has extensive experiments, and introduces a method that manages to outperform baselines on most of the tasks. My biggest issue is the experimental set-up, where the authors have optimized the hyperparameters of their own algorithm, but reused hyperparameters of the baselines that were optimized for different tasks. The contribution is slightly incremental (two tweaks to a known algorithm), but still interesting. I'm on a borderline vote for this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The contributions of this paper involve leveraging the optimal substructure property to better train goal-conditioned policies and performing subgoal skipping during training and deployment to allow for discovering more efficient plans.

The reviewers generally found the paper well motivated and clear, with initially mixed opinions about the thoroughness of the paper.  Reviewer tgwg expressed some concern that the approach may not scale well to higher-dimensional problems.  The authors replied that graph-based planning has been applied to e.g. visual perception problems, so they are optimistic.  Reviewer 4xSo found the evaluation sufficiently thorough, but pointed to some connections in the existing literature that the authors could include.  Reviewer X3ba initially gave the paper a 3 due to potentially limited applicability.  However, they updated their score to a 6 based on author explanation and additional experiments.  Reviewer syAR's biggest concern was related to baseline comparisons, especially around hyperparameter selection.  

Generally, the authors made an effort to address reviewer concerns to a reasonable standard.  

I'm comfortable endorsing the reviewer consensus for this paper to be accepted.
justification_for_why_not_higher_score: Overall, I think this paper foregrounds some interesting ideas.  I personally share the concern raised by reviewer tgwg that this class of approach might be a bit limited in the problems it is applicable to, as well as the adjacent concern by reviewer X3ba questioning how general the approach is (i.e., restriction to goal-reaching tasks).  However, within the problem-space that this paper seeks to solve, there is a clear contribution. 
justification_for_why_not_lower_score: 4/4 reviewers see a clear and meaningful enough contribution, and I agree.
Decision: Accept: poster