Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Summary:** This paper proposes CO-PILOT that can improve performance in goal-conditioned RL settings. The main contribution of CO-PILOT is the collaborative learning between the planning and RL agents: the planning agent generates the sub-task tree providing dense feedback for the RL agent to solve a task, while the RL agent collects data and refines the cost function for the planning agent. Appealing benefits of CO-PILOT include 1) the auto-generated curriculum that helps training both the planning and RL agents, and 2) no requirements of a pre-trained policy, a heuristic for the cost function, and an accurate environment model. Empirical results show that CO-PILOT outperforms RL, planning, and mixed baselines in various domains.
main_review: **Reasons for Score:** Overall, I vote for a score of 6. While CO-PILOT provides an interesting insight (i.e., the mutual boosting between RL and planning) and shows positive evaluation results, there are several concerns and questions that I would like to clarify (please refer to Concerns and Questions below). After reading the authors' responses to my concerns, I am open to raising my score.

**Strengths:**
1. The paper is well written and motivates the main insight very well.
2. The approach is principled. In particular, the paper introduces the interesting idea of auto-generated curriculums for the planning agent (top-down) and RL agent (bottom-up) to stable learn these policies. 

**Concerns and Questions:** 
1. Novelty is limited because CO-PILOT shares many similarities to the sub-goal tree method [24], except for the additional learning of the RL agent. 
2. While the result in Figure 2a show CO-PILOT outperforms the sub-goal tree baseline (SGTPG), it is unclear from the result discussion in Section 4.3 how CO-PILOT achieved this result. It will be helpful to include a discussion on what algorithmic difference resulted in better performance.
3. Did the simultaneous learning of planning and RL agents result in a non-stationarity issue? For example, in [37], the simultaneous learning of both low and high-level policies created a non-stationary problem and they addressed the issue in an off-policy learning setting. 
4. Many planning methods have important properties, such as asymptotic optimality, probabilistic completeness, and monotone convergence (Karaman and Frazzoli, 2011). Does the planning agent in CO-PILOT also maintain these properties?
5. One main benefit of CO-PILOT is the curriculum learning for the planning/RL agents. This benefit is also presented in the hindsight experience replay (HER) approach [3], where the goals used for replay naturally shift from simple to difficult ones. While it is not needed to compare against HER during the rebuttal period, it would be helpful to include the comparison against HER in the future. 

**Additional Feedback:**
1. Similar to having $\pi$ as a superscript in the Q-function $Q^{\pi}$ (denoting the agent follows $\pi$ in the future states), it will be helpful to denote the value function as $V^{\pi}$ instead of $V$. 
2. It will be helpful to add variances in some results: Figures 2a, 2c, 4a, 4b, 4c. 
3. It will be helpful to clarify whether the results (e.g., Figure 2a) are from training or testing pairs.
4. Figure 1 is not discussed in the introduction, so it would be better to place the figure where it is referenced (Section 3.3). Additionally, Figure 1b is not referenced in the text.   
5. The current pdf format does not support search (i.e., ctrl + f) (tried on Linux and Mac). It was difficult to find a specific keyword in the paper, so I hope this issue is fixed. This might be not an issue with other computers. 

**Reference:**
Sertac Karaman and Emilio Frazzoli. Sampling-based Algorithms for Optimal Motion Planning. International Journal of Robotics Research (IJRR), 2011.

**Post-Rebuttal:** Please refer to the review below. 
limitations_and_societal_impact: This paper does not discuss any potential negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper discusses how planning and goal-conditioned RL can be combined for long-horizon planning. The method proposes to use a recursive subgoal generating policy, which is applied repeatedly at k levels to create 2^k subgoals. A lower level goal-conditioned policy attempts to reach the subgoals. The objective for the higher-level policy depends on the lower-level policy by estimating the time it takes the lower-policy to go between a state and goal. The paper shows that the method outperforms planning methods (NEXT, RRT, SGTPG), RL methods (SAC, PPO) and hybrids (SoRB, HRL). Generally the paper is well-written and communicates effectively idea that each module can assist the other in these hybrid planning/GCRL methods. However, it is not fully clear to me yet from the experiments which design decisions were important, and the method is also only evaluated on relatively simple tasks.
main_review: The paper proposes CO-PILOT, which introduces a recursive subgoal generating policy to make reaching far-away goals easier for a goal-conditioned policy. The claim is that the higher level policy assists the lower level policy by generating closer goals (standard HRL), but the lower level policy also assists the higher level policy by informing the cost function between states, which gets more accurate with a higher quality low-level policy as better data is collected.

Method: the writing is relatively clear. There is something cyclic going on that I didn’t understand in the top-down planning/algorithm 1. c is defined in terms of tau; tau is defined in terms of c_phi; c_phi is optimized … how? what is the objective? From algorithm 1 it seems like it depends on c. Is there a “grounded” cost that the whole system actually optimizes (does it come from alpha being 1 in eqn 12 so it is L2 distance)?

Experiments: The experiments show that CO-PILOT outperforms all other classes of methods: planning methods (NEXT, RRT, SGTPG), RL methods (SAC, PPO) and hybrids (SoRB, HRL). In section 4.4, additional visualization on a 2D maze shows the CO-PILOT learning process, splitting a goal into subgoals and improving the policy between subgoals. However, on this task, other methods are just as good; a more helpful visualization would be to understand why other methods fail on harder tasks while CO-PILOT does not. Perhaps a similar visualization can be constructed for something like Ant (still plotting on a 2D maze by accessing the underlying state information).

Generally, the paper proposes a large system of learning a cost function, a planner/subgoal generator, and a goal-conditioned policy. It is difficult to parse from the writing and results which component matters in the end, although the experiments are relatively thorough and may have the answer already. Some questions are: if SGTPT performs similarly to CO-PILOT, does this mean the goal-conditioned reward feedback to the planner does not matter (or what is the delta between the two)? The planner incorporates L2 reward, while the maze tasks are sparse reward; what if the L2 heuristic was used for SAC as the cost instead of sparse rewards?

Tasks: the paper evaluates on a few different domains, but one criticism is that the domains are all either maze tasks from state (2D, 3D, 5D, and ant maze locomotion) or simple (BipedalWalker). In contrast, there exists several works now that have shown the ability to do goal-conditioned planning in visual domains, including SoRB (cited and compared in this work, but not in visual domains), [2], [4].

The related work is relatively complete in terms of discussing goal-conditioned RL and planning. I did not quite understand how c_phi is optimized but it looks a lot like dynamical distance learning [1]. Also consider discussing [2] and [3] which explore subgoal generation, especially [2] which generates subgoals by divide-and-conquer exactly like this paper does.

[1] Dynamical distance learning for unsupervised and semi-supervised skill discovery. Hartikainen 2019.
[2] Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors. Pertsch 2020.
[3] Automatic Goal Generation for Reinforcement Learning Agents. Florensa 2018.
[4] ViNG: Learning Open-World Navigation with Visual Goals. Shah 2021.
limitations_and_societal_impact: Limitations and societal impact are not really discussed. For limitations, it would be good to see a task where CO-PILOT fails or at least struggles to solve completely.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper addresses the problem of sequential decision making in complex, high-dimensional domains. The paper combines a learned planning policy and a learned controller to improve the performance of both. Explicit plans are used as reward shaping for the goal-conditions reinforcement learner to improve the learning rate, and the cost of the learned controller policy (“the time steps of the agent to finish a sub-task”) is used to learn a planning policy. The overall approach is compared to planning-only approaches, model-free policy learning, and a previous hybrid technique (SoRB) and the evaluation is on 3 different benchmark problems. CO-PILOT performs the best of the different techniques in both asymptotic performance and speed of learning. In some cases SoRB also has similar asymptotic performance, but requires more data and takes longer to converge to that. 

main_review: This paper proposes an interesting idea. The core insight that “planning at even a coarse level can be used for reward shaping and substantially improves RL on long-horizon tasks with sparse reward” is not especially novel and would seem to be central insight of a considerable body of work in hierarchical abstractions for sequential decision making. However, the idea that the planner and learner provide training signals to each other is interesting and potentially novel. I am moderately predisposed to recommend this paper for acceptance, but it is hard to argue strongly for it, in that the narrative is really not clearly articulated, and the experiments could still use further improvement. 

My primary concern is that the paper is sitting in a *very* crowded intersection of competing ideas, and does not really engage with the substantial literature in learning hierarchical abstractions. There are some references, so the authors seem to know about some work in hierarchical abstractions, although many more should be added, such as recent work by Konidaris (“From Skills to Symbols”, 2018) and older work such as Pineau et al (“Policy-contingent abstraction for robust robot control”, 2003), Michini and How (““Bayesian nonparametric inverse reinforcement learning”, 2012) and Surana and Srivastava (“Bayesian nonparametric inverse reinforcement learning for switched markov decision processes”, 2014). That being said, more experimental baseline comparisons are essentially, especially to hierarchical representations. 

Additionally, the paper needs to more strongly articulate what is different about their approach, and what specific benefits are enabled. For instance, what if the learner were replaced with a fixed set of motion primitives? Figure 4 might suggest that the algorithm is just learning a sampling distribution, and a comparison to “Learning Sampling Distributions for Robot Motion Planning” (Ichter et al, 2018) would be appropriate, especially if learning the controller inside the planning loop really does add something. 

The experimental results are ok, but a great deal of the motivation for this technique comes from high-dimensional and long-range planning problems. The Maze and Mujoco Ant-v1 are neither high-dimensional nor long-range (8 dimensions is not high dimensional). Bipedal-Walker is better, but not an especially hard planning problem. I would recommend substantially expanding the scale and dimensional of a Maze-like environment to really show the algorithm learning smart planning strategies and smart controllers.  

The paper contains some statements and assumptions about planning that are not quite right. 
- “Moreover, planning every step on the path is as difficult as the original RL and requires fine-grained discretization impractical for high-dimensional states.” — a fine-grained discretization is not necessarily required even for high-dimensional states, and random sampling techniques go a long way towards addressing this. Note that random sampling planners are not the same thing as sampling-based search heuristics. 
- “Planning only a few milestone states leaves the RL agent to solve relatively long-horizon sub-tasks.” This may be true, but those relatively long-horizon sub-tasks may be extremely easy to solve.  
- “Although sampling-based search heuristics can build a graph with a better exploration-exploitation trade-off, they are not optimized for the RL policy, and the graph cannot be shared across tasks.” This is also not true — multi-query planning techniques such as the probabilistic roadmap are explicitly designed for re-use.

I see the comment that this is a resubmission of a previous paper. I appreciate that the authors have gone to the trouble of reporting that, and the changes they made. 

limitations_and_societal_impact: I could not identify a clear statement of the limitations of the paper anywhere in the paper. The word “limitation” does not appear in the paper. It is possible that the authors intended that the trade-offs between ease of learning vs complexity of planning as a function of sub-task tree depth (section 4.3, final paragraph) to be an assessment of the limitations, but this is not particularly useful, and is not a statement of the limitation of the technique — what would be more useful is analysis of the kinds of problems where the technique fails. Under what conditions is the planner unable to guide the learner, and another technique should be used? 

The author checklist indicated that the authors did not discuss any potential negative societal impacts of the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents CO-PILOT, a collaborative learning approach in which a planner, generating subtasks of varying complexity, and RL policy, learning to achieve those sub-tasks, are trained together through a curriculum of tree-structured subtasks. This simultaneous curriculum-based training helps to learn functions that can solve a variety of tasks in cluttered environments. 
main_review:  The paper is well written, and most key aspects of their algorithm are neatly discussed. However, there are some concerns, and some of them can be addressed in the revised version.


1- Related work can be improved:
- Difference between HRL and CO-PILOT should be further explained. The main difference is that the latter introduces a tree-based curriculum for training high and low-level policies, leading to better performance than existing HRL methods.

- Although I see this approach is different from imitation learning-based path planning approaches, including them in the literature would be nice. 

2- Methods:
- Fig 1(b) can be described in a bit more detail in the caption.

3- Results:
- Generalization of their approach to different start and goals is not shown. For instance, in the maze, I think 100 (s,g) test pairs should be shown in the appendix to indicate how these samples were distributed, likewise for other environments. In the visuals, it seems the start and goal points were not quite distributed around the space. 

- Computational times of different algorithms during execution/testing should be compared, or at least they should be discussed in the paper. 

- Also, it is not clear why RRT* has a lower success rate than CO-PILOT. I believe the presented problems are simple enough for RRT* to solve them. Is it due to the time limit?

limitations_and_societal_impact: - Limitations of their work need to be highlighted, especially generalization to new environments and how it can be addressed. 
- Societal impacts: (N/A)
review_title: null

Meta-review: The reviewers' appreciate the interesting ideas in this submission and the generally clear presentation, and the authors detailed responses to the reviewers' concerns. The authors are encouraged to revise their camera ready to address the feedback from the reviewers. 
Decision: Accept (Poster)