CO-PILOT: COllaborative Planning and reInforcement Learning On sub-Task curriculum

Abstract: Goal-conditioned reinforcement learning (RL) usually suffers from sparse reward and inefficient exploration in long-horizon tasks. Planning can find the shortest path to a distant goal that provides dense reward/guidance but is inaccurate without a precise environment model. We show that RL and planning can collaboratively learn from each other to overcome their own drawbacks. In ''CO-PILOT'', a learnable path-planner and an RL agent produce dense feedback to train each other on a curriculum of tree-structured sub-tasks. Firstly, the planner recursively decomposes a long-horizon task to a tree of sub-tasks in a top-down manner, whose layers construct coarse-to-fine sub-task sequences as plans to complete the original task. The planning policy is trained to minimize the RL agent's cost of completing the sequence in each layer from top to bottom layers, which gradually increases the sub-tasks and thus forms an easy-to-hard curriculum for the planner.  Next, a bottom-up traversal of the tree trains the RL agent from easier sub-tasks with denser rewards on bottom layers to harder ones on top layers and collects its cost on each sub-task train the planner in the next episode. CO-PILOT repeats this mutual training for multiple episodes before switching to a new task, so the RL agent and planner are fully optimized to facilitate each other's training. We compare CO-PILOT with RL (SAC, HER, PPO), planning (RRT*, NEXT, SGT), and their combination (SoRB) on navigation and continuous control tasks. CO-PILOT significantly improves the success rate and sample efficiency.


Reviews and author responses:

The authors propose an algorithm for jointly learning a sub-goal-setting policy and a goal-reaching policy. To train a sub-goal setting policy, the authors propose to minimize the sum of cost between generated sub-goals, where the cost is determined by a model that predicts the number of steps to move between the two goals. The goal-reaching policy is trained with sparse rewards. Lastly, this time-prediction model is trained with data collected by the policy, effectively “closing the loop” between the policy and the planner. The papers compare to existing graph-based RL methods and standard RL algorithms (PPO, SAC), and find that the resulting algorithm generally outperforms prior methods on numerous navigation tasks.  The paper studies an interesting problem of combining planning with goal-conditioned RL. The method is novel and proposes a promising method to “close the loop.”

The experiments are fair, but there are additional details and comparisons that would strengthen the paper. For one, it seems that the authors did not compare to standard goal-conditioned algorithms such as HER. Adding this comparison would greatly help. Another concern is that the experiments do not describe how the prediction model is trained.

Algorithmically, it is unclear why the planning policy must be trained with a policy gradient method. Given that the cost is evaluated with a learned prediction model c, can’t the authors directly minimize Eq. 8 with back-prop?

A major concern with the paper is that I found it generally difficult to follow. There were a few statements and notations in Section 3.2 that are not explained. For example, the paper says, “As T increases, r(s, a, g_{0:T}) becomes denser” and “planning provides dense rewards and sub-tasks to train the RL agent” but it is unclear why this is the case. How is planning providing “dense rewards”? Equation 6 seems to be a sum of sparse rewards, which will generally be sparse as well. Also, from Equation 6, is the policy conditioned on a sequence of goals, g_{0:T}?

There are also a number of other writing issues:
 - Section 3.4 seems redundant with the previous sections, unless I am missing something.
 - The figure font is too small and there is a lot of standard vertical white space missing between figures, text, and section headers.
 - There is a recursive dependency between Equation 11 and 12. How is this resolved?
 - “We set a reward of  1(1000,200) to each task”)
 - Equation 11 does not give a proper update rule for the prediction model c. (as indicated in line 9 of Algorithm 1

Overall, although the experiments are promising, the redundancy and lack of clarity in writing make it difficult to recommend an acceptance of the paper, along with the other concerns mentioned above.

Edit: I acknowledge that I have read the author's response and will maintain this score.  The authors do not discuss the limitations of the work. Adding such a discussion would help.

 Thank you for your answers to my questions. 
I believe the comments by me and the other reviewers can be used to make the paper much stronger, but it requires too many changes to the manuscript which would not be able to be peer-reviewed, so I keep my score.

To your response to my path-planning concern:
I really think that the word path-planner that appears everywhere is misleading because you are doing a learning-to-plan approach. Again I think the real strength of such an approach should be on the generalization to new environments and or new goals. 

I cannot understand your argument that the paths in Fig 4 are already difficult, just because prior work has also trouble with those. 
The point is that the Euclidean distance is a very good prior for those. If you are not showing a path with significant de-tours, I am not convinced of any (learned) path planning is happening.

 > We train the path-planner using the objective in Eq. (8) to produce easier sub-tasks on which the RL agent can achieve a dense reward. In particular, the path-planner is trained to split the original long-horizon task into a sequence of easier sub-tasks that the RL agent can spend less time and travel less distance (i.e., smaller cost in Eq. 12) to accomplish. So the RL agent’s policy is only conditioned on one sub-task’s goal every time instead of the whole sub-goal sequence, i.e., condition  in Eq. 6 is the subgoal for sub-task t.

Thank you for explaining. I believe a more accurate description is that the policy is solving a "shorter-horizon" problem rather than a "more dense-reward" problem. That would also explain my confusion around, "As T increases, r(s, a, g_{0:T}) becomes denser".

Thank you for also clarify the confusion around Equation 6, 11, and 12 and for adding the comparison to HER.

> When $c_\phi$ is available, [back-propagating through $c_\phi$ directly] is feasible technically but does not perform well in practice

It would be great to include these results in the paper. 

Overall, many of the details included in this rebuttal would strengthen the paper. However, there are a number of follow-up questions that would need more explaining:
 - why was this hyperbolic function used to train the prediction model $c_\phi$? Generally, it seems that this model is a significant contribution of this work, and so spending more time on explaining how this model is trained and justifying the design decision (both verbally and empirically) would strengthen the contribution of this work.
 - If training $c_\phi$ "would be very costly in a large/continuous state problem" how generally applicable is this method? Adding a discussion of the limitations of the method would help.

The discussion with Reviewer SCqH in particular reveals that there are other details like the collision checking that are not included in the paper. It also seems that clarity was an issue for every reviewer. While I do believe that incorporating the answer to my previous questions, these additional questions, and the questions raised by other reviewers would make the paper potentially ready for acceptance, this would require significant re-writing, both at the low level (grammar and clarity) and the high level (describing the appropriate technical details, justifying the comparisons, discussing limitations), and so I plan to keep my score.

Lastly, I just remembered that this paper would be particularly relevant to discuss in the related works, and particularly when discussing the design choices behind training $c_\phi$:
Hartikainen, Kristian, et al. "Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery." International Conference on Learning Representations. 2020.

 ### Comparison to HER in the Maze environment
- Both HER and CO-PILOT train a goal-conditioned RL policy on modified/new tasks with dense reward. Their main difference is that HER relabels the failed tasks tried in history while CO-PILOT trains a sub-goal planning policy to generate a curriculum of new subtasks. Compared to HER that cannot fully control the utility of the relabeled tasks, CO-PILOT learns to propose subtasks adaptive to the RL agent’s training progress and automatically organizes these subtasks in an easy-to-hard curriculum, which provides more informative and effective guidance to RL. This difference usually leads to higher success rate and better training efficiency of CO-PILOT, as shown in the experiments below.
-We present an empirical comparison of HER with CO-PILOT on the Maze environment tasks. We chose a highly rated implementation of HER on Github and evaluated “DQN with HER” and “DDPG with HER”. For fair comparison, HER is trained on 300 $(s_0,g)$ pairs and tested on 100 $(s_0,g)$ pairs (the same as CO-PILOT, Sec. 4.1). We apply the training setting used in our paper for HER.  
- **Table.1 The success rate averaged over 6 test epochs for 100 test tasks (mean$\pm$std) in Maze (2D/3D/5D defined in Sec. B of the Appendix).  
|Method|Maze (2D)|Maze (3D)|Maze (5D)|
|:--|:--|:--|:--|
| CO-PILOT | 94.02$\pm$3.37 | 87.34$\pm$5.25 | 78.26$\pm$8.24 |
| DQN with HER | 89.12$\pm$5.34 | 82.58$\pm$6.14 | 78.81$\pm$10.62 |
| DDPG with HER | 80.85 $\pm$ 7.67 | 70.02$\pm$10.37 | 66.43 $\pm$ 11.25|

- In the table above, CO-PILOT outperforms HER on most tasks except on Maze(5D), where DQN with HER has a slightly higher mean success rate and CO-PILOT has a smaller std (more stable performance). Due to the limited time for rebuttal, we have not evaluated HER in more environments (such as Mujoco-Ant and BipedalWalker in the paper) but we will add more comparison results later.

 ### Comparison to HER in the Maze environment
- Both HER and CO-PILOT train a goal-conditioned RL policy on modified/new tasks with dense reward. Their main difference is that HER relabels the failed tasks tried in history while CO-PILOT trains a sub-goal planning policy to generate a curriculum of new subtasks. Compared to HER that cannot fully control the utility of the relabeled tasks, CO-PILOT learns to propose subtasks adaptive to the RL agent’s training progress and automatically organizes these subtasks in an easy-to-hard curriculum, which provides more informative and effective guidance to RL. This difference usually leads to higher success rate and better training efficiency of CO-PILOT, as shown in the experiments below.
-We present an empirical comparison of HER with CO-PILOT on the Maze environment tasks. We chose a highly rated implementation of HER on Github and evaluated “DQN with HER” and “DDPG with HER”. For fair comparison, HER is trained on 300 $(s_0,g)$ pairs and tested on 100 $(s_0,g)$ pairs (the same as CO-PILOT, Sec. 4.1). We apply the training setting used in our paper for HER.  
- **Table.1 The success rate averaged over 6 test epochs for 100 test tasks (mean$\pm$std) in Maze (2D/3D/5D defined in Sec. B of the Appendix).  
|Method|Maze (2D)|Maze (3D)|Maze (5D)|
|:--|:--|:--|:--|
| CO-PILOT | 94.02$\pm$3.37 | 87.34$\pm$5.25 | 78.26$\pm$8.24 |
| DQN with HER | 89.12$\pm$5.34 | 82.58$\pm$6.14 | 78.81$\pm$10.62 |
| DDPG with HER | 80.85 $\pm$ 7.67 | 70.02$\pm$10.37 | 66.43 $\pm$ 11.25|

- In the table above, CO-PILOT outperforms HER on most tasks except on Maze(5D), where DQN with HER has a slightly higher mean success rate and CO-PILOT has a smaller std (more stable performance). Due to the limited time for rebuttal, we have not evaluated HER in more environments (such as Mujoco-Ant and BipedalWalker in the paper) but we will add more comparison results later.


 We appreciate your time and suggestions! As you suggested, we are working on new experiments of the comparison to HER and will post it as soon as possible. In this reply, we address your concerns except the experimental comparison:

$\cdot$ How is planning to provide “dense reward” for RL training? 

We train the path-planner using the objective in Eq. (8) to produce easier sub-tasks on which the RL agent can achieve a dense reward. In particular, the path-planner is trained to split the original long-horizon task into a sequence of easier sub-tasks that the RL agent can spend less time and travel less distance (i.e., smaller cost in Eq. 12) to accomplish. So the RL agent’s policy is only conditioned on one sub-task’s goal every time instead of the whole sub-goal sequence, i.e., condition $g_t$ in Eq. 6 is the subgoal for sub-task t.

$\cdot$  “As T increases, r(s, a, g_{0:T}) becomes denser” and “planning provides dense rewards and sub-tasks to train the RL agent” but it is unclear why this is the case.

A larger T adds more subgoal waypoints between the starting state s and the final goal g of a long-horizon task, as shown in Fig. 1(a). Since the RL agent receives a reward when reaching each subgoal, more subgoals means that the agent has more chances to get rewards along the way. Moreover, the path-planner is trained to find a shortest path of T subgoals that minimizes the time cost spent and distance traveled by the agent between every two consecutive subgoals (as shown in Eq. (8) and Eq. (12)), so more subgoals also means easier subtasks and higher success rate for the RL agent to complete each subtask (between two consecutive subgoals) and to get the subtask reward. This can be verified by comparing the histograms for different rows in Fig. 4, which show the costs of the RL agent to accomplish each subtask. Note the range of y-axis for the histograms reduces from 4 to 1.5, indicating the subtasks become easier for the agent if increasing T.

$\cdot$ From Equation 6, is the policy conditioned on a sequence of goals, g_{0:T}?

No, as defined in L186-189, the planning policy $\pi_p(g|g_i,g_j)$ is only conditioned on a starting state $g_i$ and a goal $g_j$, not the whole subgoal sequence $g_{0:T}$. As shown in Eq. (7), we generate the sequence $g_{0:T}$ by recursively applying $\pi_p$.  

$\cdot$ Algorithmically, it is unclear why the planning policy must be trained with a policy gradient method. Given that the cost is evaluated with a learned prediction model c, can’t the authors directly minimize Eq. 8 with back-prop?

When $c_\Phi(g,g’)$ is available, this is feasible technically but does not perform well in practice because the back-prop goes through $c_\Phi(g,g’)$, which is not always accurate for arbitrary $(g,g’)$ since it is only trained on the subtasks $(g_{t-1}, g_t)$ that the RL agent has accomplished. One can think that the path-planner policy takes actions to generate subgoals and then receive a reward (i.e., the cost in Eq. (12)) for each subtask from the RL agent, which is a standard RL problem.
Back-prop is not a general solution since $c_\Phi(g,g’)$ is not always available. For example, $c_\Phi(g,g’)$  hardly adapts to a partial observation task (BipedalWalker), and training it would be very costly in a large/continuous state problem (Mujoco).

$\cdot$ Recursive dependency between eq.11 and eq.12.

This is not a recursive definition since $c_\Phi$ in Eq. (11) and $c$ in Eq. (12) are different. By substituting Eq. (11) into Eq. (12), one can get the final cost function. 

$\cdot$ Equation 11 does not give a proper update rule for the prediction model c. (as indicated in line 9 of Algorithm 1).

The time-cost prediction model $c_\phi(g,g’)$ is a MLP with three hidden layers. We train it using a monotonic hyperbolic function as the optimization objective, i.e.,
$J_\phi = \frac{e^{a \cdot c_{\phi}(s,s’)} –1} {e^{a \cdot \bar{c}(s,s’)} + 1}$
Where a is a hyper-parameter ($a=0.1$ in our experiment), $\bar{c}(s,s’)$ is the true time cost of the agent, and $c_{\phi}(s,s’)$ is the predicted time cost. We train this prediction model to converge to the true time-cost that the RL agent needs to take from $s$ to $s’$.

$\cdot$ writing issues:
$\cdot$ Section 3.4 seems redundant with the previous sections, unless I am missing something.

Section 3.4 aims at giving a high-level illustration of the mutual training idea between the planner and RL agent, which makes the training of each one easier. The “closing the loop” means that we finished to explain why this idea helps. We will rephrase it.

$\cdot$ “We set a reward of 1(1000,200) to each task”)

We set $r(s,a,g)=1$ for Maze, $r(s,a,g)=1000$ in Mujoco, and $r(s,a,g)=200$ in BipedalWalker. The rewards in all these environments are sparse for the RL agent.





 We appreciate your time and suggestions! We will improve the writing and make the text interactive. We will update more experimental results as you suggested ASAP. Here are our replies to your questions and concerns:

$\cdot$ $c(g,g’)$ a “specific usage and not the general case.”

-- $c(g,g’)$ is not for a specific usage only but can be generally applied to different tasks and environments since it mainly relies on the time cost (the second term in Eq. (12)) of the RL agent to finish a task, which is task and environment agnostic but precisely measures the true cost. The first term in Eq. (12), i.e., the Euclidean distance, is only used for “warm starting” at the very beginning of training, according to our curriculum below Eq. (12).

$\cdot$ Extend the policy gradient theorem of SGT-PG to a stochastic cost function:

-- We agree that the cost of the RL agent is stochastic because the RL agent’s policy is stochastic. However, as shown in Alg. 3, in every episode, we apply the policy gradient theorem to train the planning policy until convergence (Alg. 3 L8 and Alg .1 L8) after the RL agent finishes all the subtasks on the curriculum tree (Alg. 3 L7). And the policy gradient is based on the costs predicted by a deterministic model $c_\Phi(g,g’)$ (updated in Alg.1 L9 using the stochastic costs of the RL agent). Hence, we can treat the predicted cost as a deterministic function in each episode and apply the policy gradient theorem of SGT-PG.

$\cdot$ Please explain (formally) what is a monotonic hyperbolic function.

-- The time-cost prediction model $c_\phi(g,g’)$ is a MLP with three hidden layers. We train it using a monotonic hyperbolic function as the optimization objective, i.e.,
$J_\phi = \frac{e^{a \cdot c_{\phi}(s,s’)} –1} {e^{a \cdot \bar{c}(s,s’)} + 1}$
Where a is a hyper-parameter ($a=0.1$ in our experiment), $\bar{c}(s,s’)$ is the true time cost of the agent, and $c_{\phi}(s,s’)$ is the predicted time cost. We train this prediction model to converge to the true time-cost that the RL agent needs to take from $s$ to $s’$.

$\cdot$  you do not differentiate between costs of different levels.

-- We agree that different levels will have different costs between (s_0, g) if using the same cost function. However, the difference does not affect the training because the subgoal sequences of different layers do not compete with each other in shortest path search. In particular, the training objective for the planning policy is to find the shortest path of the minimum cost among all possible subgoal paths of the same length in each level. 

Moreover, as shown in Fig. 4, the difference becomes unnoticeable in later stages since the RL agent already learns to reach the low-level subgoals, regardless of which subgoal sequence is given. Since the cost function in Eq. (12) mainly relies on the time cost (Euclidean distance is only used at the beginning), the difference between the costs of different levels becomes very small (time does not change for different levels). However, this is not the case for SGT since it uses the Euclidean distance as its cost function, so it needs level-wise distinction between value functions.

$\cdot$ Experiments related issues:

$\cdot$  what are obstacles states? 

-- The obstacle states refer to the coordinates in the red walls shown in Fig.4 (d). The 50k tuples of data are just Euclidean distances, since the cost function in Eq. (12) mainly relies on the Euclidean distance at the beginning of training.

$\cdot$  I have to say that the details in Algorithm 2 are not clear, you can either treat every segment independently (Algo2) or not (Eq.6 and section 4.2). Please provide a clearer explanation.     

-- Eq. (6) is just a definition of the reward provided by the planned subgoal sequence. In our algorithm and experiments, we treat each segment independently as an independent task. 

$\cdot$   how much the system is impacted if the subgoals cost (c) remains the Euclidean distance?      

-- Euclidean distance is not a proper choice for the cost function because it does not take the obstacles into account and it is not adaptive to the RL agent in training. So we only use it for a warm start. We already reported the ablation result of using Euclidean distance as the cost, which is exactly what SGT-PG does. According to Fig. 2(a), SGT-PG performs poorer than our method, indicating that the planning policy training does benefit from the feedback of the SAC agent.

$\cdot$ Unclear test time behavior - you did not specify what happens during test time. I suspect that you plan subgoals and then feed them sequentially to the RL policy. What if the agent doesn't reach the next subgoal? how to decide when to advance to the next subgoal?  
    
-- In the test phase, $\pi_p$ will construct a sub-task tree by Eq. (7) given the start-goal state pair $(s_{0},g)$. The lowest level’s sub-goal sequence is then sent to the RL agent to guide the navigation. But we do not update the planning policy in the test phase. If the agent does not reach $g_j$, we switch to the next subtask and set the conditioned goal as the next subgoal. For the training phase, we set $\tau(g_i,g_j)=\tau_{\max}$ in the training phase (L221).

$\cdot$ what is the snake robot?    
 
-- Please refer to section B (L609-610) in the Appendix for a detailed description of the snake robot. 





 Thanks for your quick response! Here are our replies to your further questions:

(1) I am not sure I follow the argument here. The proposed method degrades significantly when the Euclidean distance is removed. In particular, it degrades much below the SORB level, showing that SORB outperforms the proposed method. The comparison in app. Fig. 6 is fair, and it shows that the proposed method is inferior to prior work. This should be made very clear in the main paper. In particular, the claims that CO-PILOT outperforms prior work on success rate and sample efficiency should be removed.

-- We respectfully disagree with your justification. Euclidean distance is **free** to get: given two states, it just computes their L2 distance $||s-s'||_2$. It requires nothing except access to the states, but all baselines have such access. Instead, SoRB requires a pre-trained value function $V$ to replace the Euclidean distance $||s-s'||_2$ with $V(s,s')$, which is **not free**! So the comparison in the main paper is fair to SoRB and other baselines. 

The performance drop in Fig. 6 is due to the lack of "warm start" at the very beginning, i.e., when we do not have time cost data to train the cost predictor and need something to start the training, not because that Euclidean distance provides the key information. Euclidean distance is not important (and is even harmful) to the whole training because it cannot take obstacles into account (the main skill needed to be learned by the navigation agent is to avoid the obstacles). Especially in a complicated maze environment as in Fig. 4, it is harmful. Instead, SoRB starts with a much stronger prior than Euclidean distance, i.e., the pre-trained value function. Hence, the comparison in the main paper is fair. The comparison in Fig. 6 is unfair to our method and its purpose is to evaluate the performance of CO-PILOT if it does not use any prior knowledge about the cost.

(2)  The objective specified in the response is not sound. Training with this objective will cause the predicted time cost to continuously increase since the objective is unbounded. Further, it has a hyperparameter b that does nothing (the gradient of b is 0 so it does not impact the optimization)

-- We agree that there might exist a better choice for the objective and b can be removed. We will keep trying other options and report the results later. But the current objective performs well in all our experiments and we do not have the unbounded problem empirically. 

(3) Where is this described in the paper? Eq 9 does not have this rejection step. Does this mean Eq 9 is not specified correctly?

-- We will add a detailed explanation about the collision check applied to the generated subgoals. The subgoals used to optimize the objective in Eq. (9) are the ones after the procedure we described in the answer, so they are all collision-free.



 We appreciate your time and suggestions! We will optimize the writing and formats problem as suggested. We address your main concerns below:

$\cdot$ Path planning in the paper does not use online optimization.

Instead of using online optimization for path-planning, we “learn to plan” by training an RL policy to generate the shortest path with the minimum cost. Our objective in Eq. (8) is exactly the same objective used by online optimization based path-planning (section 11.3 of [1]). In other words, we train a policy to do the same job as the online optimization. By doing so, the path-planning is more efficient (no online optimization for every subtask is needed) and more adaptive to the RL agent during training (it is trained using the agent’s latest costs). 

$\cdot$  The manual curriculum in Eq 12 is unsatisfactory and maybe not necessary?

The ideal cost in Eq. (12) should be the true time cost of the agent to accomplish the subtask. However, the second term in Eq. (12) is the time cost **predicted** by the model $c_\Phi$, which is not accurate at the very beginning of training due to the lack of collected cost data to train $c_\Phi$. Hence, we use Euclidean distance, i.e., the first term, only for “warm starting”. But Euclidean distance is not a good choice for the cost since it does not take obstacles into account. So we gradually remove it from the cost once $c_\Phi$ becomes more reliable. This leads to the curriculum of $\alpha$.

$\cdot$ In the Mujoco-Ant Maze the walls are ignored and the ant cannot rotate around its center if I understand correctly. This is an utterly strong simplification and makes a comparison in this standard benchmark problematic.

We did not ignore the walls and the ant can rotate around its center. It can collide with the wall if not well trained. In the paper, we intended to emphasize that we do not set any constraints on the agent. We will rephrase the statement to avoid misunderstanding. 

$\cdot$ task shown in fig.4 is not difficult enough.

The task in Fig.4(d) is not easy even for the latest methods, e.g., SGT, SoRB, and NEXT, as shown in Fig. 4(a). To finish the task, the agent needs to go through a narrow path while bypassing some bends. Comparing the histograms from left to right in every row of Fig. 4(d), the time costs of the agent to reach similar subgoals at the beginning of training is much higher than those after several episodes. This also implies that the task is not easy, and CO-PILOT does significantly improve the agent’s capability to finish these tasks.  

$\cdot$  Why the walker is a good example of high-level planning?

The robotic control task in BipedalWalker is very different from the navigation task in the maze environments. We aim at evaluating CO-PILOT on different types of tasks and verifying whether it is a general technique that can improve RL training in different tasks.

$\cdot$  Is the success radius the same as for the final goal?

We use the same radius for reaching the sub-goal and the final goal.

$\cdot$  Why nearby endpoints?

The planning policy $\pi_p$ is defined to predict a midpoint sub-goal given any $(g_i,g_j)$ pair and they do not have to be nearby endpoints. In our method, we recursively apply $\pi_p$ to generate coarse-to-fine subgoal sequences: given a subgoal sequence, we apply $\pi_p$ to every two consecutive subgoals (i.e., two nearby endpoints) from the sequence to produce a finer subgoal sequence. 

[1] https://www.sciencedirect.com/book/9780124170490/introduction-to-mobile-robot-control 


The paper proposes a hierarchical algorithm for goal generation in goal-conditioned RL. Specifically, SGT from Jurgenson’20 is used to hierarchically produce a tree of subgoals with a subgoal policy, and a downstream goal-conditioned RL agent is trained on these subgoals instead of just the final goal. The subgoal policy is trained with a mix of a dense and a learned reward, while the downstream policy is trained with sparse reward.
  
—— Decision ——-

The paper proposes a novel and interesting agent for hierarchical goal-conditioned learning. However, there are numerous issues with the current version of the paper.  Major baselines such as hindsight experience replay are missing from experimental evaluation, which are known to perform better on goal-conditioned RL than vanilla methods. More importantly, the proposed method leverages Euclidean dense reward while competing methods only observe sparse reward, making the experimental evaluation inconclusive. There are also technical issues with the proposed method and clarity issues.

—— Update ———

Authors' response creates numerous new questions about the method validity. In particular, it appears there are many algorithmic choices that are not specified in the paper, and are problematic in terms of technical validity. None of the issues I raised in the review are addressed. The response convinces me this paper needs an additional significant amount of work before it can be accepted.

—— Review ———

A major issue with the experimental evaluation is that the method is not compared to the baselines fairly. Section 3.2 explains how the method provides a dense signal to the goal-conditioned policy that would otherwise be trained with a sparse 0/1 reward. However, the proposed method in fact uses a dense Euclidean reward to train the subgoal policy shown in Eq 12. Since the method sees the dense reward, training the subgoal policy is much easier than for other approaches that only see a sparse reward. In App fig 6, an evaluation is provided where the dense reward is removed and the proposed method only sees a sparse reward. In this evaluation, the method is inferior to prior work such as SoRB. How can the proposed method address the sparse reward problem in goal-conditioned RL without using euclidean reward?

There is also a number of technical issues with the proposed method. 
1. The method does not appear to use hindsight relabeling. Hindsight relabeling (Andrychowicz’17) generally significantly improves performance of goal-conditioned agents, especially with sparse rewards, and is used uniformly in most recent work. Why is it not used?
2. At l217, The paper uses a “monotonic hyperbolic model” to predict the navigation cost c(g, g’). What is a monotonic hyperbolic model? How is it used to predict the cost?
3. The proposed method for training the sub-goal policy encourages is to output subgoals for which the total pairwise cost is low. However, not all robot states are actually valid e.g. in environments with obstacles or configuration space constraints (such as all environments in this paper). How does the method ensure that the subgoal policy outputs valid states? This is unclear from the paper. More generally, while the policy gradient in Eq 10 is correct, the evaluated cost c(g,g’) needs to output correct costs for all possible goals, including the invalid or the unreachable ones. How is this ensured? 

Finally, the paper suffers from numerous writing issues and grammatical mistakes. 

—— Additional Comments ——

4. The neurips submission rules (https://media.neurips.cc/Conferences/NeurIPS2021/Styles/neurips_2021.pdf) clearly state that the title should be in “initial caps/lower case”, not in lower case with some letters arbitrarily capitalized.
5. l616, “We claim that the hyperparameter used in the ablation study is the same as the original in CO-PILOT”. Is it actually the same or is this only a claim?

Andrychowicz’17, Hindsight experience replay 
Limitations and societal impact are addressed adequately.

 I thank the authors for attempting to respond to my questions. Unfortunately, the response creates many new questions instead of answering any of the previous ones.

_"the comparison is fair."_

I am not sure I follow the argument here. The proposed method degrades significantly when the Euclidean distance is removed. In particular, it degrades much below the SORB level, showing that SORB outperforms the proposed method. The comparison in app. Fig. 6 is fair, and it shows that the proposed method is inferior to prior work. This should be made very clear in the main paper. In particular, the claims that CO-PILOT outperforms prior work on success rate and sample efficiency should be removed.

_"monotonic hyperbolic model"_

The objective specified in the response is not sound. Training with this objective will cause the predicted time cost to continuously increase since the objective is unbounded. Further, it has a hyperparameter b that does nothing (the gradient of b is 0 so it does not impact the optimization)

_"We apply a collision check to the predicted sub-goal: if it’s on the obstacle, we re-predict a new sub-goal. If necessary, we repeat the above process until the subgoal is a valid state."_

Where is this described in the paper? Eq 9 does not have this rejection step. Does this mean Eq 9 is not specified correctly?

 We appreciate your time and suggestions! We will post the experimental comparison to HER ASAP.  We address your main concerns and questions as below:

$\cdot$ A major issue with the experimental evaluation is that the method is not compared to the baselines fairly. CO-PILOT in fact uses a dense Euclidean reward to train the subgoal policy shown in Eq 12.

We only use Euclidean distance to “warm start” the planner (instead of training the RL agent) at the very beginning of training when no time cost data have been collected. Among the baselines, SGT and RRT* both use Euclidean distance over the course of training. So the comparison is fair. 

$\cdot$ What if the goal is not reached?

If the agent does not reach the sub-goal $g’$, the time-cost $c(s,g’)$ is set to \tau_{max} (L 221).

$\cdot$ At l217, The paper uses a “monotonic hyperbolic model” to predict the navigation cost c(g, g’). What is a monotonic hyperbolic model? How is it used to predict the cost?

The time-cost prediction model $c_\phi(g,g’)$ is a MLP with three hidden layers. We train it using a monotonic hyperbolic function as the optimization objective, i.e.,
$J_\phi = \frac{e^{a \cdot c_{\phi}(s,s’)} –1} {e^{a \cdot \bar{c}(s,s’)} + 1} + b$
Where, a,b are two hyper-parameters ($a=0.1, b= -1$ in our experiment), $\bar{c}(s,s’)$ is the true time cost of the agent, and $c_{\phi}(s,s’)$ is the predicted time cost. We train this prediction model to converge to the true time-cost that the RL agent needs to take from $s$ to $s’$.

#\cdot$ How can the proposed method address the sparse reward problem in goal-conditioned RL without using euclidean reward?

We provide an ablation study in Section C of the Appendix and show that CO-PILOT can still improve the performance of SAC without using the Euclidean reward. 

$\cdot$ How does the method ensure that the subgoal policy outputs valid states?

We apply a collision check to the predicted sub-goal: if it’s on the obstacle, we re-predict a new sub-goal. If necessary, we repeat the above process until the subgoal is a valid state. 
 



The paper proposes a method for hierarchical reinforcement learning where the high-level policy learns to split goal-reaching tasks into halves to form an optimal subgoal sequence of increasing level of nestedness that can be executed by the low-level policy on all levels. In this way, an effective reward shaping for low-level learning is performed aiming for increased sample efficiency.
The main contribution of the paper is the mutual learning of both hierarchy levels (called planner and RL policy) in a way that they provide a mutual curriculum from easy to hard tasks.
The low-level policy is ultimately optimized to solve the final task at once.

The strongest shortcomings are in the experimental evaluation. Both the write-up and presentation need improvements.  General:
Strong points:
- The hard tasks for the planner are the easy tasks for the low-level and vice versa
- The setup of the two levels of hierarchy, in a way they create a mutual curriculum and mutually improve.
- Algorithms are described and some ablations are performed.

Weaknesses:
- What is called a planner is actually not a planner. Planning (in the usage here as path planning) means that based on a model or other edge cost function a path is optimized over a set of possible paths (at runtime). Here the planner is just producing a single path (with increasingly finer resolution). There is not online optimization performed, it is just another RL problem where a policy is learned. So this is clearly a misnomer and confusing.
- If a real planner would be used generalization to new environment structures, such as different mazes could be expected, which would really be interesting. In this case, it is just another way to overfit to one particular environment
- the manual curriculum in Eq 12 is unsatisfactory and maybe not necessary.
- Since the paper is a mainly experimental paper, the empirical evaluation needs to be solid. Here I see too strong shortcomings. There are three envs.
1) The first is the author's own maze with a single maze layout. The illustrated path in Fig.4 is not particularly good for illustration. There should be a path from bottom-left to center-left, such that many obstacles have to be circumvented.
2) In the  Mujoco-Ant Maze the walls are ignored and the ant cannot rotate around its center if I understand correctly. This is an utterly strong simplification and makes a comparison in this standard benchmark problematic.
3) I don't understand why the walker is a good example of high-level planning. Is the terrain observable to the planner? Which space are the goals defined in? These details are missing and no code is provided
- What about a task that is akin to long-term planning with sparse rewards like a multiroom with keys and doors or similar? 

- in the checklist point 4.3, the authors declare not to share the new asset  (namely their maze env). 


Novelty:
- as far as I can tell the algorithm is sufficiently novel

Clarity and presentation:
- the language contains many typos and grammar mistakes
- the presentation of the algorithm and preliminaries need improvements. I suggest to remove some details which are standard, such as the SAC updates and focus on your part to make that clear
One part that needs improvement is the connection between Eq 5 and Eq 6. It is not clear where r(s,a|g) is coming from in Eq 6. Please be more precise.
- The fonts and details in the graphics are way too small
- Fig 4 needs a better trajectory example to make the point
- consider deferring some technical details to the supplementary and explains the experiments better
- as mentioned before: your high-level is not a planner


Details:
- l126: Define the value function.... reads weird
- l187: why nearby endpoints?
- l201: I found the usage of the cost function c for a sequence of goals and for just one segment not ideal.
- Eq 12: In your algorithm, you initialize the dataset with Euclidean distances for random goal pairs. Why do you then need this manual curriculum with alpha.
- Algo2: I think a "goal-reached" statement is missing, as I guess the agent moves on to the next goal as soon as it achieves one. Is the success radius the same as for the final goal?
- You write Mujoco env. but this makes no sense. Mujoco is the simulator that is used in 100s of environments. It is the "AntMaze" env. 
 N/A

Sparse rewards in goal-conditioned RL is a challenging problem. This paper addresses this problem by decomposing the original task of reaching the goal, to smaller tasks of reaching subgoals instead. To achieve that, they train (1) a recursive subgoal predictor that builds on top of the Sub-Goal Tree (SGT) formulation, and (2) a goal-conditioned Soft Actor Critic (SAC) agent that gets rewards when subgoals are met. These models are trained in tandem, and the main novelty of this paper is that each model makes the learning of the other model easier. They show in experiments that this method works well in navigation tasks, as well a control task of walker 2D.   I like the idea and see the value it would add to the RL community. However, the mathematical and algorithmic formulation as well as the clarity of text (structure and consistency of ideas) needs *a lot of work* in order for this paper to be publication-ready. Right now, there are some theoretical points that are still unclear to me, and I think that in order to make it understandable, this work would need a (big) revision (see my detailed comments below). Moreover, I have some issues with the experimental section - see below. 

In conclusion, despite my long review below, *I really do like this work*. It is novel and interesting and I think that in the future it will be a great paper(!!). However, currently the presentation and formulation of the ideas prevent it from being publication worthy.

Major concerns:
Formulation, correctness, and readability of the paper:
1. Starting from section 3 the textual description becomes somewhat ambiguous \ a bit unreadable \ not fully specified. Like I said before, I think a clear presentation of the subgoal ideas with accurate formulation would both make this shorter, more readable, and non-ambiguous. Specific points:
(A) I think the paragraph about path-planning (section 3.2) contains some inaccurate information. First c is not used to replace V*, it is the negative of the reward function in the common RL formulation. In your case, you define a surrogate task in which c is indeed not the inverse of r, but this is your specific usage and not the general case. Second, in path planning, the setting could be seen as a goal conditioned MDP where the action space happens to be the state space. Finally, path planning doesn't have to solve the APSP problem, at least not in a sense that it needs to identify how to optimally reach between every two states at the same time (for instance, given a start and goal states all variants of RRT build a roadmap that is specific to a *specific* start and goal).
(B) Cost function issues (starting at line 212):
1. Please explain (formally) what is a monotonic hyperbolic function.
2. Since your cost function depends on the travel of the RL agent between states, and your agent is stochastic, you get a stochastic cost function. The SGT formulation you rely on in this work does not support stochastic costs functions. I suspect that part of the reason (which you address in your work) is to make sure the RL agent is able to travel between two subsequent goals. I suggest you extend the policy gradient theorem of SGT-PG to a stochastic cost function (using your assumption of reset to the next subgoal). 
(C) Unclear test time behavior - you did not specify what happens during test time. I suspect that you plan subgoals and then feed them sequentially to the RL policy. But how do you handle the case where the agent doesn't reach the next subgoal? how do you decide when to advance to the next subgoal?
(D) Equation 11, please clarify that this is the target value for the c_phi prediction. Also, what is the loss function you use to train this regression model?
(E) Another issue with the SGT formulation, you do not differentiate between costs of different levels. For instance, consider going from s->g directly, vs. going from s->g through m as a middle point. If \pi is able to produce the optimal path between s to g, and m is a detour (if the planner is not perfect) then both costs would be listed as costs for the subgoal segment (s, g). Moreover, you are likely to see more lower level paths from s->g than higher level ones (which biases the costs towards results towards the leaf prediction). This makes no sense to aggregate all into the same cost function. I also think that in the original SGT formulation there was a level-wise distinction between value functions. If you formulate all the value function, and define a specific learning rule I am sure it would make this point much clearer. 

2. Experimental section: 
All experiments: please define the state and action spaces in each domain (since this is what \pi_p and SAC predict), and if they have some real-world interpretation that would be helpful. Please consider adding more subgoal visualization that demonstrate that the subgoals themselves are meaningful (for the test agent) for solving the ant and walker (and not just a useful way to generate a curriculum for training the agent).
To further show planning subgoals in test time make sense, could you compare (1) SAC (no co-pilot at all), (2) co pilot (full), (3) SAC (co-pilot during train only, during test the agent that trained via subgoals sees the final goal only). If (2) is better than (3), it would show that subgoals planning is an important feature even in test time. 

Scenario specific comments:

Point robot maze - I don't understand the terms you use, what are obstacles states? what does the following mean "the design of the benchmark refer to figure 2" (should be figure 4 right?)
How many train and test trajectories are you left with after rejection sampling? In your test set, are you sure you have challenging start-goal pairs (this question actually applies to all scenarios)?

Ant maze - same comments as the point robot, also:
what are the 8 dimensions? I assume these include the joint angles and the center of mass, and if so, does \pi_p predict only the (x,y,z) center of mass or also the joints? one fundamental problem in many subgoal papers is the generation of accurate \ meaningful subgoals. If you predict joint angles, how do you evaluate that the produced subgoal makes sense (not necessarily by using the proxy of how well the system acts as a whole)? If you only predict subgoals as (x,y,z) states, then this means that the subgoal generation here is not more difficult than the point robot case. It does mean that the control (SAC policy) has a harder job, and *it is* interesting that you manage to train despite the additional control difficulty. However, none of your textual description actually goes into a deep analysis of this (and I think that since this paper is very experimental, this section should be more elaborate than it is right now).

Walker (not bipedal since you have 4 legs according to the figure) - 
If I understand correctly, the contribution of this scenario is showing that co-pilot can improve dense-reward scenarios, right? but why does it make sense to do subgoal planning here (or APSP for that matter)? what are the subgoals?
Also the state here is very unclear - does it include the sensor readings? Does \pi_p imagine future sensor readings? Does it make sense to do so?How does not having global information (like you had in the mazes) impact the agent (you don't know what is going to come up next)?

General experimental issues:
(1) you seed co-pilot dataset with 50K tuples of data, don't you need to add an equivalent amount of data to the baselines?
(2) Please if possible compare against MuZero, and Hierarchical Actor-Critic.
(3) After reading Eq.6, Algorithm 2, and section 4.2 multiple times, I have to say that the details in Algorithm 2 are not clear, you can either treat every segment independently (Algo2) or not (Eq.6 and section 4.2). Please provide a clearer explanation. 
(4) Re Euclidean reward: an ablation of how much the system is impacted if the subgoals costs ($c$) remains the Euclidean distance could show how the planner depends on the SAC agent for learning. Moreover, since the state in walker is not really applicable to the Euclidean distance, how much does the system improve when the dependence on the SAC agent becomes large? 

Minor concerns \ fixes:
1. Figure 1 should contain a better textual explanation of what is in the figure.
2. Can you clarify the sentence in the related work regarding MuZero? MuZero does not require a pre-trained policy, and the planning computation improves in tandem with the policy (much like your work...) and the value function. Improving the value function can be thus seen as improving the planner (as it is used in test time during MCTS planning only). It will also be interesting to see this as a baseline (if possible). The code was released https://github.com/google-research/google-research/tree/master/muzero
3. Problem formulation section - minor edits: 
(1) You did not specify the random variable R_0. 
(2) As you are working with SAC, please change the notation of the MDP and policy to indicate a stochastic policy. 
(3) In your paper you assume APSP, thus it will be good to emphasize that the state space is large. I also think that implicitly you also assume a continuous action space (since you want the "agility" to reach a set of continuous states).
4. Section 3.2 - what do you mean by closing the loop? you didn't describe the algorithm of how to select and train the subgoals predictors!
5. Algorithm 2: I think you are missing either "s_0 \leftarrow s_\tau " (consider the current state s_0) or set s_0 to the next subgoal in line 17. As you say that you need to start state to match the origin-subgoal.
6. Where do you use reward \bar{r}? (mentioned in eq. 6) I see that the RL agent sequentially trains between subsequent subgoals.
7. Figure 4 - what is the snake robot? it is not in the description... 

General comment - the writing \ grammar in this paper could be improved. It was hard reading some sentences, and required corrections were too frequent for me to send a list of edits. Of course, I do not take this into account for my acceptance decision, but in case you want to improve the manuscript, please consider improving this aspect of the paper to make it more accessible... Also, please refrain from submitting manuscripts where *the text is not interactive*, it makes reviewing even harder, and this is part of the reason I couldn't efficiently list all grammar mistakes. Not applicable

Meta-review: The paper presents an interesting idea, and the reviewers agreed that it has potential. Unfortunately, all reviewers agreed that the writing of the paper is not clear enough, and the comparison to baselines needs to be improved. I recommend the authors to fix these issues and resubmit.

Decision: accept