Imitating Graph-Based Planning with Goal-Conditioned Policies
Authors: Junsu Kim
Younggyo Seo
Sungsoo Ahn
Kyunghwan Son
Jinwoo Shin
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement Learning
Goal-Conditioned Reinforcement Learning

Abstract: Recently, graph-based planning algorithms have gained much attention to solve goal-conditioned reinforcement learning (RL) tasks: they provide a sequence of subgoals to reach the target-goal, and the agents learn to execute subgoal-conditioned policies. However, the sample-efficiency of such RL schemes still remains a challenge, particularly for long-horizon tasks.  To address this issue, we present a simple yet effective self-imitation scheme which distills a subgoal-conditioned policy into the target-goal-conditioned policy. Our intuition here is that to reach a target-goal, an agent should pass through a subgoal, so target-goal- and subgoal- conditioned policies should be similar to each other. We also propose a novel scheme of stochastically skipping executed subgoals in a planned path, which further improves performance. Unlike prior methods that only utilize graph-based planning in an execution phase, our method transfers knowledge from a planner along with a graph into policy learning. We empirically show that our method can significantly boost the sample-efficiency of the existing goal-conditioned RL methods under various long-horizon control tasks.

Published as a conference paper at ICLR 2023
IMITATING GRAPH-BASED PLANNING WITH GOAL-CONDITIONED POLICIES
Junsu Kim', Younggyo Seo', Sungsoo Ahn², Kyunghwan , Shin
1 Korea Advanced Institute of Science and Technology (KAIST)
2 Pohang University of Science and Technology (POSTECH) (junsu. kim, younggyo... seo, kevinson9473, jinwoos] @kaist ac. sungsoo.. awn@postech.....oooooooom ac.kr
ABSTRACT
Recently, graph-based planning algorithms have gained much attention to solve goal-conditioned reinforcement learning (RI.) tasks: they provide a sequence of subgoals to reach the target-goal, and the agents learn to execute subgoal- conditioned policies. However, the sample-efficiency of such RL schemes still remains a challenge, particularly for long-horizen tasks. To address this issue, we present simple yet effective self-im self-imitation scheme which distills subgoal- conditioned policy into the reach target-goal, agent should pass through subgoal, so target-goal- and subgoal- conditioned policies should be similar to each other. We also propose a novel scheme of stochastically skipping executed subgoals in a planned path, which further improves performance. Unlike prior methods that only utilize graph-based planning an execution phase, our method transfers knowledge from planner along with a graph into policy learning. We empirically show that our method can significantly boost the sample-efficinny of the existing coal-conditioned RL methods under various long-horizon control tasks.'
policy. Our intuition here is that
INTRODUCTION
Many sequential decision making problems can be expressed as reaching given goal, e.g., navigating a walking robot (Schaul et al., 2015; Nachum et al., 2018) and fetching an object using a robot arm (Andrychowicz al., 2017). Goal-conditioned reinforcement learning (GCRL) aims to solve this problem by training goal-conditioned policy to guide an agent towards reaching the target-goal. In contrast many of other reinforcement learning frameworks, GCRL is capable of solving different problems (i.e., different goals) using a single policy.
An intriguing characteristic of GCRL its optimal substructure property; any sub-path of optimal goal-reaching path is an optimal path for its endpoint (Figure la). This implies that goal-conditioned policy is replaceable by a policy conditioned on a "subgoal" existing between the goal and the agent. Based on this insight, researchers have investigated graph-based based planning to construct goal-reaching path by (a) proposing a series of subgoals and (b) executing policies conditioned on the nearest subgoal (Savinov et al., 2018; Eysenbach et al., 2019; Huang et al., 2019). Since the nearby subgoals are casier to reach than the faraway goal, such planning improves the success ratio of the agent reaching the target-goal during sample collection.
In this paper, we aim to improve the existing GCRL algorithms to be even more faithful to the optimal substructure property. To be specific, we first incorporate the optimal substructure property into the training objective of GCRL to improve the sample collection algorithm. Next, when executing a policy, consider using all the proposed subgoals the subgoal nearest to the agent (Figure 1b).
endpoint of sub-paths instead of using just
Contribution. We present Planning-guidee self- Imitation learning for Goal-conditioned policies (PIG), a novel and generic framework that builds upon the existing GCRL frameworks that use graph-based planning. PIG consists of the following key ingredients (see Figure 2):
Code available https: httpee//ggthtuuccccee www.iihub.coom junsu- m /

---

Published as a conference paper at ICLR 2023
Previous works
Our approach
(2) If (I, ",,,,,,,,,,, 1") is an optimal 1 fe-rechhin path, all the sub-paths (b) Previous works guide the agent using a 12 -rercching sub-path. Our work are optimal for reaching 1
uses all the possible sub-paths that reach 12,1", 1,,,,,,,,
Illustration of (a) optimal substructure property and (b) sub-patho considered in previous works and our approach for guiding the training of a goal-reaching agent.
Figure 1:
Training with self-imitation: propose new training objective that encourages goal- conditioned policy to imitate the noggoal-conditionel policy. Our intuition is that policies conditioned on nearby subgoals are more likely to be accurate than the policies conditioned on a faraway goal. In particular, we consider the imitation of policies conditioned on all the subgoals proposed by the graph-based planning algorithm.
Execution with subgoal skipping: As an additional technique that fits our self-imitation loss, we also propose subgoal skipping, which randomizes subgoal proposed by the graph-based planning to further improve the sample-efficinnyy During the sample-collection stage and deployment stage, policies randomly "skip" conditioning on some the subgoals proposed by the planner when it is likely that the learned policies procedure is based on our intuition that an agent may find a better goal-reaching path by ignoring some subgoals proposed by the planner when the policy is sufficiently trained with our loss.
reach the proposed subgoals. Such
We demonstrate the effectiveness of PIG on various long-horizon continuous control tasks based on MuJoCo simulator (Todorov et al., 2012). In our experiments, PIG significantly boosts the sample- efficiency of an existing GCRL method, i.e., mapping state space (MSS) (Huang et al., 2019),² particularly in long-horizon tasks. For example, MSS + PIG achieves the success rate of 57.41 41% in Large U-shaped AntMaze environment, while MSS only achieves 19.08%. Intriguingly, we also find that the PIG-trane "ra-trined policy performs competitively even without any planner; this could be useful in some real-world scenarios where planning cost (time memory) expensive (Bency al., 2019; Qureshi et al., 2019).
2 RELATED WORK
Goal-conditioned reinforcement learning (GCRL). GCRL aims to solve multiple tasks associated with target-goals (Andrychowicz et al., 2017; Kaelbling, 1993; Schaul et al., 2015). Typically, GCRL algorithms rely on the universal value function approximator (UVFA) (Schaul et al., 2015), which is a single neural network that estimates the true value function given not just the states but also the target-goal. Furthermore, researchers have also investigated goal-exploring algorithms (Mendonca et al., 2021; Pong et al., 2020) to avoid any local optima of training the goal-conditioned policy.
Graph-based planning for GCRL. To solve long-horizon GCRL problems, rrap-baseed planning can guide the agent to condition its policy on series of subgoals that are easier to reach than the faraway target goal (Eysenbach et al., 2019; Hoang et al., 2021; Huang al., 2019; Laskin et al., 2020; Savinov et al., 2018; 8; Zhang et al., 2021). To be specific, the corresponding frameworks build a graph where nodes and edges correspond states and inter-state distances, respectively. Given shortest path between two nodes representing the current state and the target-goal, the policy conditions on a subgoal represented by subsequent node in the path.
For applying graph- planning to complex environments, recent progress has mainly been made in building graph that represents visited state space well while being scalable to large environments. For example, Huang et al. (2019) and Hoang et al. (2021) limits the number of nodes in a graph and makes nodes to cover visited state space enough by containing nodes that far from each other in terms of L2 distance or successor feature similarity, respectively. Moreover, graph sparsification via
In this paper. deployment in test phase.
the term "execution" denote both (1) the roll-out in training phase and (2) the
We note that PIG is a generic framework that can be also incorporated into any planning- based GCRL methods, other than MSS. Nevertheless, we choose MSS because it is one of the most representative GCRI. works most recent works (Hoang et al., 2021; Zhang al., 2021) could be considered as variants of MSS.
2

---

policy Published as a conference paper at ICLR 2023
Plan sabh
(a) Salf-irmitttio for training
(b) Subgoal skipping for execution
((s)
P
Goal conditindd policy
Lic
(())
1
gradient
(+.g)
Figure 2: Illustration of Planning-guidess self-Imitation learning for Goal-conditioned policies (PIG). The key ingredient of PIG is twofold: (a) self-imitation for training and (b) subgoal skipping for exccution. For (a), we distill a policy via self-imitation loss term Lere- A policy is trained using the auxiliary LpIc along with off-the-shelf actor loss. For (b), we randomize subgoal provision from a planner.
policy into the surge--ooll-oondttoned
two-way consistency (Laskin et al., 2020) or learning latent space with temporal reachability and clustering (Zhang et al., 2021) also have been proposed. They have employed graph-based planning for providing the nearest subgoal to policy at execution time, which utilizes the optimal substructure property in a limited context. In contrast, PIG aims to faithfully utilize the property both in training and execution via self seff-imitation and subgoal skipping, respectively.
Self-imitation learning for goal conditioned policies. Self-imitation learning strengthens the training signal by imitating trajectories sampled by itself (Oh et al., 2018; Ding et al., 2019; Chane- Sane et al., 2021; Ghosh et al., 2021). GoalGAII. (Ding et al., 2019) imitates goal-conditioned actions from expert demonstrations along with the goal-relabeling strategy (Andrychowicz, et al., 2017). conliconnitioned supervised learning (GCSL) (Ghosh et al., 2021) trains coal-conditioned policies via iterated supervised learning with goal-relabeling. RIS (Chane-Sale Sanee et al., 2021) makes target-goal- and subgoal- conditioned policy be similar, where the subgoal is from a high-level policy that is jointly trained with levele policy. Compared to prior works, PIG faithfully incorporates optimal substructure property with two distinct aspects: (a) graph-based planning and (b) actions from a current policy rather than past actions, where we empirically find that these two differences important for performance boost (see Section 5.3). Nevertheless, we remark that PIG is an orthogonal framework them, so applying PIG on top of them (c.g.. RIS) would be an interesting future work (e.g., leveraging both planning and high-level policy).
Distilling planning into a policy. Our idea of distilling outcomes of planner into the goal-conditioned policy is connected to prior works in the broader planning context. For example, AlphaGo Zero (Silver et al., 2017) distills the outcome of the Monte-Carla Tree Search (MCTS) planning procedure into prior policy. Similarly, SAVE (Hamrick et al., 2020) distills the MCTS outcomes into the action-value function. PIG aligns with them in that we distill into the soaeee--gll-codiiiioon policy.
PRELIMINARY: GOAL-CONDITIONES RL WITH GRAPH-BASED PLANNING
In this section, we describe the existing graph-based planning framework for goal-conditioned reinforcement learning, upon which we build our work. To this end, in Section 3.1, we describe the problem setting of GCRL. Next, in Section 3.2, we explain how to train the goal-conditioned policy using hindsight experience replay (Andrychowicz al., 2017). Finally, in Section 3.3, we explain how traph-based planning can help the agent to execute better policy. We provide the overall pipeline in Algorithm in Supplemental material A, colored as black.
3.1 PROBLEM DESCRIPTION
We formulate our control task as a finite-horizon, goal-conditioned Markov decision process (MDP)
(Sutton & Barto, 2018) tuple (S, 9, A. p, r, H) corresponding to state space S, goal space G. action space A, transition dynamics (s'ls, , a) for s, E S,a E A, reward function r (s, a, s', .g). discount factor y [0, 1), and horizon H.
Following prior works (Huang et al., 2019; Zhang et al., 2021), we consider a setup where every state be mapped into the goal space using goal mapping function go S G. Then the agent
3

---

Published as a conference paper at ICLR 2023
attempts to reach certain state associated with the target-goal g. i..., (p(s) g. For example, for maze-escaping game with continuous locomotion, each state s represents the location and velocity of the agent, while the goal g indicates certain location desired to be reached by the agent.
Typically, GCRL considers the reward function defined as follows:
lll((s) - gll2 S
r(s, a, 8'
(-1
otherwise
where ò is a pre-set threshold to determine whether the state s' from the transition dynamics p(s'|s, is close enough to the goal g. To solve GCRL, we optimize a deterministic conciconii policy π S Ex G + A maximize the expected cumulative future return Vg.z (su) = -E 7 ((r( at St+1- where denotes timestep and a, = T(t.,)) g)-
3.2 TRAINING WITH HINDSIGHT EXPERIENCE REPLAY
To train goal-conditioned policies, any off-the-shelf RL algorithm can used. Following prior works (Huang et al., 2019; Zhang et al., 2021), we use deep deterministic policy gradient (DDPG) (Lillicrap et al., 2016) as our base RL algorithm. Specifically, we train an action-value function (critic) Q with parameters & and deterministic policy (actor) with parameters 0 given replay buffer B, by optimizing the following losses:
Larstic())= ((s,,ann) Cartse(0)-BBeeeaan...e (Qp(se, ar, y²
where yr = - 70 (Q0 To (0(++1,,))))))
(0) -s). Q (5t, To (St, 9)},
(3)
where the critic Qo is a universal value function approximator (UVFA) (Schaul al., 2015) trained to estimate the coal-conditioned action-value However, it is often difficult to train UVFA because the target-goal can be far from the initial position, which makes the agents unable to receive any reward signal. To address the issue, goal-relabeling technique proposed in hindsight experience replay (HER) (Andrychowiez, et al., 2017) is widely-used for GCRL methods. The key idea of HER to reuse any trajectory ending with state supervision for reaching the goal pss).. This allows for relabelling any trajectory as success at hindsight even if the agent failed to reach the target-goal during execution.
3.3 EXECUTION WITH GRAPH-BASHE PLANNING
In prior works (Huang et al., 2019; Zhang et al., 2021). graph-based planning provides subgoal, which is a waypoint to reach a target goal when executing a policy. A planner runs on a weighted graph that abstracts visited state space.
Graph construction. The planning algorithms build a weighted directed graph H = (V, E,E where each node 1 E V C g is specified by state s visited by the agent, i.e., = (p()). For populating states, we execute the two-step process following Huang et al. (2019): (a) random sampling of fixed-sized pool from an experience replay and (b) farthest point sampling (Vassilvviskkii & Arthur, 2006) from the pool to build the final collection of landmark states. Then each edge (12,13) E E assigned for any pair of states that can be visited from one to another by a single transition in the graph. A weight d(l', (2) is an estimated distance between the two nodes, i.c., the minimum number of actions required for the agent to visit node 12 starting from L!. Given y s 1 and the reward shaped as in Equation 1, one can estimate the distance d(L' 12) as the corresponding value function -V(s' 12)= ~ (s a..2 (2) where 1 p(s²) and a
( (r((s,12) (Huang et al., 2019). Next, we link all the nodes in the graph and give weight d(-,) .) for each generated edge. Then, if weight of an edge is greater than (pre-defined) threshold, cut the edge. We provide further details of graph construction in Supplemental material D.I.
Planning-guider execution. The graph-based planning provides a policy with an emergent node to visit when executing the policy. To be specific, given graph H. state and, target goal g. we expand the graph by appending s and g, and obtain a shortest path Ty = (71 (Z'...., ,,..,, !NN such that = (ss) and N = g using a planning algorithm. Then, a policy is conditioned on a nearby subgoal 12 which is easier reach than the faraway target-goal g. This makes it easy for the agent to collect successful samples reaching the target goals, leading to an overall performance boost. Note that we re-plan for every timestep following prior works (Huang et al., 2019; Zhang et al., 2021).
4

---

Published as a conference paper at ICLR 2023
4 PLANNING-GUIDER SELF-IMITATION LEARNING FOR GCRL
In this section, we introduce a new framework, named PIG, for improving the sample-efficiency of GCRL using graph-based planning. Our framework adds two components top of the existing methods: (a) training with self-imitation and (b) execution with subgoal skipping, which highlights the generality of our concept (colored purple in Algorithm in Supplementary material A). Our main idea fully leverages the optimal substructure property; any sub-path of an optimal goal-reaching path is an optimal path for its endpoint (Figure la). In the following sections, we explain our self- imitation loss as a new training objective in Section 4.1 and subgoal skipping strategy for execution in Section 4.2, We provide an illustration of our framework in Figure
4.1 TRAINING WITH IMLFAIMITATION
Motivated by the intuition that agent should pass through a subgoal to reach a target-goal, encourage actions from target-goal- and subgoal- conditioned policy to stay close, where the subgoals nodes in a planned subgoal-path By doing so, expect that faraway goal-conditioned policy policy. Specifically, we
learns plausible actions that are produced by (closer) devise loss term Lpie given stored planned path Tg 12,,,,,.... 1e..,,,,,,, (!) and transition (s, Ty)
from a replay buffer B as follows:
Lp (0)
E (s7.)))))
Σ N 1EET,\(())
1
(TO g) SG (s, M
(4)
where SG refers to a stop-gradient operation. Namely, the goal-conditioned policy imitates behaviors of subgoal-conditionel policy. We incorporate our self-imitation loss term into the existing GCRL frameworks by plugging LPIG as an extra loss term into the original policy loss term as follows:
(()) Cactor (0) + ACpIc(()
(S)
where A is a balancing coefficient, which is pre-set hyperparameter.
One can also understand that self-imitating loss improves performance by enhancing the correctness of planning. Note that actor is used to estimate distance d between two nodes 1 12; d(t' (2) s Q Ah (s') (²) as mentioned in Section 3.3. Our self-im seff-imitating loss makes πT more accurate for even faraway goals, so it leads to the precise construction of graph. Then, planning gives more suitable subgoals for an actor in execution.
4.2 EXECUTION WITH SUBGOAL SKIPPING
As additional technique that fits self-imitation loss, propose subgoal skipping, which randomizes subgoal proposed by the graph-based planming to further improve the sample-efficiency Note that the existing graph-based planning for GCRL always provides the nearest node 12 in the searched path Tg as desired goal 1" regardless of how policy is trained. Motivated by our intuition that agent may find better goal-reaching path (i.e., short-cuts) by ignoring some of the subgoals, we propose new subgoal selecting strategy.
Our subgoal skipping is based on the following insight: when a policy for the planned subgoal and the final goal agree (small Lpıc), diversifying subgoal suggestions could help find unvisited routes. Namely, the coul-conditioned policy is likely be trustworthy final-goal- and planned-subgoal conditioned policies align because it implies that the goal-conditioned policy have propagated information quite far. Leveraging generalization capability of the trained policy, suggesting the policy with diversified subgoals rather than only the nearest subgoal could help finding better routes.
To be specific, to select the desired goal l", we start from the nearest node 12 in the planned shortest path Tg. and stochastically jump to the next node until our condition becomes unsatisfied with the following binomial probability:
P[jumpl min Lprc
where a is pre-set skipping temperature and Cpra.iatest denotes Lpre calculated at the latest parameter update. We set 1" as the final subgoal after the jumping. Intuitively, the jumping criterion is likely
5

---

Published as a conference paper at ICLR 2023
2S
(a) 2DReach (b) Reacher
(c) Pusher
(d) L-shape
(e) U-shape
(I) S-shape
(g) w-shape (h) I1-shape
Figure 3: Environments used in our experiments. In all environments, at training time, am agent starts random point, and aims to reach target goal that is set randomly. At test time for AntMaze tasks, the red point and the position of an ant indicates the target goal, and the initial point, respectively.
to jump more for a smaller Larc.tateat. which is based on the fact that the policy is likely to behave more correctly for faraway goal. As subgoals are sampled from the searched shortest path and not likely to choose farther subgoal if policy is not trustworthy for faraway subgoals, our sampled subgoals are likely to appropriate for a current policy. We describe our subgoal skipping procedure in Algorithm 2 of Supplemental material A.
5 EXPERIMENT
In this section, we design our experiments to answer the following questions:
Can PIG improve the sample-efficiency on long-horizon continuous control tasks over baselines (Figure 4)?
Can policy trained by PIG perform well even without planner at the test time (Figure 5)?
How does PIG compare another self-imitation strategy (Figure 6)?
Is the subgoal skipping effective for sample-efficiency (Figure 7)?
How does the balancing coefficient affect performance (Figure 8)?
5.1 EXPERIMENTAL SETUP
Environments. We conduct our experiments set of challenging long-horizon continuous control tasks based on MuJoCo simulator (Todorov et al., 2012). Specifically, we evaluate our framework on 2DReach, Reacher, Pusher, and {L, U, S, w, -happe AntMaze environments (see Figure 3 for the visualization of environments). In 2DReach and AntMaze environments, we use a pre-defined 2-dimensional goal space that represents the (Ir, y) position of the agent following prior works (Huang et al., 2019; Kim et al., 1021). For Reacher, the goal space is 3-dimension that represents the position of an end-effector. For Pusher, the goal space is 6-dimension that represents the positions of an end-effector and a puck. We provide more details of the environments in Supplemental material B.
Implementation. We use DDPG algorithm (Lillicrap et al., 2016) an underlying RL algorithm following the prior work (Huang al., 2019). For graph- based planner and hindsight goal- relabelling strategy, we follow the setup in MSS (Huang al., 2019). We provide more details the implementation, including the graph-construcion and hyperparameters in Supplemental material D.
Evaluation. We run 10 test episodes without an exploration factor for every 50 training episodes. For the performance metric, we report the success rate defined as the fraction of episodes where the agents succeed in reaching the target-goal within threshold. We report mean and standard deviation, which are represented as solid lines and shaded regions, respectively, over eight runs for Figure 4 and four runs for the rest of the experiments. For visual clarity, we smooth all the curves equally.
Baselines and our framework. We compare our framework with the following baselines the environments of continuous action spaces:
HER (Andrychowicz et al., 2017): This method does not use planner and trains non- hierarchical policy using a hindsight goal relabbelab strategy.
MSS (Huang et al., 2019): This method collects samples using graph- policy and trains the policy using stored transitions with goal-relabeling by HER. A graph is built via farthest point sampling (Vassilvitskii Arthur, 2006) on states stored
graph-based based planner along with
replay buffer. L (Zhang et al., 2021): When building graph, this method replaces the farthest point sampling of MSS with sameligg on learned latent space, where nodes are scattered in terms of reachability estimates.

---

Published as a conference paper at ICLR 2023
MSS PIG (Ours)
MSS
L'P
HER
HIGL
1.0
Suicess rale
Success rate
0.8
Success
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0x10
0.5+10
1,0+10
0+10
1x10
2x10
0,0x10
1.5x10
3,0x10
Environment step
Envirorment step
Environment step
(a) 2DReach
(b) Reacher
(c) Pusher
1.0
1.0
0.5
Suope9 rale
0.8
rale
0.8
Success rale
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0x10
25x10
5,0x10
10x10"
5x10
10x10"
Environment step
Environment step
Environment step
(d) L-shaped AntMaze
(e) U-shaped AntMaze
(f) Large U U-shaped AntMaze
Success rale
0.4
0.6
Success tale
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0x10
10
20=10"
30x10'
0x10
10=10"
20-10" step
30<10"
0-10"
10x10
20-10"
30w10"
Envirommntt step
Environment step
(g) S-shaped AntMaze
(h) w-shaped AntMaze
(i) 11-shaped AntMaze
Figure 4: Learning curves on various continuous control tasks as measured on the success rate. We report mean and standard deviation, which are represented solid lines and shaded regions, respectively, over eight runs. We observe that PIG significantly improves the sample-efficiency of MSS most tasks. Note that HER and HIGL perform success rate of for Large U-, S-, w-, and II- shaped AntMaze, and & performs success rate of 0 for Large U-shaped AntMaze. GCSL, performs success rate of because it does long-horizon tasks, so we compare ours with GCSL-variant that uses planner in Figure 6.
planner in execution, which boosts performance in complex
HIGL (Kim et al., 2021): This method utilizes graph-based planner to guide training high-level policy goal-conditioned hierarchical reinforcement learning. In contrast, PIG uses the planner to guide low-level policy. Comparison with HIGL evaluates the benefits of directly transferring knowledge from the planner to low-level policy without going through high-level policy.
GCSL (Ghosh et al., 2021): This method learns goal-conditioned policy via iterative supervised learning with goal-relabeling. Originally, GCSL. does not utilize graph-based planner, but compare ours with GCSL-variant that uses the planner for further investigation in Figure
For all experiments, we report the performance of PIG combined with MSS. Nevertheless, we remark that work is also compatible with other GCRL approaches because PIG does not depend on specific graph-building or planning algorithms, as can be seen in Algorithm in Supplemental material A. We provide more details about baselines in Supplemental material D.2.
5.2 COMPARATIVE EVALUATION
As shown Figure 4, applying our framework on top of the existing GCRL method, MSS + PIG, improves sample-efficiency with significant margin across various control tasks. Specifically, MSS + PIG achieves a success rate of 57.41 % large U-shaped AntMaze at environment step 10 10 while MSS performs 19.08%. We emphasize that applying PIG is more effective when the task
7

---

Published as a conference paper at ICLR 2023
1.0
1,0
Succass rele
0.8
Surcess rate
Success rale
0.8
0.6
0.6
0.4
Ayys
MSS Accss
MSS+ Acea
MS5S
MSS
M55
0.0
0.0
0,0x10
0.5x10
1,0x10"
0,0+10
1,5+10
3,0x10
0.0x10'
2.5x10"
5,0H10"
Environment step
Environment step
Environment stsp
(a) 2DReach
(b) Pusher
(c) L--shaped AntMaze
Figure 6: Ablation studies about self-imitation learning for training on (a) 2DReach, (b) Pusher, and (c) L-shaped Ant Maze with four runs. MSS + LPIC and MSS + LoCsL. refer to an algorithm that applies loss term Leic and Loest. top of MSS method, respectively; subgoal skipping is not applied. We find that our loss term Lpıe is more effective than Cecst as an auxiliary term.
more difficult; MSS PIG shows larger margin in performance in more difficult tasks (i.e., U-. S-, and w- shaped mazes rather than L-shaped mazes). Notably, we also observe that MSS + PIG outperforms L P, which shows that our method can achieve strong performance without the additional complexity of learning latent landmarks. We remark that PIG is also compatible with other GCRL approaches, including L P. as our framework is agnostic to how graphs are constructed. To further support this, provide additional experimental results that apply PIG on top of another graph-based GCRL method in Supplemental material C.I.
Also, we find that MSS PIG outperforms HIGL in Fig- ure 4. These results show that transferring knowledge from planner to low-level policy is more efficient than passing through a high-level policy. Nevertheless, one can guide both high- and low- level policy via planning, i.e., HIGI.
1.0
MSS + PIG
Success rale
0.8
MES PIG wro planner
MES planner
0.6
0.4
+ PIG. which would be interesting future work. We also remark that the overhead of applying PIG is negligible in time complexity. Specifically, both the graph-based plan- ning algorithms (MSS+PIG and MSS) spend Ih 30m for 500k steps of the 2DReach, while non-planning baseline (HER) spends 1h.
0x10
5x10"
10x10
Environment step
Figure 5: Test time success rate of PIG and MSS on U-shaped Ant Maze envi- ronment over four runs. The w/o plan- ner means that planner is not used at test time, goal is directly fed into the policy instead of a subgoal.
Reaching goal without planner at test time. To further investigate whether knowledge from graph-based planning transferred into policy, we additionally evalu- ate without the planner at the test time; in other words, the planner is only used at the training time and not anymore at the test time. Intriguingly, we find that training with PIG enables successfully reaching the target-goal even without the planner at test time. As shown in Figure 5, this supports our training scheme indeed makes the policy much stronger. Such deployment without planning could be practical in some real-worle scenarios where a planning time or memory for storing a graph matter (Bency et al., 2019; Qureshi al., 2019). We also provide experimental results with a larger maze in Supplemental material C.5.
5.3 ABLATION STUDIES
Effectiveness of our loss design. In order to empirically demonstrate that utilizing (a) the graph- based planner and (b) actions from a current policy is crucial, we compare PIG (without subgoal skipping) Garinvariant that optimizes the following auxiliary objective in conjunction with the RL objective of MSS framework:
LoCsL (.s. [o a -
that is, it encourages goal-conde coal-conditioned policy to imitate previously successful actions to reach (relabeled) goal; goal and a reward is relabeled in hindsight. execution time, we also apply a grsph-based planner GCSL-variant for fair comparison. As shown in Figure 6, PIG is more
*riiinall GCSL use only Loca, RL. loss term and does use planner in execution.
8

---

Published as a conference paper at ICLR 2023
1.0
1,0
Success rele
0.8
Surcess rate
0.8
Success rale
0.9
0.6
0.6
0.6
0.4
MBS PIG sxipping
MSS PIG with shopping
without werpping
0.2
MBS PIG skipping
MSS
MS8
MSS
0.0
0.0
0,0x10
0.5x10
1.0x10"
0
1,5+10
3,0x10
0.0x10'
2.5x10"
5,0H10"
Environment step
Environment step
Environment step
(a) 2DReach
(b) Pusher
(c) L--shaped AntMaze
Figure 7: Learning curves of PIG with and without subgoal skipping (a) 2DReach, (b) Pusher, and (c) L-shaped AntMaze tasks with four runs. PIG with subgoal skipping achieves significantly better performance than without skipping in 2DReach and Pusher.
effective than using the loss LocsL in terms of sample-efficiency due (a) knowledge transferred by planner and (b) more plausible actions from a current policy (rather than an old policy).
Subgoal skipping. We evaluate whether the proposed subgoal skipping effective in Figure 7. For 2DReach and Pusher, we observe that PIG with skipping achieves significantly better performance than without skipping. We understand this is because strong policy may find a better goal-reaching path by ignoring some of the subgoals proposed by the planner. On the other hand, we find that subgoal skipping does not provide a large gain on L-shaped Antmaze, which is a more complex environment. We conjecture that this is because learning a strong policy with nigh-dimensional state inputs of quadruped ant robots is much more difficult. Nevertheless, we believe this issue can be resolved when the base RI. algorithm is more improved. We provide more experiments related to subgoal skipping (i.e., comparison to random skipping) in Supplemental material C.2, C.3, and C.4.
Balancing coefficient X. We investigate how the balancing coef- ficient À in Equation that determines the effect of our proposed loss term LPıC affect the performance in Figure 8. We find that PIG with À E fle-3, 4] outperforms PIG with À = O, which shows the importance of the proposed loss. We also observe that too large value of X harms the performance since it incapacitates the training signal of Lactor excessively. Meanwhile, one can set the balancing coefficient À automatically in task-agnostic way, which would guide researchers when they extend our work into new environments in the future. We provide experimental results with automatic setting of À in Supplemental material C.3.
16
Success rate
to
0.2
0.0
0.0x10
2.5x10"
5,0x10
Environment step
Figure 8: Effectiveness of vary- ing balancing coefficient À on L- shaped AntMaze.
6 CONCLUSION
We present PIG. new self- ceff-improving framework that boosts the sample-efficiency in goal- conditioned RL. We remark that PIG is the first work that proposes to guide training and execute with faithfully leveraging the optimal substructure property. Our main idea is (a) distilling planned-subgoal- conditioned policies into the target-goal wnget-goal-conditonne policy and (b) skipping subgoals stochastically in execution based on our loss term. We show that PIG on top of the existing GCRL frameworks enhances sample-effciennyy with a significant margin across various control tasks. Moreover, based on our findings that policy could internalize the knowledge of a planner (e.g., reaching target-goal without planner), we expect that such a strong policy would enjoy better usage for the scenarios of transfer learning and domain generalization, which we think an interesting future direction.
Limitation. While our experiments demonstrate the PIG on top of graph-based based goal coul-conditioned RL method is effective for solving complex control tasks, we only consider the setup where the state space of an agent is (proprioceptive) compact vector (i.e., state state-based RL. following prior works (Andrychowicz et al., 2017; Huang et al., 2019; Zhang et al., 2021). In principle, PIG is applicable to environments with high-dimenniooal state spaces because our algorithmic components (self-imitation loss and subgoal skipping) do not depend on the dimensions of state spaces. It would interesting future work extend our work into more high nigh-dimensional observation space such visual inputs. We expect that combining subgoal representation learning (Nachum et al., 2019; Li et al., 2021) (orthogonal methodology to PIG) would be promising.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provide the implementation details of our method Section and Supplemental material D. We also open-source codebase.
ETHICS STATEMENT
This work would promote the research in the field of coal-conditioned RL. However, there goal- conditioned RL algorithms could be misused; for example, malicious users could develop autonomous agents that harm society by setting dangerous goal. Therefore, it is important to devise method that can take consideration of the consequence of its behaviors to a society.
ACKNOWLEDGMENTS AND DISCLOSURE OF FUNDING
We thank Sihyun Yu, Jacho Lee, Jongjin Park, Jihoon Tack, Jacyeon Won, Woomin Song. Subin Kim, and anonymous reviewers for providing helpful feedbacks and suggestions in improving paper. This work was supported by Institute of Information & communications Technology Planning & Evaluation (IIT)) grant funded by the Korea governmen((MSIT) (No. 201-00000000 Artificial Intelligence Graduate School
This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea (OV)) (No.2022-0-005533 0--0095,.eeffddiectee AI Agents with Problem-solving Capability). This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government. (MSIT) (2022R CCC1011366)
REFERENCES
Marcin Andrychowiez, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. arXiv preprint arXiv: /tt::/707 707.01145. 2017.
Mayur J Bency, Ahmed H Qureshi, and Michael C Yip. Neural path planning: Fixed time, near- optimal path generation via oracle imitation. JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2019.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioned reinforcement learning with imagined subgoals. In International Conference on Machine Learning, 2021.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. In Advances in Neural Information Processing Systems, 2019.
Ben Eysenbach, Russ R Salakhutdinov, and Sergey Levine. Search the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Manon Devin, Benjamin Eysenbach. and Sergey Levine. Learning to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021.
Jessica Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Tobias Pfaff, Theophane Weber, Lars Buesing, and Peter W. Battaglia. Combining q-learning and search with amortized value estimates. In International Conference on Learning Representations, 2020.
Christopher Hoang, Sungryull Sohn, Jongwook Choi, Wilka Carvalho, and Honglak Lee, Successor feature landmarks for long-horizon goal-conditioned reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Zhiao Huang, Fangchen Liu, and Hao Su. Mapping state space using landmarks for universal goal reaching. In Advances in Neural Information Processing Systems, 2019.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, 1993.
10

---

Published as a conference paper at ICLR 2023
Junsu Kim, Younggyo Seo, and Jinwoo Shin. Landmark -aghnarr--g-iedd subgoal generation in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2.66412.99660 2014.
Michael Laskin, Scott Emmons, Ajay Jain, Thanard Kurutach, Pieter Abbeel, and Deepak Pathak. Sparse graphical memory for robust planning. arXiv preprint arXiv:2003.cm 06417, 2020.
Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang. Learning subgoal representations with slow dynamics. In International Conference on Learning Representations, 2021.
Timothy P. Lillicrap. Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discovering and achieving goals via world models. In Advances in Neural Information Processing Systems, 2021.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In International Conference on Learming Representations, 2019.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, 2018.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement leam- ing: Challenging robotics environments and request for research. arXiv preprint 1X2.:0802200964 2018.
Vitchyr Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervupe reinforcement learning. In International Conference Machine Learning, 2020.
Ahmed H Qureshi, Anthony Simeonov, Mayur J Bency, and Michael C Yip. Motion planning networks. IEEE International Conference on Robotics and Automation (ICRA), 2019.
Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi Semi- parametric topological memory for navigation. International Conference Learning Representations, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators.
In International Conference on Machine Learning. 2015.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 2017.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American Journal of Mathematical and Management Sciences, 2003.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Sergei Vassilvitskii and David Arthur. k-means++: The advantages of careful seeding. ACM-SIAM Symposium Discrete Algorithms,, 2006.
11

---

Published as a conference paper at ICLR 2023
Lunjun Zhang, Ge Yang, and Bradly C Stadie. World model graph: Learning latent landmarks for planning. In International Conference on Machine Learning, 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
12

---

Published as a conference paper at ICLR 2023
A ALGORITHM TABLE
We provide algorithm tables that represent PIG in Algorithm and 2.
Algorithm 1 GCRL with planning PIG
Input:
Number of training episodes M, horizon H
Initialize replay buffer B <- 63.
Initialize the parameters of goal-conditioned policy To.
Initialize the parameters of action-value function Q.-
for m 1, 3, ... do
.M Reset the environment. Sample a target goal and an initial state su- for f 2......... H do
Build a graph 7 EV,E.d) d) using B.
Find the shortest subgoal-path Ty from 8, g- Find a desired subgoal 1" via Algorithm 2.
Collect a transition (st, az, rx) using Tt(s;, (*). Store the transition and the planned path Ty in B. end for
Update Qs using (critic()) of Equation 2 Update Th using Lactor (0) + AO of Equation
end for
Algorithm 2 Subgoal skipping for execution
Input: Subgoal-path = ,12,12, 12,....,% ),
the latest Cprc,nateat, normalizing constant C.
Initialize desired subgoal / 12, current index f- 2 while < N do
Sample jump according to Equation 6. if jump then
Update current index +- i
Update desired subgoal 1" + 1.. else
Break while loop.
end if
end while
Output: desired subgoal
13

---

Published as a conference paper at ICLR 2023
B ENVIRONMENT DETAILS
B.1 2DREACH
green point 2D U-shaped Maze size 15 15 aims to reach target goal represented by a red point. At each step, the agent can move within [-1,1] x [-1,1] in x and y directions.
B.2 REACHER
robotic arm aims to make its end-effector reach the target position on 3D space. The state of the arm is 17 17-dimension, including the positions, angles, and velocities of itself, and the action-space is 7-dimension. Initial point and target goal are set randomly at the start of episode both at training and test time.
B.3 PUSHER
A robotic arm aims to make puck in plane reach goal position by pushing the object. The state of the arm is 20-dimension, which same to Reacher but additionally include position of puck, and the action-space is 7-dimension. Initial point and target goal are set randomly at the start of episode both at training and test time.
B.4 ANTMAZE
quadruped ant robot is trained reach a random goal from a random location and tested under the most difficult setting for each maze. The states of ant is 30-dimension, including positions and velocities. An ant should reach the target point within 500 steps for U-shaped mazes, and 1000 steps for S-, wh, and II-shaeed mazes.
14

---

Published as a conference paper at ICLR 2023
C ADDITIONAL EXPERIMENTS
C.I APPLYING PIG ON TOP OF ANOTHER GRAPH-BASED GCRL METHOD.
Additionally, also observe that applying PIG on top of another planning-based GCRL method (i.e., I,3 rather than MSS) also demonstrates significant gains. As shown in Figure 9, PIG boost for L P U-shaped AntMaze and FetchPickAAnPPlace- (Plappert et al., 2018) These experiments further highlight that PIG is generic technique improve performance of all the graph-based planning algorithms.
0.8
1.0
(Oung)
0.6
L3P PIG (Ours)
MISS
Succass rate
0.8
0.4
0.0%
2.5x10'
5,0K10
0,0x10
2.5x10' 5,0K10" 7.5x10 10.0x10"
Environment step
Envirooment step
(a) U-shaped AntMaze
(b) FetchPickAnnPPace-v
Figure 9: Test time success rate of PIG on top of another planning- bbsed GCRL method (i.e., I,P))
in (a) U-shaped AntMaze and (b) FetchPickAndPlaceev//
C.2 COMPARISON TO ALTERNATIVES FOR SUBGOAL SKIPPING.
We compare our subgoal skipping strategy to a simple baseline: random sampling of subgoals from the planned path. As shown in the Figure 10a and 10b, find that the alternative performs close ours in 2DReach, but ours outperforms in Pusher. Developing better skipping strategy is an interesting direction to explore.
1.0
1.0
0.8
Success rate
0.8
0.8
0.6
0.4
0.4
0.2
0.2
Random
0.0
0.0x10
0.5 0.5x10
1.0k10"
0.0x10
1.5x10
3.0x10
Environment step
Environment step
2DReach
(b) Pusher
Figure 10: Ablation studies about skipping strategy. We compare our skipping strategy to an alternative one: random skipping on (a) 2DReach and (b) Pusher.
C.3 HYPERPARAMETER TUNING COST.
Our PIG inevitably introduces new hyperparameters (A and a) in addition existing algorithms. but we can use a task-agnosti strategy to choose them without any computational overhead. To be specific, one can the balancing coefficient adaptively to satisfy Lpıc 0.01 Lacteri see Figure la, llb. Next, we found that the performance of our algorithms is robust to the choice of skipping temperature or see Figure llc.
15

---

Published as a conference paper at ICLR 2023
Success nale
Suoress rale
Success rale
0.6
0.8
0.4
0.4
0.4
am1
0.2
0.2
0-0.1
NSG
MSG
0+0
6.0x10
1.8x10
3.0x10"
Environment slep
Environment step
Environment step
(a) 2DReach
(b) L-shaped AntMaze
(c) Pusher
Figure 11: Experiments with (a, b) automatic hyperparameter setting of À and (c) varying a.
C.4 EFFECT OF SUBGOAL SKIPPING EXPLORATION.
To further support our statement subgoal skipping makes an agent could collect better trajectories via promoting exploration, we quantitatively measure how diverse an agent discovers states during training depending on subgoal skipping. Specifically, we employ particle-based k-nearest neighbors (k-NN) entropy estima- tor (Singh et al., 2003) to measure how diverse collected samples are. Formally, let X be random variable whose probability den- sity function is p. and { be its N i.i.d realization. State entropy defined H(X) pp))()) (x) [log p(x:) and can estimate (((X) follows:
0.100
shipping
State Entropy
0.095
without ak poing
0.090
Environment step
Figure 12: State entropy
N Σ
K
(X) &
log
D3
12.
where NN using 2DReach environment. As shown in Figure 12, we observe that using subgoal skipping makes high state entropy: that subgoal skipping makes agent collect more diverse samples, which is likely to have more chance to include better samples.
- is the A--N of within set (x We use N 128 and K 10 for experiment
C.5 REACHING A GOAL WITHOUT A PLANNER AT TEST TIME WITH A LARGER MAZE.
We also evaluate without planner test time with large U- shaped AntMaze. As shown in Figure 13, training with PIG enables successfully reaching the target-goal even without the planner at test time even in larger environment. Intriguingly, after
purtret punter
Succosa
x 10a environment timesteps, a policy trained by our approach performs better even without access to planner test time compared to MSS, which uses a planner at test time.
Sw1d"
Environment stop
Figure 13: Test time success rate of PIG and MSS on large U- shaped Ant Maze over four runs.
16

---

Published as a conference paper at ICLR 2023
C.6 EXPERIMENTS WITH STOCHASTIC TRANSITION MODEL.
PIG, along with our graph construction technique, is applica- ble stochastic environments since our algorithmic component (self-imitation loss and subgoal skipping) and graph construction mechanism (farthest point sampling and assigning edge weights) are built on visited state spaces, regardless of transition dynamics.
1.0
M65 PIG
Sucoess rate
0.8
MSS
0.6
0.4
0.2
To empirically show that PIG effective in stochastic environ- ments, we additionally provide experimental results on stochastic L-shaped AntMaze, where gaussian noise (0,, 0.05) added to the (z, y) position of an agent at every step following setups from Zhang al. (2020); Kim et al. (2021). As shown in the Figure 14, we observe that PIG successfully solves tasks in the stochastic environment. Morcover, not only in (deterministic) L-shaped AntMaze, but also in stochastic L-shaped AntMaze, PIG shows significant gain compared to the baseline (MSS). This result supports that PIG trains a strong policy that is able to reach faraway goals more sample-efficienty than the baseline thanks to our self-imitatio loss and subgoal skipping.
0.0x10
2.5x
5,0x10"
Environment step
Figure 14: Learning curves on stochastic L-shaped AntMaze as measured on the success rate.
C.7 ABLATION STUDIES WITH MORE ENVIRONMENTS.
We provide ablation studies about self-imitation loss and subgoal skipping with more environments: Reacher and Large U-shaped AntMaze. As showin Figure 15 and 16, including our ceffimmiaation loss or subgoal skipping makes significant gains performs on par.
MSS Lyye
0.6
M55
Success rate
MSIS
Zarst
MISS
Suncess
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0x10
1x10
2x10
0x10
Ex10"
10-10"
Enviromment step
Environment step
(a) Reacher
(b) Large U-shaped AntMaze
Figure 15: Ablation sutides about self-imitation learning for training on (a) Reacher and (b) Large U-shaped AntMaze with four runs. MSS Leıs and MSS Cocst. refer an algorithm that applies loss term Lpıc and Loesı on top of MSS method, respectively; subgoal skipping is not applied. We find that our loss term Leia is more effective than Lacst. auxiliary term.
1.0
PIG shipoing
0.8
MISS
withol weipping
Success rale
MSS PIG without siepping
MSS
0.6
0.4
0x10
2x10a
5x10
10x10"
Enviromment step
Environment step
(a) Reacher
(b) Large U-shaped AntMaze
Figure 16: Learning curves of PIG with and without subgoal skipping on (a) Reacher and (b) Large U-shaped AntMaze tasks with four runs.
17

---

Published as a conference paper at ICLR 2023
C.8 EXPERIMENTS WITH EXTENDED TIMESTEPS.
To assess whether the empirical improvements are in learning speed or also in asymptotic performance, we evaluate PIG and MSS with extended timesteps (i.e., from 10 x 10 to 30 x 10a
1.0
Sucoess rate
0.8
0.6
on Large U-shaped AntMaze. As shown in the Table below, we find that PIG can improve both sample-efficiency and asymptotic performances of MSS. This shows that enhanced policy learning via information distillation from the planner can also improve the asymptotic performance.
0.4
0.2
M5S
PIG
10+10"
20x10"
30x10"
Environment step
Figure 17: Learning curves on U-shaped AntMaze as measured on the success rate.
D IMPLEMENTATION DETAILS
All of the experiments were processed using single GPU (NVIDIA TITAN Xp) and 8 CPU cores (Intel Xeon E5-2630 v4). For baselines, we employ open-source codes MSS' L, and HIGL."
D.1 GRAPH CONSTRUCTION
Collection of graph-constructing states. We follow collecting scheme of graph-construction states from Huang et al. (2019). The collection is proceeded two steps: (a) random sampling a fixed-sized pool D from experience replay and (b) farthest point sampling (FPS) (Vassilvitskii & Arthur, 2006; Huang et al., 2019) from the pool 'D to build the final collection V of graph-constructing states.
Specifically, any given time, let (s) denote the shortest distance from a state s to the closest element in current V. The set V is initialized with empty set. Then, FPS runs as follows:
Step A: Choose state s! uniformly at random from the pool D and add s' into V.
Step B: Choose the next state 8², whose ((s*) is the largest among elements in D. Add o² into
Step C: Repeat Step B until we have chosen budget for the number of nodes in graph.
The diversity of the collection is ensured by farthest point sampling. Random sampling to build fixed-sized pool makes the computational complexity of planning irrelevant the size of experience replay, of which size is IM in our experiments.
Edge connection. After collecting graph-constructing states, we complete graph by adding directed edges (Huang et al., 2019). In detail, given two nodes (2 and 22, we connect them by adding two directed edges (1' 12) & (from 1 (2) and (?²,1") E E (from 12 (t). Then we assign weights as an estimated distance d(t), 12) and d(12, (1),, respectively.
D.2 HYPERPARAMETERS
We list hyperparameters used for PIG across all environments in Table and 2.
For the baselines, we used the best hyperparameters reported in their source codes for shared environments: 2DReach of MSS and HER, Reacher and Pusher for HIGL, and AntMazes for MSS, L3P, HER, and HIGL (all). For unstudied environments in the baseline papers, have searched hyperparameters for each baseline. For example, we search shift magnitude and adjacency degree for HIGL., clipping threshold and final goal adjacency threshold for L.P and MSS, and relabeling ratio for HER. We note that for PIG, two newly introduced hyperparameters (balancing coefficient and skipping temperature a) have been searched. We would like to remark that performance
Phttps://gt htw.hh.ccomm/nnnnnnnaann///////e oor.:///onnannneooddddoodoeeed -as-a- a- a-graph
"https://ee thub. wtt.tttt.b..c.ccoeoooeeooeoeeeet thobb.comm/juussu--ii/7////
18

---

Published as a conference paper at ICLR 2023
gain by PIG have been achieved without exhaustive efforts in hyperparameter search compared to baselines. For example, the baseline MSS conducted grid search on 30 (number of landmarks) 30 (clipping threshold) values in their paper, but we searched among 5 x 4 values for PIG: {1.0, 0.1, 0.01, 0.001, 0.0001) } for À and (20, 10, 5, 1) for a.
Table I: Hyperparameters across all environments.
Hyperparameter
Value
DDPG
Optimizer
Adam (Kingma Ba, 2014)
Actor learning rate
0.0002
Critic learning rate
0.0002
Replay buffer size
1M
Number of hidden layers for actors
Number of hidden layers for crities
5
Number of hidden units per layer
400
Batch size
200
Nonlinearity
ReLU
Polyak for target network
0.99
Target update frequency per episode
3
Ratio between env vs optimization steps
Gamma
0.99
Hindsight relabelling ratio
0.8
Graph
Number soft value iteration
20
Temperature
0.9
Table 2: Hyperparameters that differ across the environments.
Hyperparameter
2DReach
Reacher
Pusher
AntMaze
Ours-specific
Balancing coefficient À
1.0 1.0
0.0001
0.1
0.001
Skipping temperature
10.0
1.0
10.0
DDPG
Initial random trajectories
2.5k
20k
20k
100k (for I.-, shaped Maze)
400k (for Large U-shaped Maze) 800k (for S-, w-, II -shaped Maze)
Hindsight relabelligg range
50
50
50
200
Action L2
0.5
0.01
0.01
0.5
Action noise
0.2
0.1
0.1
0.2
Graph
Number of nodes graph
100
80
80
400
clipping threshold for distances
4.0
4.0
4.0
38.0
19


Reviews:
[{"review_id": "l-ilcNwC64", "reviewer": "Reviewer_X3ba", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel and generic framework Planning-guided self-Imitation learning for Goal-conditioned policies (PIG) to improve the sample efficiency of goal-conditioned RL (GCRL). Empirical results show that PIG significantly boosts the performance of the existing GCRL methods under various goal-reaching tasks.", "summary_of_the_review": "The current version does not convince me to recommend acceptance\n\n\n######After Rebuttal######\n\nThe additional explanation and experiments address most of my concerns", "strength_and_weaknesses": "The paper is well-written and organized, and the proposed method is simple but seems to be effective. However, the paper may not have a high impact on reinforcement learning due to some limitations of PIG.\n\nPIG seems to be effective only for goal-reaching tasks, and all the experiments are conducted in goal-reaching tasks. Whether is PIG applied to tasks with complex reward functions? \n\nWhether is the proposed graph construction technique applied to complex environments with stochastic transition models, directed-graph-based, or high-dimensional state spaces? \n\nSubgoal skipping may induce a farther subgoal that is difficult to reach by the agent. How to guarantee that the technique can provide appropriate subgoals for the agent? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cwptposMYs", "reviewer": "Reviewer_tgwg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focusses on the setting of goal-conditioned RL where subgoals are provided by graph based planners. It introduces two algorithmic contributions:\n* a self imitation loss that is inspired by the \"optimal substructure property\" ie the fact that parts of an optimal path are themselves optimal. This idea is used to regularize the policy conditioned on a final goal towards those conditioned on the subgoal along a planner path\n* subgoal skipping: instead of always attempting to reach the first subgoal the policies sometimes directly attempts more distant subgoals along a planned path to aid with exploration.\nThe proposed innovations are primarily evaluated in a number of maze environments. Empirically the proposed approach compares favourably to baselines, especially in more complex maze environments.", "summary_of_the_review": "Nice paper with a well-motivated simple new objective that appears to lead to improved performance in goal-conditioned RL with graph-based planner guidance.", "strength_and_weaknesses": "Strengths:\n* well motivated new objective that empirically seems to improve performance\n* clearly written paper with pretty thorough evaluation\n\nWeaknesses:\n* since PIG can be applied on top of L3P it might be nice to see those results as well. \n* in some of the learning curves not all (or even no) methods seem to have converged. It would be interesting to evaluate for longer to be able to assess whether the empirical improvements are in learning speed or also in asymptotic performance.\n* the specific implementation of subgoal skipping seems a little ad-hoc. Is it obvious that this is the right choice? I think mentioning the comparison to random goal sampling along the path in the main text might be good. I would also like to see an ablation over a wider range of $\\alpha$ (fig 11).\n* It's not clear to me that any of these approaches, even with the proposed improvements would scale to higher dimensional tasks. \n* figure 8: which environment is this?\n* evaluation: are you reporting standard deviation or standard error in the mean?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PxcT7YAJnw3", "reviewer": "Reviewer_4xSo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses goal-conditioned reinforcement learning and proposes two techniques to improve existing graph-based planning methods. The first technique is to distill the outcome of the graph-based planning procedure into the goal-conditioned policy. The second technique is to randomly skip subgoals along the planned path when the agent interacts with the environment. The authors name the combination of these two techniques PIG and evaluate PIG in a set of continuous control tasks. Empirical results show that PIG outperforms strong baselines and improve performance in all tasks. The authors also provide practical recommendations on setting the hyperparameters introduced by PIG.", "summary_of_the_review": "This paper proposes two techniques to improve graph-based planning methods in goal-conditioned reinforcement learning. The proposed techniques are thoroughly evaluated and demonstrate strong empirical performance against baselines. The paper is well-written and the method is clearly presented. Thus I recommend acceptance.", "strength_and_weaknesses": "# Strength\n* The paper is well-written and easy to follow.\n* The ideas of the two techniques in PIG make sense intuitively. I especially like the idea of distilling the outcome of the planning procedure to the goal-conditioned policy by self-imitation. This technique provides a way for the agent to bootstrap its own learning.\n* The empirical evaluation is thorough and solid. PIG demonstrates strong performance against baseline methods in the experiments. The ablation studies further demonstrate the efficacy of both proposed techniques.\n* I appreciate that the discussion on limitations of PIG at the end of the paper.\n\n# Weakness\n* The authors mainly discuss related works in the goal-conditioned reinforcement learning literature. However, I believe the idea of distilling the planner's outcome into the goal-conditioned policy is connected to some works in the broader planning context. For example, AlphaGoZero [2] distills the outcome of the Monte-Carlo tree search (MCTS) planning procedure into a prior policy. Similarly, SAVE [1] distills the MCTS outcomes into the action-value function. I strongly suggest the authors to broaden the scope of the related work section and add a discussion on the aforementioned works.\n* It would be good if the ablated versions of PIG are also compared to the baseline methods. For example, by reading the numbers approximately from Figure 4 and Figure 7, it looks like PIG w/o skipping performs worse than MSS. This observation would be intriguing if true because I would expect the self-imitation learning technique itself to be beneficial.\n\n# Questions and discussion\n* In Equation (4), all intermediate subgoals are equally weighted. But the idea behind the subgoal skipping technique essentially says the opposite. I wonder if we can further improve PIG by using a similar idea as subgoal skipping and adaptively weighting the subgoals in Equation (4). I would love to hear the authors' comment.\n* Why did you choose U-shaped Ant Maze for the experiment in Figure 5 instead of the larger version of it? PIG shows the largest improvement in large U-shaped AntMaze in Figure 4. In the current Figure 5, MSS + PIG w/o planner seems to perform equally to the MSS. I am very curious to see if it can actually perform better than MSS in a more challenging environment.\n* Despite providing recommendations on how to choose $\\beta$ automatically, the values are fixed (by manually tuning I suppose?) in the experiments. Why is it?\n\n# Minor suggestions\n* In the second paragraph of Section 3.3, perhaps replace the edge notation ${l^{1}, l^{2}}$ by $<l^{1}, l^{2}>$.\n* In the fourth line in \"Evaluation\" paragraph of Section 5.1, I think you mean Figure 4, not Table 4.\n* Figure 6 appears before Figure 5. Please consider fixing it.\n\n**References**\n1. Hamrick _et al_ 2020, Combining Q-Learning and Search with Amortized Value Estimates\n2. Silver _et al_ 2017, Mastering the Game of Go without Human Knowledge", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PP83eSZDs6Y", "reviewer": "Reviewer_syAR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies goal-conditioned RL, in particular the graph-based approach, where we build a graph of subgoals to improve overall performance. Compared to previous methods, this paper introduces two improvements: 1) an additional loss term to make subgoal policies compatible with end goal policies, and 2) at execution time a stochastic subgoal skipping mechanism which makes the agent sometimes aim for further ahead subgoals in the planned path. Experiments in a range of tasks show improvement on some tasks, while being on par with baselines on other environments. ", "summary_of_the_review": "The paper is well written and clear, has extensive experiments, and introduces a method that manages to outperform baselines on most of the tasks. My biggest issue is the experimental set-up, where the authors have optimized the hyperparameters of their own algorithm, but reused hyperparameters of the baselines that were optimized for different tasks. The contribution is slightly incremental (two tweaks to a known algorithm), but still interesting. I'm on a borderline vote for this paper. ", "strength_and_weaknesses": "Strength:\n* The paper is clearly written. \n* Related work is well covered. \n* Section 3 is a very clear introduction to the overall setting and previous approaches. \n* Good ablation experiments. \n\nWeaknesses: \n* My biggest issue is with the evaluation of the baselines. You indicate that for the baselines you “used the best hyperparameters reported in their source codes”. However, the baseline papers do not study all the environments you use, and are often optimized for other environments. This can make your comparison (very) unfair. If you optimized you own hyperparameters, but left their hyperparameters untouched (which were optimized for other tasks), then this will strongly influence performance. I now miss details about this procedure, and in principle I think you should put as much hyperparameter optimization effort in the baselines as you put into your own method.\n* Subgoal skipping gets a lot of attention in your introduction and methodology, but then in the experiments it only features in Fig 7 and a single paragraph of results.\n* In the second part of your results (Fig 6 and 7) you suddenly reduce the set of environments to three. How did you choose these three (why do you show these ones, and not the other ones?). You need to tell us why these are not cherry-picked. \n* The contribution is slightly incremental. Most focus lies on the self-imitation loss. I do think this is a nice insight, but effectively it adds a single loss term to a known algorithm. It is still an interesting idea, but not totally ground-breaking. \n\nMinor: \n* Sec 2: You say that progress lies in “compressing the graph by removing redundant nodes”, but then you give examples like “semi-parametric topological memory”, “farthest point sampling” and “successor features”, which do not say anything about how you reduce the number of nodes in the subgoal graph. \n* Sec 3.3: You need to explain “farthest point sampling”.\n* Sec 4.2: * I had to read Sec 4.2 multiple times to understand your motivation for Eq 6, I think you could write this down more clearly. I guess it says that, when the policy for the current subgoal and the final goal agree (small L_PIG), the goal-conditioned policy has propagated information quite far. Therefore, we may try to jump to further ahead subgoals, because the goal-conditioned policy seems trustworthy. I think you could phrase this more explicitly. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster