Planning Immediate Landmarks of Targets for Model-Free Skill Transfer across Agents
Authors: Minghuan Liu
Zhengbang Zhu
Menghui Zhu
Yuzheng Zhuang
Weinan Zhang
Jianye HAO
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
transfer learning

Abstract: In reinforcement learning applications, agents usually need to deal with various input/output features when specified with different state and action spaces by their developers or physical restrictions, indicating re-training from scratch and considerable sample inefficiency, especially when agents follow similar solution steps to achieve tasks.
In this paper, we aim to transfer pre-trained skills to alleviate the above challenge. Specifically, we propose PILoT, i.e., Planning Immediate Landmarks of Targets. PILoT utilizes the universal decoupled policy optimization to learn a goal-conditioned state planner; then, we distill a goal-planner to plan immediate landmarks in a model-free style that can be shared among different agents. In our experiments, we show the power of PILoT on various transferring challenges, including few-shot transferring across action spaces and dynamics, from low-dimensional vector states to image inputs, from simple robot to complicated morphology; and we also illustrate PILoT provides a zero-shot transfer solution from a simple 2D navigation task to the harder Ant-Maze task.

Under review as a conference paper at ICLR 2023
PLANNING IMMEDIATE LANDMARKS OF TARGETS FOR MODEL-FREE SKILL TRANSFER ACROSS AGENTS
Anonymous authors Paper under double-bling review
ABSTRACT
In reinforcement learning applications, agents usually need to deal with various in- put/output features when specified with different state and action spaces by their developers or physical restrictions, indicating re-training from scratch and consid- erable sample inefficiency, especially when agents follow similar solution steps achieve tasks. In this paper, we aim to transfer pre-trained skills alleviate the above challenge. Specifically, we propose PILoT,, i.e., Planning Immediate Landmarks of Targets. PILoT utilizes the universal decoupled policy optimization to learn goal-conditioned state planner; then, distill goal-planner to plan immediate landmarks a model-free style that can be shared among different agents. In experiments, show the power of PILoT on various transferring challenges, including few-shot transferring across action spaces and dynamics, from low-dimensional vector states image inputs, from simple robot to plicated morphology: and we also illustrate PILoT provides a zero-shot transfer solution from a simple 2D navigation task to the harder Ant-Maze task.
INTRODUCTION
Recent progress of Reinforcement Learning (RL) has promoted considerable developments in resolving kinds of decision-making chal- lenges, such as games (Guan al., 2022), robotics (Gu et al., 2017) and even autonomous driving (Zhou et al., 2020). However, most of these work are designed for single task with a particular agent. Recently, researchers have developed various goal-conditioned reinforce- ment learning (GCRL) methods order to ob- tain generalized policy to settle group of homogencous tasks with different goals simul- taneously (Liu et al., 2022a), but still lim- ited in the same settings of environment dynam- ies/reward, and the same state/action space of the agent. Many existing solutions in the domain of Transfer RL (Zhu et al., 2020) or Meta RL (Yu et al., 2020) aim to transfer among different dy- namics/reward with the same agent, but care less for the shared knowledge across agents with different state/action spaces.
Figure I: Zero-shot transferring Ant- Ant-Maze, where the Ant agent starts from the yellow point the de- sired goal (big blue star). PIL.T provide planned im- mediate landmarks (small red points) given the tempo- ral goal (green points) and the desired goal (small blue point), learned from naive 2D maze task.
There are many motivations and scenarios encouraging us to design transferring solution among agents: a) deployed agents facing with changed observing features, for instance, non-player charac- ters (NPC) trained and updated for incremental scenes of games (Juliani et al., 2018), robots with new sensors due to hardware replacement (Bohez et al., 2017); b) agents in different morphology have to finish the same tasks (Gupta et al., 2021), such as run complicate quadruped robotic fol- lowing much simpler simulated robot (Peng al., 2018); e) improving the learning efficiency with rich and redundant observations or complicate action spaces, like transferring the knowledge from compact low-dimensional vector input to high-dimensional image features (Sun et al., 2022).

---

Under review as a conference paper at ICLR 2023
Some previous works have made progress transfer across agents single task. (Sun et al., 2022) transferred across different observation spaces with aructural-ssmille dynamics and the same action spaces via learning shared latent-space dynamics regularize the policy training. On the other hand, (Liu et al., 2022b) decouples a policy as a state planner that predicts the consecutive target state, and an inverse dynamies model that delivers action to achieve the target state, which allows transferring across different action spaces and action dynamics, but limit in the same state space and state transitions.
In this paper, we propose a more general solution for transferring the multi-task skills across agents with heterogeneous action spaces and observation space, named Planning Immediate Landmarks of Targets (PILoT). Our method works under the assumption that agents share the same goal transition to finish tasks, but without any prior knowledge of the inter-task mapping between the different state/action spaces, and agents can not interact with each other.
The whole workflow of PILoT is composed of three stages, including pre-training, distillation and transfer: 1) the pre-training stage extends the decoupled policy to train a universal state planner on simple tasks with universal decoupled policy optimization; 2) the distillation stage distills the knowledge of state planner into an immediate goal planner, which is then utilized to 3) the transfer- ring stage that plans immediate landmarks in a model-free style serving as dense rewards to improve the learning efficiency even straightforward goal guidance. Fig. 1 provides quick overview of our algorithm for zero-shot transferring on Ant-Maze. Correspondingly, we first train a decoupled policy on a simple 2D maze task to obtain universal state planner, then distill the knowledge into goal planner that predicts the immediate target goal (red points) to reach given the desired goal (blue point) and arbitrary started goal (green points). Following the guidance, the Ant controllable policy is pre-trained on a free ground without the walls can be directly deployed on the maze environment without any training. As the name suggests, we are providing immediate landmarks to guide various agents like the runway center line light on the airport guiding the flight to take off.
Comprehensive challenges are designed to examine the superiority of PILoT on the skill transfer ability, we design set of hard transferring challenges, including few-shot transfer through different action spaces and action dynamics, from din-dimesionnl vectors to image inputs, from simple robots to complicated morphology. and even zero-shot transfer. The experimental results present the learning efficiency of PILoT transferred on every tasks by outperforming various baseline methods.
PRELIMINARIES
Goal-Augmented Markov Decision Process. We consider the problem of goal-conditioned re- inforcement learning (GCRL) as a yddiscounted infinite horizon augmentd Markov decision
process (GA-MDP) M = (S.A, T, r, G, Pg. o). where S is the set of states, AA is the action space, T x A > (0, 1] is the environment dynamics function, Po : - [0, 1] is the initial state distribution, and y E [0, 1) is the discount factor. The agent makes decisions through a policy T (als): + A and receives rewards r: S x A + IR, in order to maximize its accumulated reward R R= Et_ ""r(s,,a), Additionally, G denotes the goal space w.r. tasks, Pg represents the desired goal distribution of the environment, and S - G is tractable mapping function that maps the state to a specific goal. One typical challenge in GCRL is reward sparsity, where usually the agent can only be rewarded once it reaches the goal:
rg (51, g) = (the goal is reached) = 1 1(1o((+4+1) all <)
(1)
Therefore, GCRL focuses on multi-task learning where the task variationality comes only from the difference of the reward function under the same dynamics. To shape dense reward, straightfor- ward idea is to utilizing a distance measure d between the achieved goal and the final desired goal, i.e., Fg (St, 4t,9) g) = - d((ss1 ((((((+))))))))) g). However, this reshaped reward will fail when the agent must first increase the distance to the goal before finally reaching it, especially when there are obstacles on the way to the target (Trott et al., 2019).
In our paper, we work on a deterministic environment dynamics function T, such that s' = T(s, a). and we allow redundant actions, i.e., the transition probabilities can be written as linear combination of other actions'. Formally, there exists of a state sm E S, an action an E A and a distribution defined on A \ } such that LAr(o.)) (sa)a(s'lssn)) a) da T(s'lsm a,).
2

---

Under review as a conference paper at ICLR 2023
Decoupled Policy Optimization Classical RL methods learn state-to-action mapping policy function, whose optimality is ad-hoc to a specific task. In order to free the agent to learn a high- level planning strategy that can be used for transfer, Liu et al. (2022b) proposed Decoupled Policy Optimization (DePO) which decoupled the policy structure by a state transition planner and an inverse dynamics model as:
T T-|s) - J. (sd ds' = (E [(-1s, 81]
(2)
To optimize the decoupled policy, DePO first optimizes the inverse dynamics model via supervised learning, and then performs policy gradient assuming a fixed but locally accurate inverse dynamics function. DePO provides way of planning without training state planner of DePO pre-trained on simple tasks can be further transferred to agents with various action spaces or dynamics. However, as noted below, the transferring ability of DePO limits in the same state space transition. In this paper, we aims to derive more generalized skill transferring solution utilizing the common latent goal space shared among tasks and agents.
environment dynamics model. The
3
TRANSFER ACROSS AGENTS
S
+++
St
ar
S1+]
Source MDP
Sr
Srt
h
g1
c1+
Target
MDP
a
S
S
a
State Transilion
(b) Shared Latent State Transition
(c) Shared Goal Transition
Figure 2: Comparison of the different assumption for transferring across agents of previous works (Liu et al., 2022b; Sun et al., 2022) and PILoT. (a) (Liu et al., 2022b) allows for transferring across action spaces and but ask the same state space and state transition. (b) (Sun et al., 2022) transfers across different state spaces but require there exists shared latent state space and dynamics. (c) PILoT provides generalized transferring ability for both state space and action space, but asks for shared underlying latent goal transition.
In this section, we explain the problem setup of transfer across agents. Formally, we pre-train and learn knowledge from source GA-MDP M = (S,A, T, Po,, T.G.Pp. σ) and transfer to target GA-MDP M = (8, A, Po. F, y, G, pg: Φ). Here we allow significant difference between the state spaces S, Š, and action spaces A, A. Therefore, both the input / output shapes of the source policy are totally different from the target one and therefore it is challenging to transfer a shared knowledge.
To accomplish the objective of transfer, prior works always make assumptions about the shared structure (Fig. 2). For example, (Sun et al., 2022) proposed to transfer across significantly different state spaces with the same action space and similar structure between dynamics, i.e., a mapping between the source and target state spaces exists such that the transition dynamics shares between two tasks under the mapping. In comparison, (Liu et al., 2022b) pay attention on transferring across action spaces under the same state space and the same state transitions, i.e., action mapping be- tween the source and target exists such that the transition dynamics shares between two tasks under the mapping. In this paper, take more general assumption by only require agents have shared goal transition, and allow transferring across different observation spaces and action spaces. We ar- gue that this is reasonable requirement in real world since for tasks like robot navigation, different robot agents can share global positioning system constructed by techniques like SLAM that allows them to quickly figure out their 3D position the world. Formally, the assumption corresponds to:
Assumption 1. There exists a function f A - Ā such that Vs, E S, Va E A, and 38, 5 5, 3ā Á.
T(s'ss, 7(8' f(a),, f(a)))))))ssaa,, r(s,a, a(B,ā, ((s) 3(5), =
Here @ is usually a mony-to-one mapping, such as an achieved position or the velocity of a robot. can be any function, like many-to-one mapping, where several target actions relate the same
3

---

Under review as a conference paper at ICLR 2023
Pretraining Stage
Source Environment
Goal-cond Planner
Inverse Dynamic Model
3. Transferring Stage
Target Environment
9t
Goal
Policy
Planner
Bonus
2. Distillation Stage
Figure Overview universal decoupled policy optimization (PILoT) framework. PILoT leverages
transferring- by--be--ppp-ee-e t--pe-training process, including the stage of pre-training, distillation and transferring.
source action; or one-to one-to-many mapping: or non-surjective, where there exists source action that does not correspond to any target action.
4 PLANNING IMMEDIATE LANDMARKS OF TARGETS
In this section, we introduce our generalized multi-task skills transfer solution with the proposed Planning Immediate Landmarks of Targets (PILoT) framework. First, demonstrate how derive the training procedure of a universal decoupled policy structure for multiple tasks in the pre- training stage; then, characterize the distillation stage which obtain goal planner for providing the informative reward bonus or zero-shot guidance for the transferring stage. An overview of the method is shown in Fig. and we list the step-by-step algorithm in Algo.
4.1 UNIVERSAL DECOUPLED POLICY OPTIMIZATION
In order to derive a generally transferable solution, we first extend the decoupled policy struc- ture (Liu et al., 2022b) into a goal-conditioned form. Formally, we decouple the goal-conditioned policy π [alss,)) g*)
as:
(als,,g') - -f (") s') Egghh (8)).g))) (1-1s,55)
(3)
where gf is the target goal, h goal-conditioned state planner. Approximating the planner by neural networks (NNs), we can further apply the reparameterization trick and bypass explicitly computing the integral over 8' as
8 s h(e;,,g")) (uas, g") = E-- [Ma]s, h(e;; s, s.g')))
(4)
where is an input noise vector sampled from some fixed distribution, like Gaussian. The inverse dynamics I should serve as control module known in advance for reaching the target predicted by the planner. When it must be learned from scratch, we can choose to minimize the divergence (for example, KI.) between the inverse dynamics of a sampling policy ws and the d-parameterized function T i.e.,
(5) is worth noting that the model is only responsible and accurate for states encountered by the current policy instead of the overall state space. In result, the inverse dynamics model is updated every time before updating the policy.
min L = (ss Di(Ang (als, 8 (')ls,, s'))],
To update the decoupled policy, particularly, the goal-conditioned state planner ht., given that the inverse dynamics to be an accurate local control module for the current policy and the inverse dy- namies function is static when optimizing the policy function, adopt the decoupled policy gradient (DePG) as derived in (Liu et al., 2022b):
Q(s. T(ass
V 4 L F(s.a)rm.g''pp..
(6)
4

---

Under review as a conference paper at ICLR 2023
which be seen taking the knowledge of the inverse dynamies about the action to optimize the planner by a prediction error As' = a [a]s,, h(e;s)) where a is the learning rate. However, (Liu et al., 2022b) additionally pointed that there exists the problem of agnostic gradients that the optimization direction is not always lead to a legal state transition. To alleviate the challenge, they proposed calibrated decoupled policy gradient to ensure the state planner from predicting infeasible state transition. In this paper, we turn to a simpler additional objective for constraints, i.e., we maximize the probability of predicting the legal transitions that are sampled by the current policy, which is demonstrated to have a similar performance in (Liu et al., 2022b):
max (ss,s) ( (ss(ss,,, [s,g")]
Therefore, the overall gradient for updating the planner becomes:
Q(s,a.g) (V, Tals, ha (ooeer..d))Vooeeesssss "VV AVoho(Gs,))) <(a|,,9")
VoL ""
(8)
where is the hyperparamer for trading off the constraint. Note that such decoupled learning scheme also allows incorporating various relabeling strategy further improving the sample effi- ciency, such as HER (Andrychowicz et al., 2017).
4.2 GoAL PLANNER DISTILLATION
In order to transfer the knowledge to new settings, we leverage the shared latent goal space and distill goal planner from the condiiondi state planner, i.e., want predict the consecutive goal given the current goal and the target goal. Formally, we aims to obtain a w- w-paraamtterized function fu (g')gg (o'lggg"), where g' is the next goal to achieve, 9 = Φ(s) is the current goal the agent is achieved, and g is the target. This can be achieved by treating the state planner hnp (s'|s, g²) as the teacher, and fa (g' (9,99"))) becomes the student. The objective of the distillation constructed an MLE loss:
V max BenaxEg-.e g. "h [f (9. where g = c(s). o(3))
where B is the replay buffer, & is the mapping function that translates a state to a specific goal. conduct goal planning without training and querying an
With the distilled goal planner, we environment dynamics model as in Zhu et al. (2021); Chua et al. (2018).
4.3 TRANSFER MULTI-TASK KNOWLEDGE ACROSS AGENTS
typical challenge for GCRL is the rather sparse reward function, and simply utilizing the Euclid distance of the final goal and the current achieved goal can lead to additional sub-optimal problems. To this end, with PILoT having been distilled for acquiring plannable goal transitions, is natural to construct reward function (or bonus) leveraging the difference between the intermediate goals to reach and the goal that actually achieves for transferring. In particular, when the agent aims to go 9, with the current achieved goal g, we exploit the distilled planner fw from PILo'T provide reward as similarity of goals:
((8') g
(ss,a.g)= (((('')lll@)) where T((s ~ folll.
(10)
Note that we avoid the different scale problem among different agents by using the form of cosine distance. Thereafter, we can actually transfer to totally different agent. For example, we can learn a locomotion task from a easily controllable robot, and then transfer the knowledge to a complex can learn from
one with more joints which is hard to learn directly from the sparse rewards; a low-dimensional ram-based agent and then transfer on high-dimensional image inputs. To verify the effectiveness of PILoT, we design various of transferring settings in Section 6.
RELATED WORK
Goal-conditioned RL. Our work lies in the formulation of goal-conditioned reinforcement learn- ing (GCRL). The existence of goals, which can be explained as skills, tasks, or targets, making possible for our skill transfer across various agents with different state and action space. In the
5

---

Under review as a conference paper at ICLR 2023
literature of GCRL, researchers focus on alleviating the challenges in learning efficiency and gener- alization ability, from the perspective of optimization Trott et al. (2019); Ghosh et al. (2021); Zhu et al. (2021), generating selecting sub-goal, Florensa et al. (2018); Pitis et al. (2020) and relabel- ing Andrychowicz et al. (2017); Zhu et al. (2021). A comprehensive GCRL survey can be further referred to Liu et al. (2022a). In these works, the goal given to the policy function is either provided by the environment or proposed by a learned function. In comparison, in our paper, the proposed UDPO algorithm learns the next target states in an end-to-end manner which can be further distilled into a goal planner that can be used to propose the next target goals.
Hierarchical reinforcement learning, The framework of UDPO resembles Hierarchical Rein- forcement Learning (HRL) structures, where the state planner plays like high-level policy and the inverse dynamics as the low-level policy. Typical paradigm of HRL. trains the high-level policy using environment rewards predict sub-goals (or called options) that the low-level policy should achieve, and learn the low-level policy using handcrafted goal-reaching rewards to provide the ac- tion and interacts with the environment. Generally, most of works provided the sub-goals/ options by the high-level policy are lied in a learned latent space (Konidaris & Barto, 2007; Heess et al., 2016; Kulkarni et al., 2016; Vezhnevets et al., 2017: Zhang et al., 2022), keeping it for a fixed timesteps (Nachum et al., 2018; Vezhnevets et al., 2017) or learn to change the option (Zhang & Whiteson, 2019; Bacon et al., 2017). On the contrary, Nachum ct al. (2018) and Kim et al. (2021) both predicted sub-goals in the raw form, while still training the high-level and low-level policies with separate objectives. As for Nachum et al. (2018), they trained the high-level policy in an off- policy manner; and for Kim et al. (2021), which focused on conditondi HRL tasks ours, they sampled and selected specific landmarks according to some principles, and asked the high-level policy to learn predict those landmarks. Furthermore, their only sampled goal from the high- level policy for a fixed steps, otherwise using a pre-defined goal transition process. Like UDPO, Li al. (2020) optimized the two-level hierarchical policy in an end-to-end way, with a latent skill fixed for e timesteps. The main contribution of HRL works concentrate on improving the learning efficiency on complicated tasks, yet UDPO aims to obtain every next targets for efficient transfer.
Transferable RL. Before work, there are a few works have investigated transferable RL. In this endeavor, Srinivas et al. (2018) proposed to transfer an encoder learned in the source tasks, which maps the visual observation to a latent representation. When transfer to target tasks / agents, the latent distance from the goal image to the current observation is used to construct an obstacles- aware reward function. To transfer across tasks, Barreto al. (2017); Borsa et al. (2018) utilized success features based on strong assumptions about the reward formulation, that decouples the in- formation about the dynamics and the rewards into separate components, so that only the relevant module need to be retrained when the task changes. In order to reuse the policy, Heess et al. (2016) learned a bi-level policy structure, composed of a low-level nomain-ggneral controller and high-level task-specific controller. They transfer the low-level controller on the target tasks while retraining the high-level onc. On the other hand, Liu et al. (2022b) decoupled the policy as a state planner and an inverse dynamic model, and shown that the high-level state planner can be transferred to agents with different action spaces. For generalizing and transferring across modular robots' morphology, Gupta et al. (2017) tried learning invariant visual features. Wang et al. (2018) and Huang al. (2020) both exploited graph structure present in the agent's morphology, and proposed specific structured poli- cies for learning the correlations between agent's components, while Gupta et al. (2021) proposed a transformer-based policy to model the morphology representation. Theses works with such struc- tured policies, although can generalize to agents with unseen morphology, limit in modular robots with specific representation of the state space. Similar to PILOT, Sun et al. (2022) proposed to trans- fer across agents with different state spaces, which is done with a latent dynamics model trained on the source task. On the target tasks, the pre-trained dynamics model is transferred as model- based regularizer for improving the learning efficiency. In comparison, PILoT can do effectively multi-tasks skill transfer to agents with different structures, state spaces and action spaces. First, we separate the state spaces and action spaces in the decoupled policy structure, and the state predictor can be transferred to target tasks with different actions spaces. Furthermore, we distill planner in the goal space, which allows for planning consecutive target goals as transferable skills to different agent structures and state spaces.

---

Under review as a conference paper at ICLR 2023
EXPERIMENTS
We conduct a set of transferring challenges to examine the skill transfer capacity of PILoT.
6.1 EXPERIMENTAL. SETUPS
Environments, tasks and agents. a) Fetch-Reach. The agent controls the robot arm to reach a target position. b) Fetch-Push. The robotic arm controlled to push block to target position. c) Reacher. Controling a multi-joint reacher robot to reach a target position. d) Point-Locomotion. The agent controls an point robot move target position freely. Ant-Locomotion. The agent controls an ant robot to move to a target position freely. f) 2D-Maze. The agent controls a mass point in a U-shape maze to reach target position. g) Ant-Maze. The agent controls an robot move to target position in a U-shape maze.
Recch
Fetch-Puss
Reacher
2
10
(d) Pocot-Locomotion
(e) Ant-Locomotion
(I) 2D-Maze
(g) Ant-Maze
Figure 4: Illustration of tested environments.
Designed transferring challenges. 1) Few-shot transfer to high-dimensional action spaces: take Fetch-Reach and Fetch-Push as the source tasks, and change the gravity, action dynamics and spaces to get the target tasks while sharing the same state space and state transition (see Ap- pendix B.2 for details). 2) Few-shot transfer to a complicate morphology:
take Point robot with 6 observation dimensions as the source task, and Ant with 29 observation dimensions as the target task. Few-shot transfer from vector to image observations: We take Fetch-Reach and Reacher with low-dimensional vector state as the source task, and the same environment with high- dimensional image observations as the target task. 4) Zero-shot transfer: we take 2D-Maze and Ant-Locomoiin as the source tasks, which are used to train goal planner and local controller separately; then we combine the knowledge the goal planner and the local controller to master the task of Ant-Maze without any further training. All of these tasks varies in either in the state space in the action space, but pair of tasks must share the same goal space and transition, making it pos- sible transfer useful knowledge. In the sequal, show different usages of PILoT on transferring such knowledge in different challenges.
Implementation and baselines. We choose several classical and recent representative works as our comparable baselines, for both source tasks and target tasks. For source tasks we want to com- pare the learning performance from scratch of the proposed UDPO algorithm with: i) Hindsight Experience Replay (HER) (Andrychowicz et al., 2017), classical GCRL algorithm that trains coal-conditioned policy by relabeling the target goals as the samples appeared in future of the same trajectory for particular state-action sample, which is also used as the basic strategy to train the UDPO policy; ii) Hlerarchical reinforcement learning Guided by Landmarks (HIGL) Kim al. (2021), HRL. algorithm that utilizes high-level policy propose landmark states for the

---

Under review as a conference paper at ICLR 2023
Sourcir Ferch-Besch
FeichsPaash
Sourcer Beacher
Source: 2D-Mane
Sourcer Puintilacaaatioon
Rake
Bade
0
94
0.2 1.0
2520c4
2.0 3/0 503.0
16
24
1245
25 253004
10 20 10500
steps
steps
steps
steps
steps
HFR (PE
HFB
MEGA
Contractive RJ
Hygh-Omm-kttio:: Feich-Reach
Highh-iimAkciio:: FatthsPash
Vee-it-mmagge Fetch-lteach
Vec-ta-image: Reather
Comgles-Morst: Antzcccom.ion
0.6
1.0
1.3
Succiss Rude
Ruka
0.8
02
0.4
0.4
Ateps
steps
steps
steps
steps
Figure 5: Training curves four source tasks and four target tasks. High-Dim-Act high-dimensional action space. Hi -igh-Dir-Actt Vec-to- Image: fow-shot transferring from vector to image states. Complex Moppexx-Morpph few-shot transfer to different morphology. UDPO denotes the learning algorithm proposed in Section 4.1. HER (PILoT) denotes HER with the transferring rewards provided by the proposed PILoT solution.
Din--iim------ Actions ion: few-shot transfer few-shot transferring to high-dimensional action space.
low-level policy to explore. As for the target tasks, we aim show the transferring ability of PILoT from source tasks to target tasks, except HER and HIGL, we also compare a recent perl-peeformed GCRL algorithm, Maximum Entropy Gain Exploration (MEGA) (Pitis al., 2020), which pro- posed to enhance the exploration coverage of the goal space by sampling goals that maximize the entropy of past achieved goals; also, very recent algorithm, Contrastive RT (Eysenbach et al., 2022), which design a contrastive representation learning solution for GCRL. Noted that contrastive RL requires the goal space the same as the state space (e.g., both are images). For each baseline algorithm, either take their suggested hyperparameters, or try best to tune important ones.
In our transferring experiments, our PILoT solution first trains the decoupled policy by UDPO with HER's relabeling strategy for all source tasks; then, distill the goal planner and generate dense reward signals to train a normal policy by HER in the target tasks, denoted as HER (PILoTT).
6.2 RESULTS AND ANALYSIS
In the main context, we focus on presenting the training results on source tasks and the transferring results on target tasks. Additionally, we leave in depth analysis, ablation studies and hyperparameter choices in the Appendix C.
Learning in the source tasks. We first train UDPO four source tasks, and show the training curve Fig. 5. Compared with HER that is trained using a normal policy, we observe that UDPO achieves similar performance, and sometimes it behaves better efficiency. For comparing UDPO with HRL methods, we also include HIGL on these source tasks. To our surprise, HIGL performs badly or even fails in many tasks, indicating its sensitivity by their landmarks sampling strategy. In Appendix C.1 we further illustrate the MSE error between the planned state and the real state that the agent actually achieved, with the visualization of the planning states, demonstrating that UDPO has a great planning ability under goal-conditioned challenges.
Few-shot transferring to high-dimensional action spaces. The Sattt-Dii-Acttt transfer- ring challenge requires the agent to generalize its state planner to various action spaces and action dynamics. In our design, the target task has higher action dimension with different dynamics (see Appendix B.2 for details), making the task hard to learn from scratch. As shown in Fig. 5, on the target Fetch-Reach and Fetch-Push, HER is much more struggle to learn well as it is in the source tasks, and all GCRL, HRL and contrastive RL baselines can only learn with much more samples or even fail. For PILOT, since the source and the target tasks share the same state space and state transition, we transfer the goal-conditioned state planner and only have to train the inverse dy- namics from scratch. In result, by simply augmented HER with additional transferring reward, HER (PILoT)) shows impressive efficiency advantage for transferring the shared knowledge to tasks.
8

---

Under review as a conference paper at ICLR 2023
Few-shot transfer from vector to image states. The Vec-to-Image transferring challenge learns nigh-dimensional visual observation input based agents guided by the planned goals learned from the low-level vector input. From Fig. 5, we can also learn that using the transferring reward of PILoT, HER (PILoT)) achieves the best efficiency and final performance with much less samples on the two given tasks, compared with various baselines that are designed with complicated techniques.
Few-shot transferring to different morphology. We further test the Complex-Morpe transfer- ring challenge that requires to distill the locomotion knowledge from a simple Point robot to a much more complex Ant robot. The learning results from Fig. 5 again indicate impressive performance of HER (PILoT)), while we surprisingly find that MEGA, HIGI. and contrastive RI. all can hardly learn feasible solutions on this task, even worse than HER. In our further comparison, we find that this tested task is actually more hard to determine the success by requiring the agent to reach at a very close distance (i.e., less than 0.1, which can be further referred to Appendix B). In addition, the reason why MEGA fails to reach a good performance like the other tasks can be attribute to its dependency on an exploration strategy, which always chooses rarely achieved goals measured by its lowest density. This helps a lot in exploration when and the target goals' distribution is dense, like Ant-Maze. However, when the goals are scattered in Ant-Locomotion, the agent has explore a wide range of the goal space, which may lead MEGA's exploration strategy to be inefficient. On comparison, PILoT shows that, despite the task is difficult or the target goals are hard-explored. as long as we can transfer necessary knowledge from similar source tasks, agents can learn the skills quickly in few interactions.
Ant-Maze
Zero-shot transfer for different layouts. Finally, as we find the intermediate goals provided by the goal planner is accurate enough for every step that the agent encounters, we turn to a intuitive and interesting zero- shot knowledge transfer experiment for different map layouts. Specifically, we try to learn the solution on Ant-Maze, as shown in Fig. 4, which a eapl-exploring tasks since the wall between the starting point and the target position requires the agent to first increase the distance to the goal before finally reaching it. As Fig. 6 illustrates, simply deploying HER fails. With sufficient exploration, all recent GCRL, HRL and contrastive RL baselines can learn a feasible solution after a quite large times of sampling. However, since PILoT provide way of distilling the goal transition knowledge from a much simpler tasks, i.e., the 2D-Maze task, we take the intermediate goals as short guidance for the Ant-Locomotion policy pre-trained in Fig. 5. Note that the mncomoccootion policy provides the ability to reach arbitrary goals within the map. In this way, PILoT achieves zero-shot transfer performance without any sampling. This shows a promising disentangiement of goal planner and motion controler for resolving complex tasks.
1.0
HER
MEGA
HGGL
Success Rate
HER(PILOTI
0.6
0.4
0.2
1.0
2.0
3.0
4.0
5.0e6
steps
Figure 6: Training and zero-shot transferring curves on Ant-Maze task across 10 seeds.
CONCLUSION AND LIMITATION
In this paper, we provide general solution for skill transferring across various agents. In particular, We propose PILoT,, i.e., Planning Immediate Landmarks of Targets. First, PILoT utilizes and extends a decoupled policy structure to learn a goal-conditioned state planner by universal decoupled policy optimization; then, a goal-planner is distilled to plan immediate landmarks in a model-free style that can be shared among different agents. To validate our proposal, we further design kinds of transferring challenges and show different usages of PILOT, such as few-shot transferring across different action spaces and dynamics, from low-dimensional vector states to image inputs, from simple robot to complicated morphology: we also show promising case of zero-shot transferring on the harder Ant-Maze task. However, we find that the proposed PILoT solution mainly limited in those tasks that have clear goal transitions that can be easily distilled, such as navigation tasks; on the contrary, for those tasks which takes the positions of objects as goals, it will be much harder to transfer the knowledge since the goals are always static when agents do not touch the objects. We leave those kinds of tasks as future works.
9

---

Under review as a conference paper at ICLR 2023
8 ETHICS STATEMENT
This submission does not violate any ethics concern and adheres and acknowledge the ICLR Code of Ethics.
REPRODUCIBILIIY STATEMENT
All experiments included in this paper conducted over 5 random seeds for stability and reliability. The algorithm outline is included in Appendix A, the hyperparameters are included Appendix B.4 and the implementation details are include in Appendix B.3. We promise to release code to public after publication.
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in Neural Information Processing Systems, volume 30, 2017.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer reinforcement learning. In Advances in Neural Information Processing Systems, volume 30, 2017.
Steven Bohez, Tim Verbelen, Elias De Coninck, Bert Vankeirsbilck, Pieter Simoens, and Bart Dhoedt. Sensor fusion for robot control through deep reinforcement learning. In International Conference Intelligent Robots and Systems, pp. 2365-2370, Icee, 2017.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado Van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint arXiv:1812.07626. 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, volume 31, 2018.
Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learn- ing as goal-conditioned reinforcement learning. arXiv preprint arXiv:2206.075663 2022.
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In International Conference Machine Learning, pp. 1515- 1528. PMLR, 2018.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning reach goals via iterated supervised learning. International Conference on Learning Representations, 2021.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In International Conference Robotics and Automation, pp. 3396. IEEE, 2017.
Yang Guan, Minghuan Liu, Weijun Hong, Weinan Zhang, Fei Fang, Guangjun Zeng, and Yue Lin. Perfectdou: Dominating doudizhu with perfect information distillation. arXiv preprint 0rXiv:2203.164000 2022.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning. arXiv preprint 1rXiv:1703.02944 2017.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In International Conference Learning Representations, 2021.
10

---

Under review as a conference paper at ICLR 2023
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv:6610.05182, 2016.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455- 4464. PMLR, 2020.
Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelli- gent agents. arXiv preprint
02627,, 2018.
Junsu Kim, Younggyo Seo, and Jinwoo Shin. mondmark-guideee subgoal generation in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pp- 28336-28349, 2021.
George Dimitri Konidaris and Andrew G Barto. Building portable options: Skill transfer in rein- forcement learning. In International Joint Conference Artificial Intelligence, volume 7, pp. 895-900, 2007.
Tejas D Kulkamni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in Neural Information Processing Systems, 29, 2016.
Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hier- archical reinforcement learning. International Conference on Learning Representations, 2020.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. In International Joint Conference on Artificial Intelligence, 2022a.
Minghuan Liu, Zhengbang Zhu, Yuzheng Zhuang, Weinan Zhang, Jianye Hao, Yong Yu, and Jun Wang. Plan your target and learn your skills: Transferable state-only imitation learning via de- coupled policy optimization. In International Conference on Machine Learning, volume 162, pp- 14173-14196 14196. PMLR, 2022b.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Xue Bin Peng. Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-based character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Learning, pp. 7750-7761. PMLR, 2020.
Aravind Srinivas, Allan Jabri, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Universal plan- ning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pp.
PMLR, 2018
Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across ob- servation feature spaces via model-basee regularization. In International Conference on Learning Representations, 2022.
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your distance: Soly- ing sparse reward tasks using self-balancing shaped rewards. In Advances in Neural Information Processing Systems, 2019.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. 3540-3549. PMLR, 2017.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International Conference on Learning Representations, 2018.
11

---

Under review as a conference paper at ICLR 2023
Tianhe Yu. Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pp. 1094-1100. PMLR, 2020.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. In International Conference on Learning Representations, 2022.
Shangtong Zhang and Shimon Whiteson. Dac: The double actor-critic architecture for learning options. Advances in Neural Information Processing Systems, volume 32, 2019.
Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Mont- gomery Alban, Iman Fadakar, Zheng Chen, al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. arXiv preprint arXiv:2010.09776, 2020.
Menghui Zhu, Minghuan Liu, Jian Shen, Zhicheng Zhang, Sheng Chen, Weinan Zhang, Deheng Yc, Yong Yu, Qiang Fu, and Wei Yang. Mapgo: Model-assisted policy optimization for goal-oriented tasks. In International Joint Conference on Artificial Intelligence, 2021.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.07880 2020.
12

---

Under review as a conference paper at ICLR 2023
Appendices
A ALGORITHM OUTLINE
Algorithm Planning Immediate Landmarks of Targets (PILoT)
1: Source Task Input: Empty replay buffer B,, state planner hig, inverse dynamics model I and
goal planner sa-
2: Target Task Input: Empty replay buffer B,, goal planner fw, policy ne.
Pre-training stage trains UDPO on source tasks.
3: for == 0, 2, do
4: Collect trajectories
{(s, a, s'. done)) using current
policy
π
Ee- [Laaass ho ()) and store in B Sample (x, 8,, -)),r) (,,,,8,99;r)) ~ B.
5:
6: if learn inverse dynamics function then
7:
repeat
8:
Update by L' (Eq. (5)) until Converged
9:
end
Distillation stage: distill goal planner from state planner.
11:
Update w by V. (Eq, (9))
12:
end for
Transfer stage trains HER (PILoT) on target tasks.
13: for k 0, 1,2, do
14:
Collect trajectories (ssa,, ,',",,"""""" done)) using current policy my
15:
Supplement the reward with additional bonus following Eq. (10):
r=r + rss, a, g') where ÿ' (
16:
Store {(s, a, s', s',g', gg.r,, done)} in B,
Sample ((s, a, s', s','', r, done)} B.
18:
Learn by HER
end for
B EXPERIMENT SETTINGS
B. ENVIRONMENTS
We list important features of the tested environments as in Tab. 1. Note that the Goal Reaching Distance is rather important to decide the difficulty of the tasks, so we carefully choose them to meet the most of the current works.
Environment Name
Obs. Type
Obs. Dim
Act. Dim
Goal Dim
Goal Reaching Distance
Episode Length
Fetch-Reaa Reach
Vector
10
3
Fetch-Push
Vector
25
0.05
Fetch-Reach ReacR-Ricchhhhhee Dim
Vector
10
3
0.05
50
Fetch-Push- Push-High-
Vector
25
3
Fetch-Reach- Reach-Image
Image
64%64*3
3 0.05
Reacher
Vector
11
2
0.02
50
Rencher-Imagg Image
Image
64%64**
2
0.02
Loconotiono
Vector
2
0.1
500
Ant-Locomotion
Vector
29
2
0.1
100
2D-Maze
Vector
2
1.0
Ant-Maze
Vector
29
1.0
500
Table I: The environments used in experiments, where goal reaching distance denotes the range of the goal; other words, when the agent get closer to desired goal than the distance, task regarded success.
13

---

Under review as a conference paper at ICLR 2023
It's worth noted that Ant robot in the common used Ant-Maze environment (e.g., the used in Pitis et al. (2020); Eysenbach et al. (2022)) is different the one in Ant- Ant-Locomotion (e.g.. g. the one used in Zhu et al. (2021)), like the gear and ctrirange attributes. Thus, in order to test the transferring ability, we synchronize the Ant robot in these two tasks and re-run all baseline methods on these tasks.
B.2 ACTION DYNAMICS SETTING FOR HIGH-DIM----H CHALLENGE
For transferring experiments on i High gh--ghh----- Act challenge, we take 80% of the original gravity with a designed complicated dynamies for the transferring experiment (different both action space and dynamics). Particularly, given the original action space dimension and dynamics s' = fa) on state s, the new action dimension and dynamics become n = 2m and s' = (h(a)), where is constructed as:
= - cxp(a n/2 + + cxp(a/n//2 11))
here ali: : j] selects the i-th to (j - -) elements from the action vector a. In other words, we trans- fer to different gravity setting while doubling the action space and construct more complicated action dynamics for agent to learn.
B.3 IMPLEMENTATION DETAILS
The implementation of PILoT and HER based on open-source Pytorch code framework!. As for compared baselines, we take their official implementation, use its default hyperparameters and try our best to tune important ones:
MEGA (Pitis et al., 2020): http::://ggthhb..c////tttt//////
HIGL (Kim et al., 2021):
Contrastive
RL
(Eysenbach et al., 2022);
For resolving image-based based tasks, learn encoder that is shared between the policy and the critic. In par- ticular,
State Embedding
the same encoder structure for HER, MEGA and HIGL. The encoder has four convolution layers with the same 3 kernel size and 32 output channel. The stride of the first layer is 2 and the stride of the other layers 1, shown Fig. 7. We adopt ReLU as the activation function in all the layers. Af- ter convolution, we have a fully connected layer with 50 hidden units and a layer-norm layer to get the output of the encoder. When training, only the gradients from Q network are used to update the encoder. For con- trastive RL, we take their default structures for image- based tasks.
FullyConnectee 50+ NoyerNoooo
3x3 Conv, 32, ReLU
S
Figure 7: The encoder network architecture.
B.4 IMPORTANT HYPERPARAMETERS
We list the key hyperparameters of the best performance of HER in Tab. 2 and UDPO in Tab. on each source task. For each task, we first tune HER to achieve the best performance, based on which we further slightly adjust UDPO's additional hyperparameters.
For HER, we tune Replay buffer size E (1c5, 1c6). Batch size E (128, 2048, 4096). Policy x learn- ing rule {3e - 4, le
3). 33.
For UDPO, we only tune two hyperparameters, State planner coefficient E {le - 3,5e 3, 2, 5e 2, le 1}. Inverse dynamics T learning interval A (200, 500, 1500, 2000). As further shown in Section C.2, the choice slightly affects the success rate but can impose considerable influ- ence the accuracy of the state planner. The larger À will lead stronger constraint on the accuracy
hhttps://gi thub. com/ iri conal /o/ 1.$wi
14

---

Under review as a conference paper at ICLR 2023
of the state planner, but can hurt the exploration. On the other hand, A controls the training stability, and larger assumes that the local inverse dynamics does not change for a longer time. Therefore, in principle, for those tasks that exploration is much more difficult, we tend to choose small A; for those tasks that the algorithm can learn fast so that the local inverse dynamics changes drastically, should have small A. In default, choose À le 2 and A 1500.
We also list the hyperparameters of HER (PILoT) in Tab. on each target task, which is the same as the baseline HER algorithm (except the additional Transferring bonus rate.) In default, we set all Transferring bonus rate to be 1.0, and find that it can reach a desired performance. In Section C.2, we also include the ablation of the choice of this hyperparameter.
Table 2: Hyperparameters of HER on the source tasks.
Environments
Fetch-Reach
Fetch-Push
Reacher 2D Maze
Point-Locomotion
Optimizer
Adam Optimizer
Discount factor 7
0.99
Replay butter size
le5
le6
le6
le5
le6
Batch size
128
2048
2048
128
2048
Number of VecEnvs
8
4
4
Q learning rate
3e-4
Policy learning rate
3e-4
le-3
3e-4
Table 3: Hyperparameters of UDPO on the source tasks.
Environments
Ratch-Reach
Fetch-Push
Reacher Adam Optimizer
2D Maze
Loct-oLocomotion
Optimizer
Discount factor 7
0.99
Replay buffer
1c5
Ic6
lc5
le6
Batch size
128
2048
2048
128
2048
Number of VecEnvs
8
4
State planner cocfficient A
1c-2
1e-1
5e-3
Q learning rale
3e-4
Policy x learning rate
3o-4
1o.3
30-4
Inverse dynamics 7 lcarning
lc44
Inverse dynamics learning interval A (epochs)
200
1500
Table 4: Hyperparameterrs of HER / HER (PILoT) on the target tasks.
Environments
Foldh-Reach-HHig--Dit Fetch-Pull--igg-DDii
Fetch- Reach Toicl-Reach-Imag
Reacher-Image
Ant-Locomotion
Oplimizer
Adamm Optmmzer
Discount factor y
Roplay butter size
le5
le6
Batch size
2048
4096
Number of VecEavs
8
Q leaming rate
3e
Folicy learning
36-4
le-3
3o-4
1633
1e.3
Transforring boous rate
1.0
C MORE EXPERIMENTAL RESULTS
C.1 DOES UDPO REACH WHERF IT PREDICTS?
In the universal decoupled policy structure, the state planner is decoupled and trained for predicting the future plans that the agent required to reach. Therefore, it is critical to understand the plans given by the planner and make sure it is accurate enough that the agent can reach where it plans to go, for distilling and transferring. To this end, we analyze the distance the reaching states and the predicted consecutive states and draw the mean square error (MSE) along the RL leaning procedure in Fig. 8. To our delight, as the training goes, the gap between the planned states and the achieved states is becoming smaller, indicating the accuracy of the state plan.
15

---

Under review as a conference paper at ICLR 2023
Additionally, also visualize the imagined rollout by state planner the source tasks, which is generated by consecutively take predicted states as new input. We compare it with the real rollout in Fig. 9, showing the state planner conduct reasonable multi-step plan.
On the target tasks, we visualize the subgoals proposed by the distilled goal planner and the real rollout that achieved during the interaction in Fig. 10, showing the effective and explainable guidance from the goal planner.
Feid- Roach
Fasch
Mees
Loconntion
MSE
Poad Nea MSE
km
3teg
300
sige
Figure 8: Curves of the MSE between one-step prediction of the state planner and the real state achieved in the source environments.
Skap
Stop
Stop
Skop45
Skip 30
PointLocomotion 50 steps
Planned Rollout
Mho
Stia
Real Rollout
Sepp
Reacher 25 steps
Planned Rollout
Sep0
Rollout
Figure 9; Imagined rollout by state planner and the real rollout that achieved during the interaction in two tasks, showing the reasonable multi-step plan of the state planner. For Fetch-Reach and Fetch-Push environments, we cannot render the planned rollout since simulator internal states can recovered from agent obscrvations.
C.2 ABLATION STUDY
In this section, we aim investigate the robustness and the key components of proposed PILoT framework. Specifically, we first analyse the two critical hyperparameters, i.e., the inverse dynamic train frequency A and the regularization coefficient on training UDPO the pre-tre pre-training stage; then, we provide additional ablation analysis on bonus ratio 3 in the transferring stage.
Pre-training ablation on the inverse dynamic train frequency A. We first conduct ablation studies on the inverse dynamic train frequency A. In particular, this hyperparameter determines how often we train the low-level inverse dynamics and how long we regard the inverse dynamics is train the high-level state planner. It is intuitive that larger A is assuming that the local inverse dynamics does not change for a longer time; on the contrary, a small A should be used when the the local inverse dynamics changes drastically. In our experiments, find that A affects
static when
16

---

Under review as a conference paper at ICLR 2023
Seep
Fetch-Reach-Img 50 steps
Planned Goals
Real Rollout
Seo
Ant-Locomation 25 steps
Planned Goals
Real Rollout
X
Figure 10: Subgoals proposed by distilled goal planner and the real rollout that was achieved during interaction in two target tasks, showing the effective and explainable guidance from the goal planner. For Ant- Locomotion environment, the goal planner is distilled from Point- boint-Locoooton environment, in which the point agent has faster moving speed than the ant. Nevertheless, the resultant trajectory in Antomoccoooionn can still match the planned goals roughly.
the training stability, as it embeds some prior about the policy training in certain tasks. As is shown Fig. 11, Fetch-Push requires smaller than Point-Locomotion, since Fetch- more simple, and the policy training is much faster than the policy training in the naint-Lccomtton.
Fetch-Push
Point-Locomotion
4-100
A-500
1.0
A-000
1.0
A-1000
Rate
Success Rate
1-150
A- 1000
A-2000
2-2000
A-2500
0.5
a-5000
0.5
A-5000
0.0
0.0
1.0
2.0
3.0
6.0e5
2.0
3.0
4.0
5.0
6.0e5
10.0
Pred Real MSE
7.5
Pred Real MSE
5.0
2.5
0.0
1.0
2.0
3.0
4.0
5.0
6.0e5
1.0
2.0
3.0 steps
4.0
5.0
6.0e5
steps
Figure 11: Ablation study on inverse dynamic train frequency A with seeds, given the other parameters the same in Section 6.2.
Pre-training ablation the regularization coefficient X. The regularization coefficient À is another critical hyperparameter in UDPO training. The choice of balances the policy gradient term and the constraint term in state planning updates. Particularly, larger À puts more weight on the
17

---

Under review as a conference paper at ICLR 2023
supervised penalty objective, which reduces the planning of infeasible next states. However this can hurt the exploration ability offered by policy gradient objective. The results Fig. 12 supports the intuition. In both environments, the models trained with À 0.1 performs best in reaching where they plan, but can not finish the goal-reaching task well. On the other hand, a extreme small results in quite large gap between reached states and planned states. Such gap can make the subsequent transferring impossible. Therefore, the recipe is to find the medium À which achieves competitive success rate while keeping the value of prediction real MSE from explosion.
Fetch-Push
Point-Locomation
A=0000
A=0.001
1.0
1-0.005
1.0
A-0.005
Success Rate
Success Rate
A-0.01
1-0.55
2-0.05
A-0.1
A-0.11
0.5
0.5
0.0
0.0
1.0
2.0
3.0
4.0
5.0
6.0c5
0
1.0
2.0
3.0
4.0
5.0
6.0c5
Pred Real MSE
40
Pred Real MSE
4
20
1.0
2.0
3.0
6.0c5
1.0
2.0
3.0 stops
4.0
6.0c5
stops
Figure 12: Ablation study on state planner coefficient with seeds, given the other parameters the same in
Section 6.2.
Transferring ablation on the bonus ratio 3. In the transferring stage, bonus ratio 3 balances similarity reward from distilled planner and the sparse reward from the environment. We observe from Fig. 13 that 1.0 is an appropriate choice for at least tasks tested in this paper. When 3 is smaller (e.g., 0.1, 0.2, 0.5), the success rate converges more slowly, and the final performance is also worse. This indicates that small 33 can not provide strong enough signals for the policy to follow planned landmarks, leading
decreased transfer efficiency. On the other hand, when much larger (e.g., 2.0, 5.0) is adopted, the performance becomes even worse than insufficient guidance from small Also, in Fetch-Reach-Imaee environment, the training curves are quite unstable. Although the planned landmarks are useful in overcoming the sparse reward issue, they can not completely replace the final sparse reward. As Fig. 8 shows, even in the source environments, there exist small but not negligible errors between planned states and achieved states. Using large 3 can let the policy misled by the goal planner and overfitted to those errors.
Ant-Locomotion
Fetch-Reach-Image
1.00
0-0.1
1.00
0-0.2
8-0.2
Success Rate
8.0.5
Success Rate
0.75
8.0.5
0.75
8-1.0
6-1.0
3.2.0
0.50
2-2.0
0.50
6-5,0
0.25
0.25
0.00
0.00
2.0
3.0
4.0
5.0e6
0.4
0.8
2.0e5
steps
steps
Figure 13: Ablation study bonus ratio with 5 seeds, given the other parameters the Section 6.2.
18

---

Under review as a conference paper at ICLR 2023
C.3 MORE ZERO-SHOP TRANSFER VISUALIZATION
In this section we illustrate more zero-shot transfer cases including success cases and failure cases. In fact, the failures should be attributed the inaceuracy of the controller (policy) since the goal planner always gives the right way to success. If we can train a more accurate local controller, the performance of success rate can no doubt be further improved.
C.3.1 SUCCESS CASES
Figure 14: Success zero-shot transfer Ant-Maze.
19

---

Under review as a conference paper at ICLR 2023
C.3.2 FAILURE CASES
Figure 15; Failure cases zero-shot transfer on Ant-Maze.
20


Reviews:
[{"review_id": "V8ICZNgc7I5", "reviewer": "Reviewer_LhAd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to transfer policies across agents having different observations and action spaces in a given environment, but a shared space of subgoals and the same set of subgoals that lead to a given goal. To do so, it trains a policy in an origin environment, decoupling it into an inverse dynamic model, and a goal-conditioned state planner. This decoupling allows to transfer to new agents by learning a new inverse dynamic model and distilling the goal-conditioned state planner from the original agent. The approach is tested in different benchmarks, changing the observation and action space between domains, dynamics, and layout. The proposed approach can learn in the new domains much faster than competing baselines, and even execute goal-conditioned policies in a zero-hot manner given changes in environment layouts.", "summary_of_the_review": "My main concern is with the clarity of the paper. I believe I understand what the method is doing, but the assumptions  it relies on don't seem to match with the described experiments. There are a lot of missing details in the method section. I also struggle to understand the difference between Fig2 a and Fig2 c. Even if the ideas in this paper are interesting, and results promising, much of the paper is to unclear to be accepted at ICLR. I am giving the paper a weak reject, with the hope that the questions above will be clarified, and that the paper can be updated to clarify experiments and framing of the method.", "strength_and_weaknesses": "\n**Stengths:**\n- Relevance:\n\t- The approach addresses a relevant problem which occurs in practice in multiple scenarios.\n\t- The policy distillation approach can be trained without experience in the target environment, since it is distilled from the expert policy, it assumes though that all the states correspond to subgoals (as per equation 10), or that the subgoals are known a priori. In the second case, this seems like a big assumption. In the first case, isn't this the same setting as the one in Figure 2a? \n- Thorough ablations and analysis of experiments.\n- The method manages to learn policies in the source domain competitively with other goal-conditioned methods, and shows better transfer properties in new domains.\n- The method manages to transfer to new domains, new morphologies and observation spaces with minimal new data from the new domain, and it even can perform goal conditioned tasks in cases where the target and source domain have the same morphologies but different layouts.\n\n**Weaknesses:**\n\n- Method:\n\t- The method requires us to learn an inverse dynamics model for every new policy. Isn't it a big assumption that we can learn that for a wide enough coverage of states? Couldn't we just train a policy with the data used to learn the inverse dynamics model?\n- Wrong claims and gaps in literature review:\n\t- It's not true that most of the work in RL is focused on a single task, single agent. There is extensive literature in methods [1,2] and benchmarks [2] to evaluate multi-task RL. There are also works [4] addressing agents that operate across different state and action spaces. Similarly, when introducing Decoupled Policy Optimization, see any method with successor representations for instance.\n- Clarity:\n\t- The paper needs significant rewriting, the motivation in the introduction is confusing and difficult to follow. I would suggest starting with the SLAMcase study and why we care about similar goals even when the state and transitions are different.\n\t- I found the paper very difficult to read, specially the abstract and intro, which don't really talk about the main point of the paper, shown in Figure 2. The paper required me to keep checking the references (Sun et al, Liu et al.). It would help if the paper was a bit more self-contained and explained the important aspects of these references in a clearer way.  \n\t- It would help in decoupled policy optimization if authors explained the terms in Eq. 2.\n\t- Similarly, it would help if the examples in section 3 for different assumptions were followed by some example of agent transfer that satisfies the different assumptions.\n\t- Figure 3 is not very informative, I would recommend specifying there what are the losses, what the colors represent and explaining the process more detailedly in the caption.\n\t- Experiments are also unclearly described.\n\t- It would be good if authors pointed more explicitly how their training method is akin to HER, which they only mention in the method section as a method that can be used, to later find in the experiment section that this is how it is trained.\n\t- I would suggest changing the name HER (PILOT and UDPO Pilot) and just call it under the same name, or the same name with an underscore original/transferred. \n\t- I would like some text describing the reason why the baselines are selected, what are the ideas that want to be tested when comparing with HER vs HIGL vs MEGA vs ContrastiveRL? Some of it inferred from the related work, but it would be good to reiterate on that when explaining the baselines.\n\t- In the zero-shot transfer experiment, how is the goal transition function learned? How can we learn from the Ant-locomotion, and without any retraining, that one cannot follow a straight line in ant-maze to complete a goal.\n\n\n**Clarification Questions:**\n- What does it mean in practice the equation in the end of page 2? What kind of environments satisfy this condition? There is also an abuse of notation, using T to indicate the state at the transition function and the likelihood of a state transition.\n\n- How does the objective in Eq. 7 help? We are maximizing the planner to match the actions from the policy, but the policy is being learned based on this planner. When the policy is random at the beginning, how does this give information to h?\n\n\n\n- The diagram c in figure 2 seems to imply that the requirement is that the transitions between subgoals is the same. What happens if there are two agents with similar subgoals but one can achieve it faster than the other? For instance, an agent that moves half the distance of the other agent. Would those two agents be transferable under this setting? I am not sure Figure 2. c represents that case. In fact, the transfer experiments work with different dynamics, which means that while both agents may have a set of shared subgoals, some of the intermediate states (or whatever projection for them) will be different, so I am not sure how the method would work here. For instance, if the dynamics change and suddenly an action moves the arm at a faster pace, some of the goal states in the original environment won't appear in the transfer environment.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OsYmsrHM1wZ", "reviewer": "Reviewer_1XEZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the subject of few-shot and zero-shot skill transfer between tasks and agent morphologies, two important multi-task reinforcement learning settings. The paper proposes a three-stage approach that first trains a goal-conditioned state-based planner in a source domain that is trained to sample next states that, when passed through an inverse dynamics model, lead to optimal actions. In this first stage, the composition of the one-step planning model and the inverse dynamics model is a policy for the source domain. This first stage is based on prior work by Liu et al., 2022b, but is extended by the authors to a goal-conditioned setting, whereas it was originally used by Liu et al., 2022b without goals.\n\nThis modularization of the policy is important to the authors’ method, as it can be distilled into a shared latent (goal) space that lends itself to transfer between different agents and tasks. This contribution, ie distilling a one-step planning model into a shared latent (goal) space is novel, though it is not clear why this is advantageous over directly learning this one-step model in the shared latent (goal) space in the first stage, instead of performing distillation post-hoc.\n\nIn the final stage of the method, a policy is learned in a target domain using goals specified in the shared latent (goal) space from stage two, and a reward bonus derived from the goals sampled from from the distilled one-step planning model (to mitigate reward sparsity). The advantage of the proposed method is that the combination of informative goals and a dense reward can improve sample efficiency. Overall, the proposed method is complex, illustrated by the three distinct stages of training required to learn a policy in the target domain. Evaluation is performed on seven distinct simulated environments; however, the results are somewhat mixed. \n", "summary_of_the_review": "Overall, this paper proposes a surprisingly complex method in light of the gains observed. The weaknesses of the paper only slightly out-weigh its strengths, and once these weaknesses are resolved, I am willing to reconsider my evaluation and increase my review score. However, in light of these weaknesses, I am currently unable to recommend the paper for acceptance, and encourage the authors to discuss my questions and contribute new results in their rebuttal.", "strength_and_weaknesses": "Strengths Of The Paper:\n1) The paper generalizes the Decoupled Policy Gradient, which has been shown to be effective in the standard RL setting, to a goal-conditioned multi-task setting.\n2) The proposed method is appealing because it facilitates building an informative dense reward in domains where it may otherwise be hard to construct one, such as for visual reinforcement learning tasks, where estimating the pose of the agent (for pose-based goals) is challenging.\n\n3) Experiments are generally of high-quality, including proper documentation of hyperparameters, a description for how tuning was performed, and an appropriate set of ablations (which are in the Appendix, not the main body of the paper).\n4) Seven tasks are used for evaluation, which is an appropriate number.\n\nWeaknesses Of The Paper:\n\n1) The first main weakness of the paper is the empirical results. Particularly, in Figure 5, Page 8, the complete method (UDPO) is compared against HER and HIGL, two baseline goal-conditioned reinforcement learning approaches, and the results do not support the authors’ claim that “Compared with HER that is trained using a normal policy, we observe that UDPO\nachieves similar performance, and sometimes it behaves better efficiency.” On three of five source tasks, HER appears to outperform UDPO. On one of five, performance is nearly the same, and on the last (the 2D-Maze pointmass environment), UDPO outperforms HER. This may suggest UDPO incurs a penalty on the source-task it is initially trained. Transfer results are more compelling, where HER with the PILoT reward bonus is the best on four of five tasks.\n\n2) For zero-shot experiments, where a task-specific planner is used with a pretrained lower-level on the AntMaze task (see Figure 6), more appropriate baselines for this experiment would be Planning with Goal-Conditioned Policies (Nasiriany, et al. 2019) and Hierarchically Decoupled Imitation for Morphological Transfer (Hejna III, et al. 2020). These suggestions are RL algorithms designed for zero-shot transfer of lower-level skills to new tasks and morphologies, and they operate with similar assumptions as this paper (that one has a pre-trained goal-conditioned lower-level policy in the target domain to be adapted to the new task).\n\n3) Figure 5 is missing a comparison against other intrinsic reward methods. Currently, this figure shows that the combination of the informative goals sampled from the shared planner, and the bonus reward computed from these goals improves efficiency. However, it remains unclear how the proposed intrinsic reward compares to existing techniques for densifying sparse rewards. This is a particularly important evaluation to include because Figure 13, Page 18 suggests that the majority of the performance gains are due to the reward bonus (see $\\beta = 0.1$ results).\n\nSuggestions For The Paper:\n\n* Agent-Agnostic Reinforcement Learning with Transformers is discussed in the related works, and Gupta et al. (2021) is not the only, nor the most recent paper that explores this. Consider referencing My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control, (Kurin et al. 2020) and AnyMorph: Learning Transferable Policies By Inferring Agent Morphology, (Trabucco et al. 2022), two other recent papers in this domain.\n\n* There are a few typos, such as “Euclid distance” on Page 5, which is likely intended to read “Euclidean distance” instead. Fixing these will improve the readability of the manuscript.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MY8NgUDEHI", "reviewer": "Reviewer_7J1u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for transferring skills for goal-conditioned tasks with different action / observation spaces. The method seems to build on Decoupled Policy Optimization (DePO) to learn a planner for the next state, and an inverse dynamics model to produce the action, but additionally conditions on the desired goal. To transfer to a new task, a new goal planner is learned in the target domain via distillation (using the planner in the source domain as the teacher). Experimental domains include openai gym environments including fetch robot, ant-locomotion, reacher. ", "summary_of_the_review": "Overall I am not in favor of acceptance because the key idea of distillation for observation/action space transfer is well known in the literature, the decoupled policy architecture is from a previous paper[1], and the authors do not show that the decoupled architecture helps distillation, and so the contribution of the paper is unclear. ", "strength_and_weaknesses": "Strengths \n\n1. The problem considered by the paper is important - enabling transfer of skills for agents between different tasks. This is because collecting samples is expensive for real systems, and efficiently reusing data can help the agent learn new useful tasks. \n\nWeaknesses \n\n1. Missing comparison to regular distillation \n\nThe main component of this approach that is used for transfer is the distillation between the state planners. However, distillation between policies for transfer is quite standard in the literature, especially for the case when the action space is the same, but only the observation space is different. This includes both the low-level state to high-dim images [1] and morphologically different robot [2] settings considered in this paper. It is possible that using the decoupled policy and distilling in just the state space will yield more robust transfer, but currently there are no experiments that examine /show this. For the other transfer setting considered in the paper (action space), the DePO [3] paper includes very similar results, establishing that the decoupled policy structure helps since only the inverse dynamics model needs to be trained, and the state planner can be reused. \n\n2. Novelty/Significance \n\nThe paper largely builds on DePO, using the same decoupled policy structure and training objective formulation. The only changes are that goal conditioning is added to the policy, and there is an extra step of distilling the state planners when trying to transfer to new tasks. However, the idea of distillation for transfer is also quite standard as described in the point above [1-2]. Furthermore the experiments don't show transfer across different semantic tasks - i.e where the task is actually different (eg reaching to pushing etc), but rather for the same task, but just for different observation modalities or with different action space (the action dynamics setting does seem a little contrived as its not natural for the robot to suddenly have to deal with the kind of new actions considered in the paper). As discussed in the point above, it is known that student-teacher distillation helps for this kind of transfer (has also been used a lot for sim-real applications), and it's unclear if the decoupled policy structure adds any benefit to this application. \n\n\n[1] Jain, Divye, et al. \"Learning deep visuomotor policies for dexterous hand manipulation.\"\n[2] Hejna, Donald, Lerrel Pinto, and Pieter Abbeel. \"Hierarchically decoupled imitation for morphological transfer.\"\n[3] Liu, Minghuan, et al. \"Plan Your Target and Learn Your Skills: Transferable State-Only Imitation Learning via Decoupled Policy Optimization.\" \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "HJAeNRSzlbA", "reviewer": "Reviewer_7PL4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to achieve few or zero shot transfer pre-trained skills across tasks with different state and action spaces and environment dynamics, reducing the need for retraining. \n\nThe method involves 3 main steps. First, a goal-conditioned extension of DePO is used to learn an inverse dynamics model $I(a | s, s^\\prime)$  and state transition planner $h_{\\pi}(s^\\prime | s, g^t)$. Next, the state transition planner, specific to the source task, is distilled into a general goal transition planner $f_\\omega (g^\\prime | g, g^t)$ by learning an embedding function $g = \\phi(s)$ and using $h_\\pi$ as a teacher. Finally, the goal planner is used to provide a dense reward $r(s, a, s^\\prime, g^\\prime) = cos_similarity(\\phi(s^\\prime), g^\\prime)$ in the target task. Unlike prior methods, PILoT does not require the source and target tasks to have the same state or action space. The primary contribution of this paper is to switch from learning latent state representations to learning goal representations, which relieves the need for a shared action space and shared transition dynamics.\n\nThe dense rewards provided by the goal planner successfully help the agent learn the target task faster than several baseline methods, and to achieve zero-shot transfer learning from a point policy to an ant with a pretrained locomotion policy. ", "summary_of_the_review": "While the authors present some interesting ideas, the environments and baselines are lacking. Without additional experiments, it is difficult to determine whether this method represents an improvement over prior works. The writing could use some further proofreading and revision. Therefore, I am hesitant recommend this paper for acceptance as is, but would be happy to reconsider after some revision. ", "strength_and_weaknesses": "Strengths:\n1) The paper is well structured and clear about its goals, contributions and placement compared to related work. \n\n2) The method makes intuitive sense and could be extended to many interesting tasks. In the tasks presented it represents a clear improvement over prior methods. \n\nWeaknesses:\n\n1) There are many typos and confusingly worded sections that make the paper difficult to understand. The authors should carefully proofread and correct the incomplete sentences and other grammatical issues. \n\n2) The methods in this paper are closely related to DePO (Liu et al., 2022b) and the method proposed (Sun et al., 2022). It would be helpful to compare against these two methods when it is applicable, such as when transferring from vector to image based state representation, and in the zero-shot experiment. The environments used in the paper are also quite simple. I would like to see results on some of the environments used in prior works, such as Sawyer Bin and PointSpiral from Contrastive RL (Eysenbach et al., 2022) or PointMaze and FetchPickAndPlace from Maximum Entropy Gain Exploration (Pitis et al., 2020), which were both used as baseline algorithms in this paper. \n\n3) Although this is a transfer learning method, it seems that every each source task is only used to transfer knowledge to one target task. It would be nice to see one distilled goal planner be used to speed learning on several tasks. As it is now, it could be more fair to compare FLOPs used to train source + target policies to FLOPs used training directly on the target task. \n\nQuestions:\n\n1) Is the sampling strategy $\\pi_B$ in Section 4.1 always the goal conditioned policy being learned by DePO? \n\n2) How is the embedding function $\\phi$ learned for the target task? \n\n3) Could the goal planner be more general if it is distilled from several state transition planners?\n\nMinor:\n\n1) Where is the small blue point in Figure 1?\n\n2) Referring to the starting position (green points) as goals is confusing\n\n3) Equation 2 uses notation $h_{\\pi}(s^\\prime | s)$ and $I(\\cdot | s, s^\\prime)$ from the DePO paper, but they are not defined until later\n\n4) Figure 5 has five source and five target tasks", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject