Learning to Execute: Efficient Learning of Universal Plan-Conditioned Policies in Robotics
Authors: Ingmar Schubert
Danny Driess
Ozgur S. Oguz
Marc Toussaint
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Planning
Robotics

Abstract: Applications of Reinforcement Learning (RL) in robotics are often limited by high data demand. On the other hand, approximate models are readily available in many robotics scenarios, making model-based approaches like planning a data-efficient alternative. Still, the performance of these methods suffers if the model is imprecise or wrong. In this sense, the respective strengths and weaknesses of RL and model-based planners are complementary. In the present work, we investigate how both approaches can be integrated into one framework that combines their strengths. We introduce Learning to Execute (L2E), which leverages information contained in approximate plans to learn universal policies that are conditioned on plans. In our robotic manipulation experiments, L2E exhibits increased performance when compared to pure RL, pure planning, or baseline methods combining learning and planning.

Learning to Execute: Efficiently Learning Universal Plan-Conditioned Policies in Robotics
Ingmar Schubert', Danny Driess', Ozgur S. Oguz', and Mare Toussaint'
Learning and Intelligent Systems Group, TU Berlin, Germany
Machine Learning and Robotics Lab, University of Stuttgart, Germany
(ingmar sscuubert danny driess@campus. tauussain@@tuu-erllllaa ozgur. oguzeipvs. uni-stutrgart.de
Abstract
Applications of Reinforcement Learning (RI.) in robotics are often limited by high data demand. On the other hand, approximate models readily available in many robotics scenarios, making model-based approaches like planning a data-efficient alternative. Still, the performance of these methods suffers if the model is imprecise or wrong. In this sense, the respective strengths and weaknesses of RL and model- based planners are complementary. In the present work, investigate how both approaches can be integrated into one framework that combines their strengths. We introduce Learning to Execute (L2E), which leverages information contained in approximate plans to learn universal policies that are conditioned on plans. In our robotic manipulation experiments, L2E exhibits increased performance when compared to pure RI., pure planning, baseline methods combining learning and planning.
1 Introduction
central goal of robotics research is to design intelligent machines that can solve arbitrary and formerly unseen tasks while interacting with the physical world. Reinforcement Learning (RL) (Sutton & Barto 2018) is a generic framework to automatically learn such intelligent behavior with little human engineering. Still, teaching an RL agent to actually exhibit general-purpose problem-solving behavior is, while possible in principle, prohibitive in practice. This is due to practical restrictions including limited computational resources and limited data availability. The latter limitation is particularly dramatic in robotics, where interaction with the physical world is costly.
On the other hand, for many robotics scenarios, there is a rough model of the environment avail- able. This can be exploited, e.g., using model-based planning approaches (Mordatch et al. 2012 Kuindersma et al., 20166] Toussaint et al. 2018). Model-based planners potentially offer a more data-efficient way reason about an agent's interaction with the world. Model-based planners have been used in many areas of robotics, such as for indoor and aerial robots (Faust et al. 2018), visual manipulation (Jeong et al. 2020). or humanoid walking (Mordatch al. 2015). Still, if the model does not account for stochasticity contains systematic errors, directly following the resulting plan will not be successful.
The present work starts from the observation that both pure RI approaches and pure planning approaches have strengths and limitations that fairly complementary. RL makes assumptions about the environment but is data-hungry, and model-based planning generally implies model simplifications but is data-efficient. For robotic manipulation tasks, seems natural try and integrate both approaches into one framework that combines the strengths of both. In the present work we seek to add an additional perspective the open question of how this can be achieved best.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We introduce novel approach that we call Learning to Execute (L2E). Our approach translates sparse- reward goal- conditiind Markov Decision Processes (MDPs) (Bellman 1957) into plan-conditiones MDPs. 1.2F exploits a simple planning module to create crude plans, which are then used to teach any off-the-shelf off off-policy RI. agent to execute them. I.2F. makes use of final-volume- preserving reward shaping (FV-RS) (Schubert et al. (22TT, allowing it to train universal plan-conditioned policy with high data efficiency. The contributions this work are:
We introduce L2E, which uses RL efficiently learn to execute approximate plans from model-based planner in plan-conditioned MDP. We describe formally how FV-RS can be used as tool construct such plan- mlan-conditioned MDPs from coal-conditioned MDPs.
We introduce plan replay strategies efficiently learn universal plan-conditioned policies.
We demonstrate, using robotic pushing problems, that L2E exhibits increased performance when compared to pure RL methods, pure planning methods, or other methods combining learning and planning.
We discuss work related to ours in section 2 introduce our method in section[4] We present our experimental results section discuss limitations in section 6, and conclude with section
explain background and notation in section 3 and
2 Related Work
2.1 Goal-Conditioned Policies
Goal-conditioned universal policies (aelbling 1993. Moore al. 1999: Foster & Dayan 2002
Schaul et al.] [0155] Veeriah et al., 2018;; Nasiriany et al. 2019) not only act based on the state the agent finds itself but also based on the goal it tries to achieve. Hindsight Experience Replay (HER) (Andrychowicz et al. 2017) is particularly efficient way to learn universal policies. Here, achieved outcomes of the agent's 's interaction with the environment are interpreted as desired goals in order to improve sample efficiency in sparse-reward settings.
L2E draws great inspiration from this work, but in contrast to HER, L2E learns universal plan- conditioned policy. This means that the L2E policy in general can execute multiple plans leading to the same goal. Although this presents more complex learning task, show in our experiments that by incorporating plan information using plan-based FV-RS, the sample efficiency of L2E is significantly improved over HER.
2.2 Plan- and Trajectory-Conditionn Policies
Plan-conditioned policies create behavior that depends on plans that are input to the decision making.
Lynch al. (2020) learn plans and how to execute them from data generated by a human "playing"
with teleoperated robot. The resulting policy is conditional on a latent space of encoded plans. Our work differs from this paradigm in that human interaction is not needed. Both [Lynch et al. (2020) and Co-Reyes et al. (2188) 8) directly imitate a planned trajectory by maximizing its likelihood. In contrast, the plans used in the present work are not directly imitated. Using FV-RS guarantees that the fully trained L2E agent will reach its goal after finite time even if the plan provided is wrong. Guo jet al. (2019) learn trajectory-conde conditioone policies to self-imitate diverse (optimal and suboptimal) trajectories from the agent's past experience.
We instead assume in this work that the plan is provided by an external model-based planner. This allows the L2E agent to use external information during training that could not be concluded from its own experience yet.
2.3 Learning from Demonstration
L2E learns how to execute plans in order to achieve different tasks. In this sense, it is related to Learning from Demonstration (LfD) techniques that exploit demonstrations when learning a task.
Existing work (Argall et al. 2009 Hussein et al. 2017 Ravichandar et al. (020) differs significantly both in how the demonstration examples are collected and how the policy then derived. [Tayoor
et al. (2011) derive approximate policy from human demonstration, and then use this to bias the

---

exploration in a final RL stage. Hester et al. (0117) train a policy on both expert data and collected data, combining supervised and temporal difference losses. [Salimans Chen (2018) use a single demonstration as starting points to which the RI. agent is reset at the beginning of each episode. Peng al. (20T8) use motion capture data to guide exploration by rewarding the RI. agent to imitate In Cabi et al. (2019), demonstrations are combined with reward sketching done by a human. Interactive human feedback during training is another source of information used in Thomaz et al. (2006); Knox & Stone (2010). Kinose & Taniguchi (2020) integrate RL and demonstrations using generative adversarial imitation learning by interpreting the discriminator loss as an additional optimality signal in multi-objective RL.
While these LfD approaches are related L2E in that external information is used to increase RL efficiency, it is in contrast assumed in L2E that this external information is provided by a planner.
2.4 Combining Learning with Planning
Similarly to demonstrations, external plans can be exploited to facilitate learning. Faust et al.) (2018) connect short-range goal-conditioned navigation policies into complex navigation tasks using probabilistic roadmaps. In contrast, L.2E learns single plan-conditioned policy for both short-term and long-term decision making. [Sekar et al. (2020) use planning in learned model to optimize for expected future novelty. contrast, 1.2E encourages the agent to stay close to the planned behavior. Zhang et al. (2016) use model-predictive control to generate control policies that are then used to regularize the RL agent. In L2E, no such intermediate control policy is created, and a reward signal is computed directly from the plan. In Guided Policy Search (Levine & Koltun| 2013). differential dynamic programming is used to create informative guiding distributions from a transition model for policy search. These distributions are used to directly regularize the policy in supervised fashion, while L2E makes use of FV-RS as a mechanism to interface planning and RL. CCristiaaoo et al.) (2016) learn an inverse dynamics model to transfer knowledge from policy in the source domain in the target domain. The idea of integrating model-based and model-free RL has also been studied independently of planning (Pong al.. 2018 Janner et al. 2019). In contrast, in L.2E the model is translated by a planner into long-horizon plans.
policy
In the experiments section, compare L2E against two representative examples from the literature mentioned above. The first is using a plan to identify subgoals that are then pursued by an RL agent, as done in Faust et (2018). The second is executing the plan using an inverse model, similar to the approach in Christiano et al.
(016)). These two baselines and L2E can be seen as representatives of a continuum: Christiano et al.
2016 follow the plan very closely, trying to imitate the planner at each time step. Faust et al. (2018) relax this requirement and only train the agent to reach intermediate goals. Finally, in 1.2F, the agent is free to deviate arbitrarily from the plan (although it is biased to stay close), as long as it reaches the goal. We find that 1.2E results in significantly higher success rates when compared against both baselines.
3 Background
3.1 Goal-Conditinned MDPs and RL
We consider settings that can described as discrete-time MDPs M = (8, T,7, R. Ps). S and A denote the set of all possible states and actions, respectively. T S A S 15-Ri RØ the transition probability (density); T(s'Is, a) is the probability of the next state being s' if the current state is and a is chosen as the action. The agent receives a real-valued reward R(s, a, after each transition. Immediate and future rewards are traded off by the discount factor 7 E [0, 1). Ps: S Rt the initial state distribution.
The goal of RL is to learn optimal policy x²; SS A -> B that maximizes the expected discounted return. In other words, RL. algorithms generally try to find
argmax
EE aone [R(s at s141)
from collected transition and reward data D = {(8;, ai Pi, 2o More specifically for this work, we are interested in applications in robotics, where both E and A are typically continuous. There
3

---

exists wide range of algorithms for this case. For the experiments in this paper, soft actor-critt (SAC) (Haamoja et al. 2018) is used.
In a goal-conditioned MDP Mc = (s, A.A,,,, RG, Ps. Pa). the reward function Ra(s, a, 8 ,') has an additional input parameter, the goal E G. Here, Pa : + R is the distribution of goals. The optimal goal-conditioned policy Ttz acts optimally with respect any of these goals.
3.2 Final- Volume-Preeserving Reward Shaping
We use approximate plans as additional source of information for the RL agent. For sparse- reward goal-driven MDPs, FV-RS (Schubert et al. 2021) offers an efficient way to include additional information by adding an additional term
R(s, a, s') Rrv(s, a, 8') - R(s, a, 5') + Frv(s, a')
(2)
to the reward function, accelerating exploration. In general, the optimal policy '' corresponding to the original MDP and the optimal policy oby corresponding to the shaped MDP will be different. FV-RS however restricts the allowed modifications Fiy (s, a, s') in such way that after finite time, the optimally controlled agent ends up in a subset of the volume in which it would have ended up without shaping. As result, external information can be made available for the RL algorithm without changing the long-term behavior of the resulting optimal policy.
Specifically in the present work, we consider goal-conditioned MDPs in which the goal-conditioned reward RG of the underlying MDP either 1, if the goal is reached, further assume that the L2E agent is given an external plan represented as an intended trajectory
everywhere else. We
p (P1, P2:...) in state space. We intend reward the agent for staying close to the plan, and for advancing towards the goal along the plan. A natural way of achieving this is to use plan-based shaping reward (Schubert et al. 2021). The single-plat plan shaping function introduced there can be generalized to the multi-plan setting in the present work in the following way:
Fiv(s, a, s', p) - Ra(s, 5 f()) kss)
d d3 Pk(s))
exp
(3)
2
L
20
Here, fp) denotes the goal that leads E oo), k(s) argmin, (d(p:, s)), and d(.,-) is measure of distance in state space. For the pushing experiments discussed in this work, d(.,-) is the euclidean distance in state space ignoring the coordinates corresponding
the orientation of the box. The first term in eq. (3) ensures that the assigned shaping reward FFy is always smaller than the maximum environment reward (at most 1/2), and that if the binary environment reward is I, no shaping reward is assigned. The second term rewards the agent for advancing towards the goal along the plan, and the third term rewards the agent for staying close to the plan. For sufficiently high discount factor Fry is final-volume preserving, meaning that the long-term behavior of the optimal agent is unchanged.
4 Learning to Execute
L2E considers goal-conditioned MDPs MG (see section 3.1 for which an approximate planner Ω is available. L2E uses FV-RS to construct corresponding plan-conditioned MDP Mp from goal-conditioned MDP Mg and planner Q. In the following sections to we introduce our notion of plan-conditioned MDP Mp and describe the components of the L2E algorithm. We then summarize the L2E algorithm in section 4.4
4.1 Plan-Conditioned MDPs
Plans are provided by a model-based planner, which can be described as a distribution Ω Fx$xG -> RJJ over set of plans P. Given initial state and goal, 2(pls, g) is the probability that the planner outputs p as a possible plan of how to achieve from state s. The distinction between goals and plans that plans conditional on both a goal and an initial state. Therefore, both initial state and goal can be inferred using the plan only.
In plan-conditioned MDP Mp (S, Rp, Ps. Pp), a plan p E P is given to the reward function Rp(s, a, 8', s'))) as additional input parameter. Pp P B is the distribution of plans. The optimal plan-conditioned policy TÝ behaves optimally with respect to any of these plans, creating distribution =p(( s, p) over actions that is conditional the current state and the current plan.

---

4.2 Constructing the Plan-Conditioned MDP
We use FV-RS to shape the reward function RG of the original goal-conditioned MDP Mg (8, G. A, T,y, RG. Ps. Pa) with plan-dependem term Fiy a, s' p) (see equation 3
as, a, ,',g) ( (s, a, ,' Ro(s, a, ,' + FFv (s,,,,, a, a, s'p))
(4)
We call f() the goal for which the plan was created. If a planner St should be such that can not be recovered from the resulting plan
D[-1s.0) g). we can always construct new ğ ~ f such that أ = [p. g]. Since now 9 can be recovered from p deterministically, we can assume that f always exists without loss of generality, We can interpret the shaped reward function
(5) as a plan-conditioned reward function a plan-conditioned MDP Mp = ( A,, Rp, Pp). The distribution over initial states and plans Pp of Mp is still missing, and can be constructed as
Rp(8, a, pp) = R (8, a, [()),P)
Pr(s.p) =
(6)
In practice, Pp can be sampled from by first sampling Ps. g ~ Pa and then subsequently sampling p ~ g))
Thus, we have constructed plan- plan-conditioned MDP Mp by combining goal-conditioned MDP Mc with an approximate planner ! and a FV-RS shaping function Fiv. For reference later in this paper, write as a shorthand notation Mp C(Ma, Ω2 Frv). Furthermore, will refer Mp as the corresponding plan-conditioned MDP to Mc and vice versa.
In contrast to potential-based reward shaping (Ng et al. 1999) FV-RS does not leave the optimal policy invariant. As a result, generally 3p E F : [()) F xp(-1,p)) words, the optimal policy of Mp and the optimal policy of MG will not result in identical behavior. In fact, while (. g) learns one policy for each goal g. - p) can learn different behavior for each plan in
the set of plans {p IP f(p) 9) leading towards the same goal g.
4.3 Plan Replay Strategy
In order to efficiently learn a universal plan- condiiiitt L2E policy, the reward for experienced episodes is evaluated with respect to many different plans. In HER (Andrychowicz et al.) (017). is assumed that each state 8 E can assigned an achieved goal. Recorded episodes are then replayed with respect to goals that were achieved during the episode, i.e. the recorded transitions are re-evaluated with respect to these goals. This ensures that recorded transitions were successful in reaching the replayed goals, resulting in highly informative data.
In L2E, transitions are replayed with respect to plans. However, there is no meaningful relation between each state s E S and unique "achieved plan". Therefore, the L2E agent replays transitions with past plans that were recorded some point during training and were stored in replay buffer D. The replay plans are chosen according to plan replay strategy S..
A plan replay strategy Sn provides distribution over replay plans, conditioned on the replay buffer D and the buffer containing the current episode Dep (see algorithm | for a definition of D and Dep). For replay, n plans are sampled according this strategy {pi, Pn] ~ S. Dep: D). We consider two types of replay strategies. Uniform replay Sumi samples n unique plans uniformly from the replay buffer D. Reward- -iased replay Sbia first uniformly samples m unique plans from the replay buffer D, and then returns the plans P; that would have resulted in the highest sum of rewards (ee,,ar. (( D.p (p(Sk, ak $k ps) for the episode stored in Dep The idea behind using weward-biased replay is to bias the replay towards transitions resulting in higher reward.
4.4 L2E Algorithm
The L2E algorithm outlined in algorithm] 1 First, the corresponding plan-conditioned MDP Mr = C(Ma. Ω Frv) is constructed from the original goal-conditioned MDP Mg. the planner Ω and the shaping function Fry as described section 4.2 The agent acts the environment trying to follow one randomly sampled plan per episode. The episode is then added to the replay buffer,
5

---

Algorithm 1: Learning to Execute (L2E)
Input :Goal-conditioned MDP Mg. approximate planner 52, FV-RS shaping function Frv, plan replay strategy Su, off-policy RI. Algorithm A
Output:Uniiee Universal plan-conditioned optimal policy wpp for the corresponding plan-conditioned
MDP M CM,, S2, FFv)
Construct plan-conditioned MDP Mp C(Mo, Q, FFv) as detailed in section 4.2
2 Initialize replay buffer D + (
while T not converged do Initialize episode buffer Dep t f Sample initial state and goal (sp: 9) Po:
Sample plan p ~ (2 9):
8 f- 80:
while Epiisode not done do Sample action a ~ Tp((- s,p); Sample transition s' ~ s,);; Collect shaped reward r +- Rp(s, sp: Add to episode buffer Dep Dep U (s,,,,r, s + s'; end
10
11
12
13
14
15
Add episode to replay buffer D +- D Dep: Get replay plans {Pi..... .P ~ Sp( Pep: D);
16
17
for Preplay in Pl.... Pr. do
18
for (8, a, T, 8 p) in Dap do Calculate replay reward Treplay Rp a, S' Preplay);
19
20
Add replayed transition to buffer D + DU {(s, a, Freplay: s' Proplay) ) end end
21
22
23
Update policy using off-policy RL algorithm Tp + A(Tp, D)
24
end
along with data from episode replays with respect other plans. These other plans sampled from the replay buffer according to the replay strategy Sn, A generic off-policy RL algorithm is used to update the agent using the replay buffer. This process is repeated until convergence.
We would like to emphasize that the L2E algorithm agnostic to the exact type of off-policy RL algorithm. By combining state and plan into "super state" for the purpose of passing the replay buffer to the off-policy RL algorithm, L2E can be interfaced with any off-the-shelf implementation.
5 Experiments
We evaluate the L2E agent against several baselines using two simulated robotic manipulation tasks, namely a pushing task and an obstacle avoidance task. These two environments are chosen to compare different approaches open-source version of the opanAl gym FetchPush-vl task (Brockman et al. 2016)), the obstacle task was chosen represent robotic manipulation tasks with segmented state spaces. This allows discuss limitations exploration in such environments as well.
variety of challenges. While the pushing task can be seen as
A video of the experiments is available in the supplementary material. The complete code to fully reproduce the figures in this paper from scratch can be found at guthub.com/ischubet///// and the supplementary material. This includes the implementation of the environments, the implementation of the L2E agents and the baselines, and the specific code used for the experiments in this paper.
The experiments section is structured as follows. In section 5.1 we discuss the environments and planners that are used in the experiments. We briefly introduce the plan embedding used for A.5 In section 5.3
L2E agent in section 5.2 additional experiments on this can be found in section
we introduce the baselines against which we compare our method. In section 5. we discuss our experimental results. Implementation details of the L2E agent are given in section A.I

---

1.0
Succoss Configuri ContiliCCoiiiaaatttn
0.6
0.4
LZE $100
Pian
LZE S00
HER Fu
Planned Subgools + RL.
0.0
Number of experienced transitions
1e6
(h)
L2E St00
Plan
L2E S
HER fu
Planned Subgoals + RL
Rlat Inttal Configuratioo
0.6
0.2
Number of experienced transitions
(c)
(d)
Figure 1: Results for the basic pushing environment (figures (a) and (b)) and the obstacle pushing environment (figures(c) and (d)). Lines represent averages over N independently trained agents times M test rollouts per agent and time step. the colored areas indicate the confidence intervals (standard deviations of the mean). Renderings created with open-source code by Orthey (202I). (a) The simulated pushing setup (inset). The spherical end effector tries to push the to the green target. Once the box is pushed off the table, it can not be recovered. An approximate planner creates simple manhattan-lic plans (blue line and transparent boxes in large figure; the end effector is planned to change height to push the box from the other side). The path of the box is planned fully, but only shown at some time steps. Recent end effector movement is indicated grey. Learned behavior can differ significantly from the original plan. (b) Comparison to the baselines introduced in section 5.3 L2E shows significantly higher sample efficiency and success rates. See section A.7 for longer of L2E Shias 1000- (c) In the simulated obstacle setup, additional obstacle added. In this environment, the planner can create (and the RL agent learns to execute) multiple plans from the same initial configuration to the same goal (inset shows alternative plan). (d) Comparison to the baselines introduced in section 5.3 1.2E performs significantly better than the pure learning HER baselines, the pure planning baseline ("Plan"), and the "Planned Subgoals RI." baseline. While using an inverse model initially more efficient, L2E achieves significantly better results given enough data.
5.1 Environments and Planners
Figure and Figure Ic show renderings of the basic pushing environment and obstacle pushing environment, respectively. We use the open-source Nvidia PhysX engine (phy 2021) to simulate box size 0.4 0.4 being pushed on table of size 3 x by a spherical end effector of radius 0.06. The 10D state space of both the goal-conditioned MDP Mg and the corresponding plan-conditioned MDP Mp consists the 3D position of the end effector, the 3D position of the box, and the 4D quaternion for the orientation of the box. The agent controls the 3D velocity of the end effector. The
7

---

maximum velocity in any direction is 0.1 per time step. The end effector movement resulting from the agent's 's actions slightly distorted by random noise. In the obstacle pushing environment, the agent additionally has to evade an obstacle in the middle of the table.
In the goal-conditioned MDP Ma; each goal is represented as desired 2D box position on the table. The goal-dependent sparse reward function Rg is 1 if the box is within 0.1 of this desired goal, and 0 if not. The initial state-goal distribution Pa uniform across the table for the non-colliding box position and goal position. The end effector is always initialized at the origin and the box is always initialized with fixed orientation parallel the table.
For the basic pushing environment, we use crude manhattan like planner Ω that deterministically outputs plans consisting of two separate contacts leading the initial state to the goal as shown in Figure Ta For the obstacle pushing environment, plans consist of four contacts, corresponding to an additional intermediate box position which is chosen random (see Figure Tch. Thus, the agent learns to execute an infinite number of plans for each combination of start and goal.
Plans are represented as a trajectory of length 50 for the basic pushing environment and 100 for the obstacle pushing environment, consisting of 6D elements representing end effector position and box position. For the basic pushing environment, we additionally report results for less dense plans in section A.6 The orientation of the box is not specified in the plans. We construct the plan-conditioned MDP Mp described in section 4.2 using this planner and the FV-RS function in equation 3 We use the width parameter a = 0.5 throughout the experiments.
5.2 Plan Encoding
The plans p are embedded before they are provided to the policy. A plan encoding is an injective function o: + C from the set of plans to latent space C. If F is a manifold in some high- dimensional space, the dimensionality of the latent space must be at least as high as the dimensionality of the manifold. Since P is task-dependent, the encoding will be task-dependent as well. For the basic pushing environment (Figure la). P is a 4D manifold (since the plans only depend on the initial and final 2D box positions). For the obstacle task (Figure Ic P is a 6D-manifold (since the plans depend on one intermediate box position as well).
In the experiments discussed in the present work, we encode plans analytically using box positions as described above. We experimentally compare this with either learning the encoding or not using any encoding at all in section A.5
5.3 Baselines
We compare L2E against (1) direct plan execution, (2) plan execution with inverse dynamics model, (3) using RL to reach subgoals, and (4) HER. We describe these basclines in detail in section A.2
5.4 Results
Both at training and evaluation time, we run episodes of length 250. For each method (i.e., L2E and all baselines), we independently train 10 agents. After N environment transitions, we evaluate the agents. We reset to random initial positions and goals/plans and run the experiment until the goal reached or until the episode ends. We repeat this process M = 30 times for each agent, and store whether the rollout was successful in reaching the goal. We denote the result of the m-th evaluation of the a-th agent for method q, evaluated after learning for N environment transitions, as F(1) (N).
As can be seen from the video given in the supplementary material, even though the L2E agent uses plan information by the shaping reward, it can also learn to deviate considerably from the plan if closely following it will be suboptimal for reaching the goal fast. For example, while the simple planner (see Figure Ia and Figure Tc) suggests to re-establish the contact during the sequence, the 1.2E agent almost always moves and turns the box using a single contact.
guiding bias during exploration, and is encouraged to stay close to the plan
5.4.1 Basic Pushing Environment
To allow for fair comparison, we spent considerable amount of effort to optimize the HER replay strategy as well as the L2E strategy. Details on this are given in section A.4
8

---

The results for the pushing setup are summarized in Figure lb We observe that both L2E versions outperform all baselines in terms of the asymptotical performance. L2E with biased replay strategy 100,1000 exhibits high sample efficiency especially in the beginning, resulting in success rates significantly higher than 50% after 4000 episode rollouts or Million time steps. Directly executing the plan results in very low success rates of significantly less than 20% on average. Executing the plan with an inverse model (IM) still shows significantly worse long-term performance than the RL methods. HER results in better policies than the IM baselines, but is relatively data hungry. This can be improved slightly if the HER agent is only used to reach subgoals given by the planner.
Pushing is challenging interaction that requires reasoning for several time steps ahead. A typical failure mode of the IM baseline (see also videos) is that the box moves away from the intended trajectory too much, so that the agent is not able to correct for it within one time step. In contrast, the 1.2E agent learns to deviate from the planned trajectory if this required to reach the goal.
We find that L.2,, combining a model-based planner and a universal plan-conditioned policy, outper- forms our baselines that are pure planning or pure learning approaches. In addition, L2E outperforms the two baselines that also combine learning and planning.
5.4.2 Obstacle Pushing Environment
L,2 performs significantly better than the pure learning HER baselines, the pure planning baseline ("Plan"), and the "Planned Subgoals + RL" baseline. While using an inverse model is initially more efficient, L2E achieves significantly better results if given enough data.
Comparing the basic pushing environment (section 5.4.T) to the obstacle environment, L2E learns slower in the latter. This is in part due the higher dimensionality of the latent space of plan encodings (see also section [.2)). posing more challenging learning problem to the L2E agent. In contrast, the "Plan+IM" baseline independent of the size of the plan space, and performs comparably to the experimental setting in the original version.
The obstacle in the middle segments the state space into two parts. In order move from one side to the other, an agent already has to be able to reason about long-term results of its actions. As evidenced by the results for HER, this poses a significant challenge for pure RL. Incorporating planner knowledge helps the agent to overcome this chicken-and-egg problem.
6 Discussion
Learning plan-dependent policies as opposed to goal-dependent policies has the additional advantage that the former can learn to execute multiple plans that lead from the same initial state to the same goal, as shown in the obstacle environment. Thus, the policy learns multiple strategies to achieve the same outcome. In principle, this allows it to adapt to changed scenarios where some of these strategies become infeasible. If, e.g., the environment changes, it suffices to only update the planner's crude model of the environment so that it creates plans that are feasible again. These can then be directly fed into the policy without retraining. We explore this possibility in section A.3] using simple 2D maze environment with moving obstacles. We find that the achieves 90% success rate in this quickly changing environment, while the conddiiiit HER policy does not improve beyond 60% success rate.
L2E policy consistently
We used rather simple plans to support the RL agent during training, and demonstrated that these are already sufficient to significantly speed up learning in our experiments. In fact we demonstrate in section section|A.66thal in the basic pushing example, the L2E agent is very robust against plans of even lower quality. Using simple plans enabled us to use an analytical encoding: for very complex scenarios it might be beneficial to learn the encoding using auxiliary objective (see, e.g.. Co-Reyes et al. (2018)) ). We present results on using a variational autoencoder (VAE) in section A.5
The use of FV-RS biases the RL agent towards following the plan. While it was shown in the experiments that the RL agent can learn to deviate from the plan, plans that are globally misleading can act as distraction to the agent. In the present work, is assumed that plans can be used to guide the agent during learning, increasing sample efficiency. Independently of the specific method used to achieve this, misleading plans will always break this assumption.

---

Comparing the basic pushing environment to the obstacle pushing environment, the amount of data needed for learning a plan-conditioned policy clearly depends on the size of the plan spaces that are considered. For very large plan spaces, more data will be needed to master the task. Still, including planner information into the learning process makes decisive difference, as demonstrated by the relative performance of L.E and HER in the obstacle example.
While SAC was used for the experiments in section L2E can be used in combination with any off-policy RL algorithm. L2E reformulates goal-conditioned MDP as a plan-conditioned MDP, and provides a replay strategy to efficiently solve the latter. is agnostic how this data then used by the RL agent.
The specific FV-RS shaping function used in this work applies to MDPs with sparse rewards. We focused on this since sparse rewards are common robotic manipulation. In addition, they often present notoriously hard exploration tasks, making external plan-based information used by L2E particularly useful. However, FV-RS in general is not restricted to sparse-reward settings, and by using a different shaping function, L2E could be applied in other settings well.
Apart from FV-RS, there are alternative schemes of reward shaping such as potential- reward shaping (PB-RS) Ng et al. (1999). In principle, these could also be used to increase the sample efficiency of the RL agent. We chose FV-RS for two reasons. First, in the original paper Schubeert et al. (221), it was demonstrated that FV-RS leads to significantly higher sample efficiency than PB-RS. Second, since PB-RS leaves the optimal policy invariant, the behavior of the fully converged policy trained with PB-RS will only be goal-dependent, and not depend on the rest of the plan.
The original HER paper (Andrychowicz et al., 2017) considers the use of a simple form of reward shaping in combination with HER as well. It is found that reward shaping dramatically reduces the performance HER in robotic pushing task. In the present work, we show contrast that including plan information using FV-RS shaping improves the performance of RL in similar task. A possible explanation to reconciliate these seemingly contradictory results is already offered by Andrychowicz al. 2017): While simple Nomain-agnosse shaping functions
be distraction for the RL agent,
sppeiis reward shaping functions can be beneficial. This view is supported, e.g., by similar results by Popov al. (017). Andrychowicz et al. (017)) state that however "designing such shaped rewards requires a lot of domain knowledge". In this context, one could view 1.2E as an automated way extract such
knowledge from model-based planners and make it available. We specifically believe that 1.2E can be useful in robotic manipulation tasks, where domain knowledge is in fact readily available in many cases. Here, L2E offers way to exploit this.
Conclusion
We introduced L2E, an algorithm that links RL and model-based planning using FV-RS. RL generally results in well-performing policies but needs large amounts data, while model-based planning is data-efficient but does not always result in successful policies. By combining the two, L2E seeks to exploit the strengths of both approaches. We demonstrated that L2E in fact shows both higher sample efficiency when compared to purely model-free RL, and higher success rates when compared to executing plans of a model- planner. In addition, L2E also outperformed baseline approaches that combine learning and planning in our experiments.
Acknowledgments and Disclosure of Funding
The authors would like to thank Valentin N Hartmann for stimulating discussions. The research has been supported by the International Max Mlanck Research School for Intelligent Systems (IMPRS- IS), and by the German Research Foundation (DFG) under Germany's Excellence Strategy EXC 2120/1-39083161 "IntCDC" and EXC 2002/1 "Science of Intelligence".
References
Nvidia physx product site, May gameworks-ppysx- overvien
2021.
URL https://developer.nviii com/
10

---

Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In Advances in neural information processing systems, pp. 5048-5058, 2017.
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. A survey of robot learning from demonstration. Roboties and autonomous systems, 57(5):469 483, 2009.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv: 2016.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv preprint arXiv:1909 12200, 2019.
Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Picter Abbeel, and Wojciech Zaremba. Transfer from simulation to real world through learning deep inverse dynamics model. arXiv preprint arXiv:1610.035/1 2016.
John Co-Reyes, YuXuan Liu. Abhishek Gupta, Benjamin Eysenbach, Pieter Abbeel, and Sergey Levine. Self-consistent trajectory autoencoder: Hierarchical reinforcement learning with trajectory embeddings. In International Conference Machine Learning, pp. 1009-1018, PMLR, 2018.
Aleksandra Faust, Kenneth Oslund, Oscar Ramirez, Anthony Francis, Lydia Tapia, Marek Fiser, and James Davidson. Prm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-bast basedd planning. In 2018 JEEE International Conference on Robotics and Automation (ICRA). pp. 5113-5120. IEEE, 2018
David Foster and Peter Dayan. Structure in the space value functions. Machine Learning, 49(2): 325-346, 2002.
Yijie Guo, Jongwook Choi, Marcin Moczulski, Shengyu Feng. Samy Bengio, Mohammad Norouzi, and Honglak Lee. Memory based arXiv preprint arXiv: 10247, 2019.
policies for learning from sparse rewards.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, pp. 1861-1870. PMLR, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Amold, Deep g-leanning from demonstrations. arXiv preprint arXiv: 17Xvv:1704.03322, 2017.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When trust your model: Model-based policy optimization. arXiv preprint arviv: /e..vv::96.... 09253, 2019.
Rae Jeong, Yusuf Aytar, David Khosid, Yuxiang Zhou, Jackie Kay, Thomas Lampe, Konstantinos Bousmalis, and Francesco Nori. Self-supervised sim-to-reat adaptation for visual robotic manipula- tion. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 27 1722447.. IEEE, 2020.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAL, pp. 1094-1099, Citeseer, 1993.
Akira Kinose and Tadahiro Taniguchi. Integration of imitation learning using gail and reinforcement learning using aeceaccheeeemntt rewards via probabilistic graphical model. Advanced Robotics, 34 (155:105) 2020.
W Bradley Knox and Peter Stone. Combining manual feedback with subsequent mdp reward signals for reinforcement learning. Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume I, pp. 5-12. Citeseer, 2010.
11

---

Scott Kuindersma, Robin Deits, Maurice Fallon, Andrés Valenzuela, Hongkai Dai, Frank Permenter, Twan Koolen, Pat Marion, and Russ Tedrake. Optimization- based locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous robots, 40(3):429 455, 44044499-55,,0 2016.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet, Learning latent plans from play. Conference on Robot Learning, pp. 1113-1132. PMLR, 2020.
Andrew W Moore, Leemon Baird, and Leslie P Kaelbling. multi-value-fuuctions. E cient automatic action hierarchies for multiple goal mdps. In Proceedings of the international joint conference on artificial intelligence, pp. 1323, 1999.
Igor Mordatch, Emanuel Todorov, and Zoran Popović. Discovery of complex behaviors through contact-invariant optimization. ACM Transactions Graphics (TOG), 31(4):1-8, 2012.
Igor Mordatch, Kendall Lowrey, and Emanuel Todorov. Ensemble-cio: Full-body dynamic motion planning that transfers to physical humanoids. In 2015 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 5307-5314. IEEE, 2015.
Soroush Nasiriany, Vitchyr H Pong, Steven Lin, and Sergey Levine. Planning with goal-conditioned policies. arXiv preprint arXiv:1911.088 1.8453,, 2019.
Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations:
Theory and application reward shaping. In ICML, volume 99, pp. 278-287, 1999.
Andreas Orthey. spy-scripting.
scrppppi 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019, URL i neur cc/pppprrr aat--ytoorda-aa--mmrattetee amw-impraatieeetttleee www.-hoooooooonncceeeeeee.ee www.rraawnneeeeeeeeeeeoee.eeea libbary.. pdf
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. Deepmimic: Example-guide deep reinforcement learning of physics-basee character skills. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
Vitchyr Pong, Shixiang Gu, Murtaza Dalal, and Sergey Levine. Temporal difference models: Model- free deep rl for model-based control. arXiv preprint arXiv:1802.0900 09081, 2018.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Vecerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint 7XXv:1704.03073 2017.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3. htttps://github.comm// www.:lR-ooooooooe. www.a///e/../////aaaaeeeeeeeeeee baselllnsseeeeeeeeeeeeeee 2019.
Harish Ravichandar, Athanasios S Polydoros, Sonia Chernova, and Aude Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3, 2020.
Tim Salimans and Richard Chen. Learning montezuma's 's revenge from a single demonstration. arXiv preprint arxiv:
2018
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning, pp. 1312-1320, 2015.
12

---

Ingmar Schubert, Ozgur S. Oguz, and Mare Toussaint. Plan-based relaxed reward shaping for goal-directed tasks. ICLR 2021 9th International Conference on Learning Representations. 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via superviudd world models. In International Conference on Machine Learning, pp. 8583-8592. PMLR, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. 2018.
Matthew Taylor, Halit Bener Suay, and Sonia Chernova. Integrating reinforcement learning with human demonstrations of varying ability. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volune Volume 2, pp. 617-624, 2011.
Andrea Lockerd Thomaz, Cynthia Breazeal, et al. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Aaai, volume 6, pp. 1000-1005. Boston, MA, 2006.
Mare A Toussaint, Kelsey Rebecca Allen, Kevin Smith, and Joshua B Tenenbaum. Differentiable physics and stable modes for tool-use and manipulation planning. 2018.
Vivek Veeriah, Junhyuk Oh, and Satinder Singh. Many-goals reinforcement learning, arXiv preprint arXiv: 2018.
Fanjin Zeng. Python implementation of rapidly random tree (rri) path-planning algorithm, 2019. http:://.......oooooo/ooooooaaaa
Tianhao Zhang, Gregory Kahn, Sergey Levine, and Pieter Abbeel. Learning deep control policies for autonomous aerial vehicles with mpc-guided policy search. In 20/6 IEEE international conference on robotics and automation (ICRA), pp. 528-535, IEEE, 2016.
13


Reviews:
[{"review_id": "yTFoLo0OfLG", "reviewer": "Reviewer_PkED", "summary": "The paper proposes Learning to Execute (L2E), which can make use of a pre-designed planner in reinforcement learning (RL) in robotics. The authors extend the idea of final-volume-preserving reward shaping (FV-RS) and formulate plan-conditioned MDPs for designing L2E. The method was evaluated through a simulation experiment.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has certain originality regarding the idea of introducing FV-RS, especially from the theoretical viewpoint.\nHowever, the validity of this method was not demonstrated sufficiently. Comparison with more pre-existing methods is expected. \n\nThe quality of this paper is generally reasonable.\nThe clarity of this paper is also reasonable. \n\nHowever, the contribution is not significant. It is incremental because it is a straightforward application of FV-RS. For that application, some assumptions, which seem unrealistic in the real-world robotics environment, are introduced.  This point relates to the definition of \"model-based planner\" in this paper, which will be discussed in this review's \"limitation\" part. \nAt least, the significance is not demonstrated sufficiently through the experiment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper is motivated by the real-world robot application of RL.  (At least, the authors insisted so.) Unfortunately, the approach and discussion do not seem to grasp the nature of the real world and actual situations in robotics. That seems to be noted to start the discussion of the sample efficiency. A critical issue of this paper is the unclear definition of \"model-based planner.\" It is quite unclear to me. What kind of controller system and settings in robotics do the authors assume?  As far as I read through the paper, it is some idealistic Mathematical element. \nSpecifically, I am not sure what kind of \"model-based planner\" can satisfy the condition (5). The goal of a planner (or a controller) in state space is not always available in robotics. At least, the uncertainty of the environment and some clear and realistic assumptions should be considered.   Of course, in a simulation environment and a simple scenario like an experiment in this paper (which we may call a \"toy problem\"), we can find a setting satisfying the condition. If \"model-based planner\" in robotics does not satisfy the condition (5), the approach is not valid for solved the problem raised at the beginning of this paper.  \nAlso, the term \"model-based\" is also used in \"model-based RL,\" and policy is also called planner in some context. From these views, the authors should clarify the definition of \"model-based planner\" in this paper and show examples of them in robotics.\n\nThe comparison is limited. The method is about the integration of a pre-designed policy and RL via sub-reward. In comparison, I recommend the authors to compare the proposed method using other types of sub-reward approach, e.g., [Andrew 1999].\n\nAlso, this study is closely related to the integration of imitation learning (IL) (i.e., learning from demonstration or expert information) and reinforcement learning. For example, the following paper integrates IL and RL via sub-reward based on control as an inference framework. The relationship is worth mentioning. \n\n- Akira Kinose, Tadahiro Taniguchi, Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model, Advanced Robotics, 34(16), pp.1055-1067, 2020. DOI: 10.1080/01691864.2020.1778521\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "sc8zw8DoLJb", "reviewer": "Reviewer_4YB3", "summary": "This paper addresses the problem of sample complexity in reinforcement learning. The approach is to drive down the sample complexity using reward shaping, where the reward shaping function is given by a planner. The paper shows how to use final-volume-preserving\nreward shaping (FV-RS) to convert a plan into a reward shaping function. The approach is evaluated on a simple pushing domain against planning in two forms, and Hindsight Experience Replay. The L2E approach proposed here outperforms the comparison algorithms. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like the idea of this paper and would argue for its acceptance. The technical ideas are strong, and the paper is (mostly) well-written. The primary limitation of this work is the experimental evaluation, especially in an increasingly crowded field of prior work combining planning and learning.  It is not clear to me that the most relevant baseline is HER. There are a number of more relevant works, such as PRM-RL (Faust et al, 2014) and Hoel et al (2020), both of which explicitly combine planning and learning for exactly the same purpose as L2E, and there are other references as well. I think the real novelty is the use of FV-RS as the shaping function, but the other papers in this line of research need to be considered. \n\nI would recommend substantially strengthening the experimental evaluation. The single domain of box pushing is not particularly compelling, and does not show off the strength of incorporating a planner in the reward shaping function — what would be better would be problems with a much longer horizon and places where a significant deviation from a greedy strategy would be important, but with stochasticity to be avoided. (BTW: the plan shown in figure 1 was very hard to interpret. I now realize that the reason the end effector is changing height is because it is going over the box to change sides to push, but it took quite a while to figure that out. Some more explanatory text would help considerably. Maybe also draw the box at different along the trajectory with a significant alpha channel value.) Figure 1b was not a particularly effective way of communicating the results. It was not clear to me if these are the average and variances of the 10 agents on one problem, or the average and variances of the 10 agents times 30 random start/goal states per agent. \n\nEven for these experiments, I would recommend running L2E and HER longer — it is not clear to me that L2E $ S^{bias} _{10,1000}$ has converged, especially when looking at 1d. I assume the loss in performance for 10 plans from 1000 samples is because it has experienced fewer transitions. And if indeed L2E $ S^{bias} _{10,1000}$ has converged, I would want to know why it is not exploring further. \n\nEquation 3 is very important, and it is confusing and feels somewhat arbitrary — the paper relies too much on explication from the FV-RS paper, including replicating notation without explanation. The fact that f(p) is the final state of the plan is never stated, although the fact that f(p) is the goal on the following page allows the reader work backwards. $p_{k(s)}$ should be explained, as well as the fact that the subscript is indexing the plan state. Even with the the FV-RS paper in hand, the justification for the $k(s)+1/L$ term is unclear to me. I would encourage the authors to add explanatory text for the terms in the FV-RS reward shaping equation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I appreciated that the author checklist pointed at the specific section where they were attempting to discuss limitations. Of the papers that I reviewed, this paper probably provided the best analysis of the limitations of the work. That being said, the discussion of limitations was quite cursory. There is room for considerably more analysis of the limitations of this work, especially in the experimental stage. It seems clear that the biggest limitation of the work is the loss in optimality induced by FV-RS. I expect this loss in optimality will create very negative consequences when the optimal plan and optimal policy are wildly divergent. \n\nFor instance, imagine a world with two routes to the goal, where one route is highly stochastic with very high expected cost, but is much shorter, e.g., a short cut along a cliff’s edge. The other route is much longer is far less stochastic (or just has no negative outcomes), so much lower expected cost (a route away from the cliff’s edge). if the planner assumes a deterministic world, it may incorrectly assume that the cost of the path along the cliff’s edge is much lower than the cost of the longer route, and very strongly bias the learner in the wrong direction. \n\nThis is not to say that this paper is wrong or incorrect in some way, but that obvious limitation does have to be addressed. \n\nThe author checklist indicated that the authors did not discuss any potential negative societal impacts of the work. \n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours "}, "extra_text": {}}, {"review_id": "BkBYZBNOARE", "reviewer": "Reviewer_TpKX", "summary": "This paper presents a method for combining plan-conditioned policies with reward-shaping and a given approximate planner to a sparse-reward, seemingly challenging robotic manipulation task. However, the paper is lacking in experiments, which is the main reason I am voting for rejection.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Paper Strengths\n**Intuitive Method**\nThis paper demonstrates that a plan-conditioned policy combined with an FV-RS reward shaping function is able to allow for better performance on goal-conditioned, sparse reward tasks. The method is novel, although the contribution isn’t too surprising.\n\n**Hyperparameter Tuning**\nThe authors perform a proper hyperparameter tuning scheme for HER when comparing against the baseline.\n\n**Performance Improvement**\nThe method presents a modest performance improvement over the baselines.\n\n## Paper Weaknesses\n**One environment, one task**\nThe authors should evaluate on at least 2, preferably 3+ environments/tasks to truly demonstrate the advantage of their method. This is one large reason I am currently not voting for acceptance.\n\n**Method Clarity**\nQuestion: Are plans embedded or given in full to the plan-conditioned policy?\n\n**Ablation Studies**\nHow does planner quality affect L2E? How do plan lengths affect L2E? Can you show ablations here?\n\n**FV-RS Introduction**\nThe $F_{FV}$ reward is discussed but not given an intuitive explanation. This section, and the method in general, would be more clear if all 3 terms were explained intuitively in Eq.3 and their effects on policy behavior were discussed explicitly.\n\n**Missing Related Works**\nCombining model-free and model-based RL is presented as a contribution, however there are works that have done this before (see for example, “When to Trust Your Model: Model-Based Policy Optimization by Janner et al, and “Temporal Difference Models” by Pong et al). The authors should add citations to these works and contrast L2E with them.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "L2E limitations are actually not discussed, despite a reference to the discussion in the checklist.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "3--5H7-QAvL", "reviewer": "Reviewer_CAJo", "summary": "The paper proposes to condition policies on plans. Therefore, the policy learns to execute an approximate plan and correct it if necessary. This idea is instantiated in an algorithm that uses SAC for policy optimization and a \"crude manhattan-like planner\" for planning. The algorithm is evaluated on a box pushing task, where an improvement in sample-efficiency over goal-condition RL is shown, and an improvement in success rate compared to model-based plan execution with learned inverse dynamics model is demonstrated.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of conditioning policies on expert plans as presented in this paper is novel according to my knowledge. The paper is of good quality and it is written clearly. Together with the code in the supplementary materials, the paper provides sufficient details to reproduce the results. However, the evaluation is performed only in one environment and there is no discussion or evaluation of the influence of the function that encodes the plans before passing them to the policy. This lack of evaluations makes it hard to judge the significance of the proposed method as it is not clear if it generalizes to other settings. Since the main competitive approach considered in the paper is HER, it might make sense to add evaluations of the proposed L2E algorithm in the environments from the original HER paper.\n\nMajor comments\n- line 231: \"we analytically encode the plans into a 4D latent space\". Provide the details of this mapping in the paper since this is one of the crucial elements of the proposed approach. Why 4D? What properties should this mapping satisfy? Does the dimensionality of the latent space depend on the task? Either theoretical study or empirical evaluations of the choices of this mapping should definitely be provided, because conditioning the policy on plans instead of goals is the main difference of the proposed method  compared to goal-conditioned RL.\n- lines  272-276 say that L2E agent significantly deviates from the plans. This appears to imply that conditioning on the plan may not be so informative to the agent. What is the advantage of having a plan-conditioned policy instead of a goal-conditioned one? It is shown in the pushing experiment that with the chosen hyperparameters one achieves faster convergence, but are there any benefits in general that one should expect from feeding in plans instead of goals?\n- it would be beneficial to the reader to contrast the proposed approach to guided policy search (GPS) and DeepMimic, as those methods also involve imitation of an imperfect teacher, although they don't explicitly feed the plans as input to the policy\n\nMinor comments\n- line 178: plan replay strategy S_n is defined as \"a distribution over n replay plans\". However, in lines 181-182 it is said that the uniform replay strategy samples n unique plans uniformly from the replay buffer D, which contains more than n plans. Therefore, I think the authors mean that S_n is a distribution over integers 1, 2, ..., len(D). So, the subscript n seems misleading in S_n because the distribution itself does not depend on how many samples are drawn from it.\n- lines 198-199: what is meant by \"unstable dynamics\" in the pushing task? Every state of the system seems to be a stable equilibrium state if no control input is applied. Maybe a different term instead of \"unstable\" would be more appropriate here. The same in line 285.\n- typo:  line 146, policy should be conditioned on state instead of action\n\nComments after rebuttal\nI thank the authors for addressing my questions and  for providing additional experiments. I raise my score from 5 to 6. There are still quite a few concerns raised by other reviewers, such as providing comparisons to more directly related learning+planning algorithms instead of HER. Furthermore, the environment used for additional experiments is still a custom made environment, therefore it is hard to directly relate it to other papers. I am also not convinced by the argument that having a plan-conditioned policy is necessary to have a multi-modal policy: in soft Q-learning this is achieved by using an energy-based model for the policy, and if it is goal-conditioned, it may produce different solutions as samples from a multimodal distribution. For all these reasons, I can only say that the paper is marginally above the acceptance threshold.\n\nFinal comments\nThe authors addressed all my concerns and promised to add the results provided in the anonymous github repository to the paper. With these additions, I consider the paper appropriate for publication and raise my score to 7.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines learning and planning to increase data efficiency, motivated by robotics application scenarios. In more detail, the authors leverage the fact that in such scenarios, often, a coarse model is available, that allows for making a coarse, but probably suboptimal plan. The authors propose using this plan to provide shaping rewards to a RL agent that is then free to further optimize the policy. For this, they build on "plan based final volume preserving reward shaping" introduced by Schubert et al. The main contribution of the authors is to
- Allow policies that generalize over instances by conditioning on the plan
- Using this formulation to introduce plan-replay-strategies. 

Initially, the reviews was mixed, but after the discussion phase all reviewers recommend acceptance. The main reasons being: 
- Strong and original technical ideas
- Mostly well-written
- The experimental evaluation was initially somewhat limited, but during the discussion phase the authors have made these stronger by adding another environment and clarifying the baseline methods used. 
- The methods limitations could be more explicitly discussed. 

I'd like to add that I think that the authors could more explicitly discuss what is novel in their paper compared to the earlier Schubert paper. E.g., the FV-RS is directly introduced in a 'universal' formulation (3), making the transition to universal policies and reward shaping functions more smooth but also making this part of their contribution less explicit. 
Decision: Accept (Poster)