Training Transition Policies via Distribution Matching for Complex Tasks
Authors: JU-SEUNG BYUN
Andrew Perrault
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement Learning
Hierarchical Reinforcement Learning
Inverse Reinforcement Learning

Abstract: Humans decompose novel complex tasks into simpler ones to exploit previously learned skills. Analogously, hierarchical reinforcement learning seeks to leverage lower-level policies for simple tasks to solve complex ones. However, because each lower-level policy induces a different distribution of states, transitioning from one lower-level policy to another may fail due to an unexpected starting state. We introduce transition policies that smoothly connect lower-level policies by producing a distribution of states and actions that matches what is expected by the next policy. Training transition policies is challenging because the natural reward signal---whether the next policy can execute its subtask successfully---is sparse. By training transition policies via adversarial inverse reinforcement learning to match the distribution of expected states and actions, we avoid relying on task-based reward. To further improve performance, we use deep Q-learning with a binary action space to determine when to switch from a transition policy to the next pre-trained policy, using the success or failure of the next subtask as the reward. Although the reward is still sparse, the problem is less severe due to the simple binary action space. We demonstrate our method on continuous bipedal locomotion and arm manipulation tasks that require diverse skills. We show that it smoothly connects the lower-level policies, achieving higher success rates than previous methods that search for successful trajectories based on a reward function, but do not match the state distribution.


Published as a conference paper at ICLR 2022
TRAINING TRANSITION POLICIES VIA DISTRIBUTION MATCHING FOR COMPLEX TASKS
Ju-Seung Byun & Andrew Perrault Department of Computer Science Engincering
The Ohio State University Columbus, OH 43210, USA (byun. 83, perrault. 17w......edleee.e
ABSTRACT
Humans decompose novel complex tasks into simpler ones to exploit previously learned skills. Analogously, hierarchical reinforcement learning seeks to leverage lower-level policies for simple tasks to solve complex ones. However, because each lower-level policy induces different distribution of states, transitioning from one lower-level policy to another may fail due to an unexpected starting state. We introduce transition policies that smoothly connect lower-level policies by pro- ducing a distribution states and actions that matches what is expected by the next policy. Training transition policies is challenging because the natural reward signal-whether the next policy can execute its subtask successfully-is sparse. By training transition policies via adversarial inverse reinforcement learning to match the distribution of expected states and actions, we avoid relying on task- based reward. To further improve performance, we use deep Q-learning with a binary action space to determine when to switch from transition policy to the next pre-trained policy, using the success or failure of the next subtask as the re- ward. Although the reward is still sparse, the problem is less severe due to the simple binary action space. We demonstrate our method on continuous bipedal locomotion and arm manipulation tasks that require diverse skills. We show that it smoothly connects the lower-level policies, achieving higher success rates than previous methods that search for successful trajectories based on reward func- tion, but do not match the state distribution.
INTRODUCTION
While Reinforcement Learning (RL) has made significant improvements for a wide variety con- tinuous tasks such locomotion (Lillierap 2016 Heess al., 2017) and robotic manipulation GGoss et al. 2018), direct end-to-end training on tasks that require complex behaviors often fails. When faced with complex task, humans may solve simpler subtasks first and combine them. Anal- ogously, hierachical reinforcement learning (HRL) uses multiple levels of policies, where the actions of higher-level meta-controller represent which lower-level policy to execute (Sutton et al. 1999 1992). A multitude of methods has been proposed to train HRL, such as assign- ing subgoals for lever-level policies (Kulkarni et al. 2016), employing pre-trained lower-level policies (Frans et al. 2018), and off- aff-policy RL. (Nachum et al. 2018).
Dayan & Hinton
All HRL methods face common challenge: switching smoothly from one lower-level policy to another when the meta-controller directs that switch should occur. For example, running hurdles may require switching between running and jumping repeatedly while maintaining forward momen- tum. Previous approaches this challenge (Frans et al. 2018 Andreas et al.) 2017 Lee et al. 2019) involve either retraining existing lower-level policies and/or introducing a new policy to exccute the switch. We focus the latter and call the new policy the transition policy, following Lee et al. 2019).
A transition policy needs to switch between the states produced by lower-level policy to an appropriate starting state for another. A natural way to train such a policy is to give a reward de- pending whether the transition is successfully executed not. However, this method induces

---

Published as a conference paper at ICLR 2022
A prior pre-trained policy
Ty. A post pre-trained policy
Transition interval
R122 A transition policy for a and R2
I: Initial states for T322
Distribution m.
Distribution of
Figure I: Transitions between lower-level policies for different subtasks often fail because one pol- icy #2 is not trained on states generated by another policy Th A transition policy 11,2 starts from the state distribution induced by и at time of transition. We propose training 11,2 to produce states and actions from the distribution of T22 using inverse reinforcement learning techniques.
sparse reward that is difficult to learn from due to the temporal credit assignment problem (Sutton 1984). Instead, states produced by the transition policy to the distribution of states expected by the next lower-level policy (Figure [] While the goal of IRL is infer expert's reward function from demonstrations of optimal behavior (Ng et al., 2000 Ziebart et al. 2008). some recent approaches to IRL (e.g,,
inverse reinforcement learning (IRL) techniques match the distribution of
Finn al. (2016); Ho & Ermon (2016); Fu et al. (2018)) learn policy that imitates the expert's demonstrations in the process. In particular, lar, Ho & Ermon (2016) propose generative adversarial network (GAN) structure (Goodfellow et al. 2014) that matches the distribution of the policy (gen- erator) with the distribution of the given data (expert demonstrations). We use this GAN structure for distribution matching, and, consequently, we avoid the problem of designing an explicit reward function for transition policies.
Although the transition policy is trained to produce the distribution of states and actions that is expected by the next lower-level policy, not all states in the distribution's 'ss support have equal prob- ability of success. To increase the rate of successful execution, we introduce a deep Q-network (DQN) (Mnih et al., 2013) to govern the switch from the transition policy to the next lower-level policy. The DQN has two actions, switch or stay, and a simple reward function where a positive re- ward is obtained if the next lower-level policy is successfully executed; otherwise, negative reward given. The issue of sparse rewards is present here, but it is less of an issue because of the simple decision space.
The main contribution of this paper is new method for training transition policies for transitioning between lower-level policies. The two parts of our approach, the transition policy that matches the distribution of states and the DQN that determines when to transfer control, are both essential. Without the DQN, the state selected for transition may be familiar the next policy, but lead to a poor success rate. Without the transition policy, there may be no appropriate state for the DQN transition from. The proposed method reduces issues with sparse and delayed rewards that are encountered by prior approaches. We demonstrate our method on bipedal locomotion and arm manipulation tasks created by [ee al.) (2019), in which the agent requires to have diverse skills and appropriately utilize them for each of the tasks.
2 PRELIMINARIES
The problem is formulated as a infinite-horizon Markov decision process (MDP) (Puterman) 2014 Sutton & Barto 2018) defined by the tuple M AS P, R, y, 11.. The agent takes action a₂ E A for the current state $; E at time t, then the reward function R : x A + R returns the reward for (St, at), and the next state 8++1 is determined by the transition probability P(* 0r). 7 is the discount factor and denotes the initial states distribution. The agent follows a policy To (t)
that produces& distribution of actions for each state. The objective of reinforcement learning (RL) is to find the optimal 0 that maximizes the expected discounted reward:
8" = argmax
E sonu Ge He Set P-
Σ r R(s ac)
(1)
2

---

Published as a conference paper at ICLR 2022
Meta-controller
with access to policies
7g111
T
Signal transition from n, to
Signal transition from ग. to TT
TTc,d
Transition interval
Transition interval
Figure 2: HRL structure with transition policies and DQNs. The meta-controller has n pre-trdined lower-level policies {TT. T2, ... ). To transition between two policies (for example, Ta and ma), the meta-controller signals transition interval. We train a transition policy (Ta,,) that activates at the beginning of this interval and DQN (9a.b) that determines when during the interval the switch from Ta to ny should take place.
2.1 INVERSE REINFORCEMENT LEARNING
Whereas standard RL trains a policy to maximize reward function, inverse reinforcement learning (IRL) (Ng et al. 2000 Ziebart et al. 2008) seeks to learn the reward function from expert demon-
strations of an optimal policy. Several approaches to the IRL problem (Ho & Ermon 2016 Fu
et al. 2018) adopt GAN structure (Goodfellow et al.. 2014) the policy (generator) tries to select actions that the discriminator cannot distinguish from the expert's. As a by-product, the generator produces policy with the same distribution the expert's 's at convergence. We use the state-action discriminator (Fu et al. 2018) rather than the trajectory-centric discriminator (Finn et al. 2016 Ho
& EErmon 2016):
exp(Jv.0 a1 8t+1)) expífv.o at, s(41)} + To (agas))))
De az 8141) =
(2)
where fe (8t, 0e 8+41) = (8,, (e) +hh (8,) time i, Po the reward approximator, and Φ are the parameters of shaping term has which alleviates unwanted shaping for Jø.
This discriminator Da.o is trained to distinguish data from the expert and the generator's policy via binary logistic regression, and the policy is updated with Proximal Policy Optimization (PPO) (Schulman et al. 2017) according to the rewards Tv.o provided by the discriminator:
( at Sc+1) log D az, S141) - log(1 Dv au D, S(+1))
2.2 DEEP Q-LEARNING
2013) is variant of Q-learning (aakinns & Dayan 1992) that produces neural network called deep Qenetwork (DQN) that aims learn determin- istic optimal action from among discrete set of actions for each state. Deep Q-lcarning suffers from overestimating Y/ (Trun & Schwartz 1993) because of the max term, which impedes the learning process. Double Q-learning (van Hasselt et al., 2016) addresses this problem by decoupling the networks for choosing the action and evaluating value. The target network Qø makes learning more stable by providing target y/: that is fixed for long periods of time:
Deep Q-learning Riedmiller 2005 Mnih et al.
y = rz + 7 Qa (8 argmax.,,, Q0(s4+1,
The DQN is updated using a loss function such as the mean squared error (u/ Qa a²)) We train DQN with double Q-learning to get signal that tells when switch from transition policy to the next lower-level policy.
3 APPROACH
In this section, we present methodology for transitioning between lower-level policies hierar- chical reinforcement learning (HRL). We assume that receive pre-trained lower-level subtask
3

---

Published as a conference paper at ICLR 2022
Algorithm Training Transition Policy Taabb
1: Input: pre-trained policies (Ta, Jb transition interval
2: Initialize transition policy (generator) Ra.6 and discriminator D Equation
3: Collect the states ST, of Fa at the start of the transition interval
4: Collect the trajectories (states and actions) Tm, of iT. during the transition interval
5: Initialize transition policy Ta,b and discriminator Dv.o
6: for = 1 to n do
7: for j j- to m do
8: Sample s Sz. and set the state of Ta b to s
9: Collect the trajectory Tu..s generated by the current To.b
10: end for
11: Train D&. with binary logistic regression to distinguish TTa and T
12: Calculate reward ri set for all (8t, at, 8t-1) E TT... with Equation
13: Optimize Ta.b with respect to Tx.,s and Tv.
14:
end for
policies input along with transition intervals-windoww where transition between two specific lower-level policies should occur. We seek to maximize performance on complex task by com- bining the pre-trained policies without re-training them. To do this, we train a transition policy for each pair of pre-trained policies that we must transition between. The transition policy is activated the start of each transition interval. Then, a DQN governs the switch from the transition policy to the next prescribed pre-trained policy. The DQN improves the rate of transition success relative to simply switching at the end of the transition interval
In Section 1. we describe our training procedure for transition policies. In Section 3.2, we describe how we collect data and design rewards for DQNs. Our framework summarized by Figure 2
3.1 TRAINING THE TRANSITION POLICY
Let [T T2...., TnJ } denote n pre-trained policies and let Ta. denote a transition policy that aims to connect policy Ta to Tp. The meta-controller dictates a transition interval for each transition between pre-trained policies-a period of time when the transition should take place. If we were to simply switch from Ra to Ty at the start of this interval, my can easily fail because it has never been trained on the last state produced by Ta. To prevent this problem, Ra,b needs to be able to start from any of the last states produced by Fa and lead to a state from which Tr can successfully perform the next task. Which states are favorable for starting mt? A possible answer is the states produced by mb itself.
We train na. have the same distribution T. with IRL (Ng al. 2000 Ziebart al. 2008 Ho
& Ermon 2016 Fu et al. 2018). We specifically choose adversarial inverse reinforcement learning (AIRL) (Fu et al.) 2018), which has the GAN (Goodfellow et al. 2014) structure. The training procedure for Ja,b is described in Algorithm [ Ra,6 is trained with two classes of data: one from T, and one from Th. From Te. we collect the states Sr,, of na at the start of the transition interval. From Th. we collect the trajectories Tr, of ma during the the transition interval. The initial state of the generator Na,b is set to 8, sampled from ST..- Then, Ta.b tries to generate trajectories that fool the discriminator D (Fquation 2 into classifying it as an example of T The discriminator D is trained with binary logistic regression to distinguish between the data from म. vs. To.b. When D can no longer distinguish between them, No,b can be said to have a similar distribution to Th.
For example, we prepared a pre-trained Wulking forward policy कı and Jumping policy ma for one of our experimental environments, Hurdle, manually shaping rewards for these two subtask policies. We use a transition interval of 3-4 meters from the hurdle. Thus, S,, is the last states of π the the sampled set of states when the agent's location is meters from the hurdle. Ta is the trajectories
'ee assume that the environment is static enough that it is not critical that the meta-controlle has the opportunity to replan after the transition policy terminates. In principle, such replanning could be allowed by introducing an additional layer of transition policies, i.e., by training training meta-transition policies that transition between transition policies and actions.
4

---

Published as a conference paper at ICLR 2022
Algorithm 2 Training Deep Network do.b
1: Input: pre-trained policies (Ta, Tb), transition policy Ta,b, transition interval
2: Initialize 4 and replay buffer B
3: for i to n. do
4: Set environment to state Ta at the start of the transition interval
5: while the current state is in the transition interval
6: Ot ~ (8t)
7:
Execute environment for (st, 0+) and get 8441 and success-fail-alive signal
8:
Run da and get an action dg
9:
if og switch
10:
set s = $t+1 and break
11:
else if 0g stay
12:
Store (St: aq. y 8141) in B if fail or s is outside the transition interval
13:
Store (st; 0g. 0, 8841) in B if r alive
14:
end if
end while
16:
while To has not achieved subgoal and has not failed
Gç (8+)
18:
Execute environment for (st, 0z) and get 81+1 and signal
19:
end while
20:
Store (8, ag 8t, in B if success
21:
Store (8, ag: ry. 88+1) in T = fail
22:
Update 4a with minibatch from B according to Equation
23:
end for
of #ं from 3-4 meters from the hurdle. The transition policy æ1,2 2 is initialized with of the last states of गy and learns to match the distribution πe just before the jump is initiated.
3.2 TRAINING THE DEEP Q-NETWORK FOR TRANSITION
With Ta,b. can proceed from the last states of the to the distribution of states of Tp. In order to select a state to switch from T..5 to Th that is more likely to lead to a successful execution of Ty. we introduce DQN (Riedmiller 2005 Mnih al. 2013 van Hasselt et al.) 2016), which we denote as 9a,b- At each time step, la,b outputs whether to switch, i.e., hand control from Ta,b to Ty, or stay, i.e. continue use Wa,b-
As shown in Figure 3 we design a simple reward function to train Ga.b- The process begins when the agent enters the transition interval and switches from F. Ta.b at one of the states in S... Then, lo becomes active and generates switch or stay action at cach subsequent time step, according the current state. If qu,b takes the switch action, the agent switches to me. The reward for this transition depends on the outcome of the subtask that Ty aims to complete. A positive value r, is given if the agent completes the next subtask; otherwise, the agent gets a negative reward ry. qa,6 takes the stay action, the policy aa, is continued. The agent immediately receives an alive reward of unless the agent enters a failure state, in which case it receives a reward of ry (For example, in the Hurdle environment, failure states are those where the agent collapses.)
la.b is exposed to sparse reward issue because of delayed reward for the switch action, but the issue is less severe compared to training the entire transition policy using the sparse subtask success reward. The multi-dimensional continuous control space is reduced to a single dimensional binary decision.
In summary, we introduce a method to train transition policies that smoothly connects pre-trained policies without having to design complex reward function and fine-tune the pre-trained policies. We use a DQN to increase the performance of the following pre-trained policy by controlling the timing of the switch from the transition policy to the next pre-trained policy.
5

---

Published as a conference paper at ICLR 2022
Tp
Tip
JTp
success: F,
success: F,
success: Fa fail:
success:
fail:
fail:r,
fail:
switch
switch
switch
switch
TT.b
JTub
Tigh
Tayh
bb((18,))
bal
stay
stay
stay
stay
alive: 0
alive:
alive: 0 fail:
fail:
fail::r
fail:
Figure 3: The sequential decision problem faced by the DQN during the transition interval. The transition interval runs from the ending state of Tax to 8t+kk The DQN qa. is active until it outputs a switch action, which passes control from the transition policy Ta,b to Ti. The switch reward depends on whether T6 achieves the subgoal: reward of rs is given for success a reward of r for failure. A reward of 0 is given for the stay action unless the agent enters failure state, in which case y is given.
4 RELATED WORK
Training agents that can generate diverse behaviors in complex environments is still open problem in RL. Although some complex tasks can be solved by careful modification of the reward function not simple and may produce unexpected behaviors (Ried-
(Ng et al. 1999), the modifications miller et al., 2018). HRL approaches may be able to avoid these issues by combining lower-level policies for subtasks that are easier train. We briefly summarize relevant past work in HRI..
The option framework (Sutton et al. 1999) use a two-level hierarchy: the higher-level policy is in charge of selecting the lower-level policies (options) and selecting the termination conditions for each option. The option-critic architecture for continuous problems (Bacon et al.) 2017) trains the higher-level policy jointly with the options. This approach may lead to options that terminate at every step, flattening the hierarchy. Combining pre-trained policies has been shown to be successful for learning diverse behaviors, under the assumption that the subtasks are known and well-defined (Heess et al., 2016 Florensa et al. 2017). Given subgoals and pre-trained policies. the h-DQN endeavors use the lower-level policies achieve the subgoals (Kulkarni et al. 2016). Frans et al. (2018) proposed an approach to solving unseen tasks by using pre-trained policies with the master policy. The feudal reinforcement learning framework (Dayan Hinton 1992) creates a hierarchy where there are managers and sub-managers in which manager learns to assign subgoals to their sub-managers (these sub-managers also have their sub-managers). All of these approaches require switching between lower-level policies, and, in principle, their performance could be improved by a transition policy. Because we do not alter the pre-trained policies at all, our method may perform poorly if the task is unseen and the pre-trained policies must be fine-tuned to perform well or if successful transition requires that the pre-trained policies are altered.
Our contribution is most elosely related past work on how transition between lower-level poli- cies. Tidd et (2021) introduced estimators that decide when to switch the current policy to the next policy. They trained the estimators with the assumption that the policies have common states. Our work is inspired by past approaches that train transition policies to connect sub-policies smoothly. Lee al. (2019) introduced proximity predictor that predicts the proximity to the initial states for the next policy. The proximity predictor is used as dense reward function for the transition policies, which otherwise struggle with sparse rewards. our work, we approach the sparse reward issue with distribution matching, and instead of transition policies producing signals for switching policies (Lee et al. 2019), we have independent Q-networks that give the transition signals.
An alternative to using generative adversarial reinforcement learning to match the state and action distribution would be to imitate the trajectories directly with behavioral cloning (Bain & Sammut
1993 Ross & Bagnell 2010). However, behavioral cloning suffers from the distribution drift prob- lem in which, the more the policy generates actions, the more the actions tend to drift away from the distribution of the expert; hence, we use the GAN structure of IRL for distribution matching.

---

Published as a conference paper at ICLR 2022
Table 1: Success count table for three manipulation tasks. Each entry represents average success count with a standard deviation over 50 simulations with three different random seeds. Our method performs better than using a single policy trained with TRPO and shows close results to Lee et al. (2019). Both Lee et al. (2019) and our method approach perfect success counts (5 for Repetitive picking up, 5 for Repetitive catching, and for Serve).
Repetitive picking up Repetitive catching
Serve
Single
0.69 + 0.46
4.54 1.21
0.32 0.47
Without TP
1.49 = 0.59
4.58 1.10
0.05 0.22
With TP
3.55 + 1.64
4.75 + 0.87
0.99 + 0.81
Lee et al. 2019)
4.84 + 0.63
4.97 + 0.33
0.92 + 0.27
With TP and Q (Ours)
4.77 + 0.91
4.76 + 0.82
1.00 + 0.00
EXPERIMENTS
We evaluate our method on six continuous bipedal locomotion and arm manipulation environments created by Lee et al.) (2019): Repetitive picking up, Repetitive catching, Serve, Patrol, Hurdle, and Obstacle course. All of the environments are simulated with the MuJoCo physics engine (Todorov) jet al.) 2012). The tasks are detailed in Section 5.1 and Section 5.2. Section 5.3 shows the results method performs better than using only single policy simply using pre-trained policies and is comparable to Lee et al. (2019) for the arm manipulation tasks and much stronger for the locomotion tasks. In Section 5.4, we visualize how transition policy trained through IRL resembles the distribution of pre-trained policies.
5.1 ARM MANIPULATION TASKS
A Kinova Jaco arm with 9 degrees of freedom and fingers is simulated for three manipulation tasks. The agent needs to do several basic tasks with a box. The agent receives the robot and the box's state as an observation.
Repetitive picking up: Repetitive picking requires the robotic arm to pick up the box 5 times. After picking up the box from the ground, the agent needs to hold the box in the air for 50 steps (to demonstrate stability). If the agent does so, the environment increments the success count by 1, and the box randomly placed on the ground. We prepared pre-trained policy Pick trained on an environment where the agent needs to pick up and hold the box once.
Repetitive catching requires the robotic arm to catch the box times. After catching the box, the agent needs to hold the box in the air for 50 steps. If the agent does the environment increments the success count by I, and the next box thrown 50 steps later. The pre- trained policy Catch is trained on an environment that requires the agent to catch and hold the box once.
Repetitive catching:
Serve: Serve is task in which the agent picks up a box from the ground, tosses it, and then hits the box at a target. For this task, there are two pre-trained policies Tossing and Hitting. The performance without using a transition policy is very poor because Ilitiing's 's initial states are different from the ending states of Tossing. By introducing the transition policy connecting Tossing 's end states to the start states of Hitting smoothly, the agent performs the task perfeetly.
5.2 BIPEDAL LOCOMOTION TASKS
The agents for three locomotion tasks have the same configuration. The agent is a 2D bipedal walker with two legs and one torso and 9 degrees of freedom. The agent receives (joint information, velocity, distance from an object) tuples as the state. All of the locomotion tasks require the agents to do diverse behaviors accomplish each subgoal.

---

Published as a conference paper at ICLR 2022
Table 2: Success count table for three locomotion tasks. Each entry represents average success count with a standard deviation over 50 simulations with three different random seeds. Our method performs better than using a single policy trained with TRPO and Lee al. (2019).
Patrol
Hurdle
Obstacle Course
Single
1.37 + 0.52
4.13 = 1.54
0.98 + 1.09
Without TP
1.26 + 0.44
3.19 = 1.61
0.41 + 0.52
With TP
3.11 + 1.58
3.43 + 1.60
2.44 + 1.63
Lee et al. (0199)
3.33 + 1.38
3.14 + 1.69
1.90 + 1.45
With TP and Q (Ours) 3.97 + 1.38
4.84 + 0.60
3.72 + 1.51
Patrol a task in which the agent alternately touches two stones on the ground. For this task, the agent must walk forwards and backwards. We trained subtask policies Walk forward, Walk backward, and Balancing. Balancing helps alleviate the extreme change between Walk forward and Walk backward. The initial direction is determined randomly, and the corresponding pre-trained policy is executed. Before switching to the opposite direction's 's pre-trained policy, balancing runs for 100 steps.
Patrol:
In Hurdle environment, the agent needs to walk forward and jump over a curb 5 times. The two subtask policies Walking forward and Jumping. Each of the subtask policies a sub-environment: Walking forward is trained to walk 1000 steps on a flat surface, and Jumping learns to jump over one obstacle. As shown in Table 2 our method works very well and close to the perfect success count.
Hurdle:
trained on
Obstacle Course: Obstacle course has types of obstacles, curbs and ceilings. There are two curbs and three ceilings; hence the agent needs to be able to walk, jump, and crawl to get through the obstacles. The order of obstacles is randomly decided before starting. We prepared three subtask policies: Walking forward, Jumping, and Crawling. Each subtask policy was trained on the sub- environment of Obstacle course to complete each task.
5.3 COMPARATIVE EVALUATION
To prove that utilizing pre-trained policies is advantageous when faced with complex problems, we compare our method with the performance of a single end-to-end policy (Single), only using pre- trained policies without transition policies (Without TP), and with transition policies (With TP), and [ee et al. (2019h's method. All subtask policies and the single policy trained with TRPO (Schulman et al.) 2015). Note that we also conducted tested other training algorithms such as PPO (Schulman al. 2017) and SAC (Haarnoja al. 2018), but the results
comparable TRPO. Table I and are the final results, showing that our method of using transition policies and DQNs has higher success rates than using single policy. All arm manipulation tasks achieve close to perfect scores, similar to Lee et al. (2019), and in the locomotion tasks, our method outperforms all baselines.
5.4 VISUALIZING TRANSITION POLICY DISTRIBUTION
The goal of our transition policy is start from the ending states of one pre-trained policy, and then induce a trajectory with the same distribution as another pre-trained policy during the transition interval, which train using IRL. employing the GAN structure. We can see that the success rates significantly increase when this kind of transition policy is used (Tables I and 2), relative to single policy trained end-to-end or pre-trained policies with immediate transitions. These results can be interpreted an indication that our transition policy has succeeded in distribution matching. Figure illustrates t-SNE results for the actions generated by Walking forward-Balancee Balancing and Tossing -itting. Green dots mean the distribution of the transition policy in the corresponding transition interval. They are located between the distributions of the other two colors, an indication that transition policy is able bridge two different pre-trained policies.
8

---

Published as a conference paper at ICLR 2022
Walking forward
Balancing
Transition policy
Tossing
Hitring
Transition policy
Figure 4: Three classes of distributions projected into 2D space with t-SNE. Both figures illustrate the actions of two pre-trained policies and the corresponding transition policy connecting them in transition interval. The left figure shows for Walking forward-BBllaning and the right one stands for Bassing-+litttin The actions from the transition policy lie between those of the pre-trained policies
CONCLUSION
We have presented approach for HRL in which agent utilizes pre-trained policies for simpler tasks to solve a complex task. Transitions between pre-trained policies are essential; hence, pre- trained policies are often fine-tuned to suit complex task (Frans et all 2018). Instead of fine- tuning, we introduce transition policies (Lee et al. 2019) to connect the pre- pre-trained policies. Since transition policy learns the partial distribution of the following pre-trained policy through IRL, we do not need to create complicated reward function for a transition policy. A DQN is used detect states with high transition success as additional assistance. HRL frameworks often have trouble training the structure when rewards are sparse. We alleviate this issue by exposing only exposing DQN with a binary action space to the sparse reward signal. The DQN is less susceptible to the sparse reward issue than the transition policy with continuous control. In our evaluation, we find the proposed method be competitive with previous work in arm manipulation and much stronger in bipedal locomotion.
Our framework assumes that can prepare suitable pre-trained subtask policies for HRL. This assumption is strong requirement because many complex tasks are difficult to decompose into smaller subtasks, and even decomposition possible, the subtask policies may need to depend on the next subgoal, i..., if object needs to be picked up in certain way allow for success in placement. Nevertheless, since our transition policy training procedure is not dependent on explo- ration and sparse rewards, we believe that work methods and open new directions for HRL. Even in the case where subtask policies must be retrained achieve success, the distribution matching approach may be useful, e.g., by mixing the subtask reward with the discriminator's reward.
give meaningful help existing hierarchical
REPRODUCIBILITY STATEMENT
We describe the learning architectures, training procedures, as well as how we select transition in- tervals and hyperparameters in the appendix. We also prepared the pre-trained policies to fa- cilitate follow-up work. The source code is available athttpp:://gthhie com/ shashacks/ IRL Transition
REFERENCES
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning, 2017.
Pierre-Lut Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI Conference Artificial Intelligence, 2017.

---

Published as a conference paper at ICLR 2022
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in Neural Infor- mation Processing Systems, 1992.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, 2016.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. In International Conference on Learning Representations, 2017.
Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. In International Conference on Learning Representations, 2018.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran. Vikash Kumar, and Sergey Levine. Divide-and conquer reinforcement learning. In International Conference on Learning Representations, 2018.
lan J Goodfellow, Jean Pouget-Abadic Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural
Information Processing Systems, 2014.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, 2018
Nicolas Heess, Gregory Wayne, Yuval Tassa, Timothy P. Lillicrap, Martin A. Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. CoRR, abs/1610.051822 2016.
URL fttt:///rxiive org/ abaaaassll1l1 05182
Nicolas Heess, Dhruva TB, Srinivasan Sriram, Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, S. M. Ali Eslami, Martin A. Riedmiller, and David Silver. Emergence of locomotion behaviours in rich environments. CoRR, abs/1707.0222s 2017. URL http:
//axxiv.org/aas////// 0226
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Joshua Tenenbaum. Hierarchi- cal deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Youngwoon Lee, Shao-Huu Bua Sun, Sriram Somasundaram, Edward Hu, and Joseph J. Lim. Compos- ing complex skills by learning transition policies with proximity reward induction. Interna- lional Conference on Learning Representations, 2019.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Inter- national Conference on Learning Representations, 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint 1r12iv 2013.
Ofir Nachum, Shixiang Gu. Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. In Advances in Neural Information Processing Systems, 2018
10

---

Published as a conference paper at ICLR 2022
Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
Martin I. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Martin Riedmiller. Neural fitted Q iteration first experiences with a data efficient neural reinforce- ment learning method. In European Conference on Machine Learning, 2005.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch. In International Conference on Machine Learning, 2018.
Stephane Ross and Drew Bagnell. Efficient reductions for imitation learning. In International Conference on Artificial Intelligence and Statisties, 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.063477 2017. URL http::://axw....rg////o///o/
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 11(1)1111-211, 1999.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
Sebastian Thrun and Anton Schwartz. Issues in using function approximation for reinforcement learning. In Proceedings of the Fourth Connectionist Models Summer School, pp. 255-263. Hills- dale, NJ, 1993.
Brendan Tidd, Nicolas Hudson, Akansel Cosgun, and Jürgen Leitner. Learning when switch: Composing controllers to traverse a sequence of terrain artifacts. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model- control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double Q- learning. In AAAI Conference on Artificial Intelligence, 2016.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine Learning, 83-4))279-292, 1992.
Brian Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference Artificial Intelligence, 2008.
A APPENDIX
To verify our method, we use six environments- three of them are robotic arm manipulation tasks and the other three are for the bipedal locomotion tasks. The arm manipulation tasks are conducted on floor composed of square (0.1 x 0.1 m) tiles. The agent receives an input size of 49 as an observation. An observation consists of the robotic arm's 's information like joints and velocity and the box's state. The bipedal locomotion tasks are conducted floor composed of square (1 m x m) tiles and receive an input size of 26, plus an additional input when obstacles exist. The details in Section B.2.
11

---

Published as a conference paper at ICLR 2022
B PRE-TRAINED POLICY
All policies are trained with PPO and provided by Lee et al. (2019) for fair comparison. The pre- trained policies have a two-layer tanh network with 32 units. For each epoch, we collect a rollout of size 10000 and use the Adam optimizer with mini-batch size of 64 and learning rate 1e-3 to update a policy.
B. ROBOTIC ARM MANIPULATION
There are four pre-trained policies (Picking, Catching, Tossing, and Hitting). Picking is used for the task Repetitive picking up requiring the agent to pick the box five times, and Catching used for the task Repetitive catching requiring the agent to catch the box five times. The task Serve utilizes Tossing and Hitting to serve like tennis.
Picking: The agent for Picking aims to pick the box one time the floor. The box is randomly placed on a 0.1 m x 0.1 m tiles with a center (0.5, 0.2). After picking up the box, the agent needs to hold it 50 steps in the air.
The agent for Catching aims to catch the box one time. The box is thrown from the air near the agent. After catching the box, the agent needs hold 50 steps in the air. The box is located at (0.0, 2.0, 1.5) and thrown for the next catching.
Catching:
Tossing: The agent for Tossing aims toss the box for serve. The box is randomly placed on 0.005 m x 0.005 tiles with a center (0.4, 0.3, 0.005).
Hitting: The agent for Hitting aims hit the box and the box needs go towards the given target. The box is randomly placed at position 0.005 m x 0.00 5m square region with a center (0.4, 0.3, 1.2).
B.2 BIPEDAL LOCOMOTION
There are five pre-trained policies (Walking forward, Walking backward, Balancing, Jumping and Crawling). The task Parrol requires three pre-trained policies: Walking forward, Walking backward, and Balancing to move back and forth. The task Hurdle uses Walking forward and Jumping to pass five curbs. The task Obstacle course utilizes Walking forward, Jumping, and Crawling to pass curbs and ceilings.
The observation of Patrol has an additional observation of size 1. The additional observation for distance between the agent and the stone that needs to be touched. The Obstacle course task has two additional observations: one is for the distance between the agent and the starting point of an obstacle, and the other is for the distance between the agent and the obstacle ending point.
Walking forward: The agent for Walking forward must walk forward for 1000 steps. This agent receives size 26 input an observation.
Walking backward: The agent for Walking backward must walk backward for 1000 steps. This agent receives size 26 input as observation.
Balancing requires the agent to balance in place for 1000 steps. This pre-trained policy is used control extreme changes in Walking forward and Walking backward. This agent receives 26 size imput as an observation.
Balancing:
Jumping: Jumping requires the agent to jump over curb. The curb size is height 0.4 and length 0.2 m and is randomly located between 5.5 m-8.5 m. This agent receives size 28 input as an observation.
Crawling:
Crawling requires the agent to crawl under a ceiling. The ceiling size is height 1.0 m and length 16 and is located on 2.0 This agent receives size 26 input observation.
12

---

Published as a conference paper at ICLR 2022
Table 3: Update numbers for the manipulation tasks with how many interactions required to get the success-fail data.
Repetitive picking up Repetitive catching
Serve
# Update
15000
20000
25000
# Interaction
1700000
3700000
4100000
C TRANSITION POLICY
To get transition polices, we use IRL, specifically the state-action discriminator (Equation (1)) pro- posed in AIRL [8]. The generator (transition policy) has the same network size as the pre-trained policy and is also trained with PPO according to rewards provided by the discriminator. The state- action discriminator De (St, Qg. St+1) contains the reward approximator (s, 0z) and the shaping term (se) that each of the networks has a two-layer ReLU network with 100 units.
We collect the last 10000 states that we define for the preceding pre-trained policy and one million trajectories for the subsequent pre-trained policy. To train a transition policy for our task, the transi- tion policy interacts with the corresponding environment 10 million times, and every 50000 times, the transition policy and the corresponding discriminator are updated. We use the Adam optimizer (Kingma & Ba 2015) with mini-batch size of 64 and learning rate le-4 for the transition policies and the learning rate 3e-4 for the discriminators.
is used for this task. The transition interval is from after holding the box 50 steps to before picking the next box on the floor. We collect Picking's demonstration of starting from the beginning holding the box for 50 steps; hence the transition policy starts from the air and pick up the next box.
Repetitive picking up: One transition policy,
Repetitive catching: One transition policy, Catching >Catching, is used for this task. The transition interval is from after catching the box and wait for 50 step to before catching the next box. We collect Catching's ss demonstration of starting from the beginning until 85 steps.
Serve: One transition policy, Tossing- is used for this task. The transition interval is from the box passes 0.7 m by Tossing to before hitting the box. We collect Hitting's demonstration from after 20 steps to until hitting the box.
Patrol: There exist four transition policies, Walking forwad++Balancingg and Walking back- wards=Balancing. The transition intervals are the same except for the position. The transition intervals for Walking-+Balancinn is from when the agent pass stepping stone to balance for 100 steps. The transition interval for Bollaccing~Walle Walking walks m to the corresponding direction.
is from after balancing 100 steps to until
There exist two transition policies, Walking perward++JJuppinn The transition interval of Walking trwaard-Jummpng is m-4 m from the curb. The transition interval of Jumping- Walking forward is 2.5 m-3.5 from when the agent jumps over the curb. We collect the Walking forward's demonstration from the beginning to 4 m.
Hurdle:
Obstacle course: There exist four transition polices, Walking fowward-- and Walking for- ward--Crawling The transition interval Walking forward-+Juppinn is 2 m-3 m from the curb, and Jumping--Wallee
forward is the same as Hurdle. The transition interval of Walking for- ward-+ mardd+Crawlln Crawling m-3 m in front of a ceiling, and after passing the ceiling. Walking forward Bowaard>Crawling uses the same Walking forward trajectories as Walking
forward is 2 m-3 m
D DEEP Q NETWORK
Our DQNs have two-layer ReLU with 128 units. The Adam optimizer is used with learning rate lc-4 and mini-batte size of 64 to update DQNs. All of the replay buffers of DQNs store one million samples. We simply set r, and ry to -1 for all of the six environments. If DQNs can be trained
13

---

Published as a conference paper at ICLR 2022
Table 4: Update numbers for the bipedal locomotion tasks with how many interactions required to get the success-fail data.
Patrol
Hurdle
Obstacle Course
# Update
30000
20000
30000
# Interaction
30000000
1 11000000
47000000
separately like when training transition policies, the total interaction with an environment can be greatly reduced. However we trained all DQNs together here and the number of interactions and updates can be seen in Table [
14


Reviews:
[{"review_id": "ud6nx3FObGK", "reviewer": "Reviewer_CNo5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- the paper is clearly written (for comments on relatively minor\n  exceptions of this, see below, all easy to fix)\n\n- no unnecessary complexity of the description or the approach: idea\n  and solution appear to do what is necessary for solution but not\n  introduce complexity for complexity sake. More sophisticated\n  solutions may be possible, but any extension or modification of the\n  idea could then be benchmarked against the presented approach.\n\n- I like the idea of training a transition policy for existing tasks\n\nWeaknesses:\n\n- The idea of learning the transition only after the first task is\n  completed is an important limitation of the approach, dependent on\n  the specific tasks. For some task combinations, it will be important\n  to also modify the final steps of the first task, dependent on what the\n  second task is supposed to do (or on how the second task begins).\n\n  As an example, a robot with task A to approach an object, and then\n  manipulate the object with task B might need to approach the object\n  differently dependent on task B, or on the environment prior to\n  beginning B. The transition policies here will only start after\n  completing A, but I would suspect could also lead to backtracking,\n  or oscillating behaviours.\n\n  It would be good to see a discussion on how such issues might be\n  addressed or avoided even if the approach in the current form\n  doesn't do that.\n\n- I was not sure if the experiments are the best to demonstrate or\n  explore strength and weaknesses of the solution. I would be curious\n  about a setup where the difficulty in transitions can be\n  systematically changed. This would allow to also compare the\n  adaptability of different methods, and a more thorough\n  evaluation. Taking the example from above - an environment where the\n  agent has to approach an object and move the object into a specific\n  direction (eg by pushing towards a target state). This task is easy\n  if agent, object, and target are aligned, but more difficult if the\n  agent approaches the object from the side, and hard if the agent\n  approaches the object from the target direction.\n\n\nOther comments:\n\n- Alg. 2: is the transition interval input to alg 2, or how is the\n  condition in line 5 checked?\n- Alg. 2: \"run $q_{a,b}$\" - I assume this one output (action:\n  stay/switch), or is it two (Sec 3.2, first paragraph), and if it's\n  two, why/what is it?\n- Alg. 2: the condition in line 16 is ambiguous - should it run while\n  $\\pi_b$ has failed or not failed?\n- it appears the assumption is the environment is reasonable static so\n  that executing policies a and b can be scheduled, or at least\n  policies implicitly check that they can still be executed at the\n  beginning (there is no explicit option for replanning after $\\pi_{a,b}$ terminates).\n- the final sentence in the reproducibility statement is missing a verb\n", "summary_of_the_paper": "The paper addresses the problem of learning to subsequently execute\ntasks, and transitioning from a first task to a second task. The work\ncontributes a method to train transition policies, and a method to\ndecide when to start executing the second policy, stopping the\ntransition policy. The approach is evaluated on a few simulated robot\nlocomotion/manipulation tasks.\n", "summary_of_the_review": "The paper presents a nice idea for an interesting problem, and a\nstrength of the paper is the level of complxity used to solve and\ndescribe the problem. For complex combinations of tasks (where tasks\ndepend on each other), the approach may not be sophisticated enough as\nthe sole solution to the problem. Experiments and results are OK, but\ncould be more systematic.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z27A42JZeYu", "reviewer": "Reviewer_M3VJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The paper is well-written. I am not an expert in the area of HRL, but could still nicely follow the story of the paper, and the implementation details\n- The problem studied is interesting, and seems challenging. \n- The approach is well-justified\n\nWeaknesses\n- Some of the details in the background section slightly break the story and could be removed/moved to appendix, especially all the details about deep q-learning \n- The number of baselines is quite marginal. I am not expert in the area and hence am unsure if more baselines could be added.\n", "summary_of_the_paper": "The paper tackles the problem of connecting pre-trained policies $\\pi_a, \\pi_b$ in order to solve more complex tasks through abstraction. In order to do so, they leverage inverse RL (leveraging adversarial learning) to train a transition policy $\\pi_{ab}$ aiming at transitioning between the two policies $\\pi_a, \\pi_b$. It is trained by enforcing its state-action occupancy to match that of the next pre-trained policy $\\pi_b$ via inverse RL. They also propose leveraging a DQN with simple reward structure to control the transitioning from the trained transition policy $\\pi_{ab}$ to $\\pi_b$.\n\nContributions:\n - A relatively simple approach to transitioning between pre-trained policies, which has potential for impact when aiming to tackle complex tasks.\n - Empirical demonstration that their approach succeeds at tackling complex tasks by combining pre-trained policies successfully, and outperforming a single policy trained with PPO/SAC\n", "summary_of_the_review": "I am not expert in the area of HRL, and hence I am only trying to provide an educated guess. I am recommending weak acceptance given the problem is interesting and the approach seems to make sense, along with an interesting set of experiments.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LU8P5Xw2e1", "reviewer": "Reviewer_1ZCq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Let me start with the caveat that I am not an expert on RL, so that my knowledge of related work is limited. This said, I found the paper remarkably well-written and easy to follow. I have only a few questions below, some of which may be due to lack of expertise.\n\n* You characterise your technique of policy distribution matching as an instance of Inverse RL, but in the first paragraph of p.2, you say that the technique \"avoid[s] the problem of designing an explicit reward function ...\", which looks to be conflicting with the term \"Inverse RL\". Can you please clarify ? \n\n* In Tables 1 and 2, while the \"Single\" and \"With TP\" rows are reasonably clear from the text, unless I missed it, I could not find a description of \"Without TP\".\n\n* You do not say much on how you determine the success-fail rewards $r_s,r_f$ hyperparameters. Can you clarify?\n\n* At a more general level, a number of important aspects seem to be left to manual decisions, such as the design of the subtasks, their order, or the definition of the transition intervals. Should the non-expert reader assume that the automation of such decisions is beyond the current state of art?", "summary_of_the_paper": "The paper considers the problem of solving a complex task requiring different skills by combining \"subtask\" policies pretrained for each individual skill. One approach here consists in finding ways to transition smoothly between the policy $b$ for a subtask and the policy $b$ for the next subtask. The paper proposes to do that based on training a \"transition policy\" that starts from a state produced by $a$ and attempts to match the _distribution_ of state-action pairs associated with $b$. Additionally, a DQN controls the exact timing at which the transition policy passes the control to $b$, which improves the success rate of the transition. Experiments are conducted on some simulated arm manipulation and bipedal locomotion tasks with similar results to current baselines on the first group and superior results on the second group.\n", "summary_of_the_review": "From the position of a non-expert of the domain, I found the paper well-written and instructive, and was convinced by the idea of matching the transition policy and target policy distributions, but cannot really judge the novelty.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Description of paper content:

The paper proposes a strategy to train a “transition policy” that can connect two pre-trained policies. The transition policy tries to reach state-action pairs that are within the occupancy distribution of the second policy using Inverse RL. The technique was evaluated on robot manipulation and locomotion problems. 

Summary of paper discussion:

The discussion was not lengthy. The reviewers felt the paper was quite well-written, instructive, and novel, yet also implied the experimental results were less systematic than might be desired. All reviewers were weakly supportive of publication and made few critical comments. The salient ones concerned the experimental domains, the number of baselines, and the question of the generality of the approach.
Decision: Accept (Poster)