Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
Authors: Youngwoon Lee
Joseph J Lim
Anima Anandkumar
Yuke Zhu
Venue: CoRL2021 Poster (2021)
Keywords: Long-Horizon Manipulation
Skill Chaining
Reinforcement Learning
Adversarial Imitation Learning

Abstract: Skill chaining is a promising approach for synthesizing complex behaviors by sequentially combining previously learned skills. Yet, a naive composition of skills fails when a policy encounters a starting state never seen during its training. For successful skill chaining, prior approaches attempt to widen the policy's starting state distribution. However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences. In this paper, we propose to chain multiple policies without excessively large initial state distributions by regularizing the terminal state distributions in an adversarial learning framework. We evaluate our approach on two complex long-horizon manipulation tasks of furniture assembly. Our results have shown that our method establishes the first model-free reinforcement learning algorithm to solve these tasks; whereas prior skill chaining approaches fail. The code and videos are available at https://clvrai.com/skill-chaining



Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization
Youngwoon Lee Joseph J. Lim'
Anima Anandkumar Yuke Zhu
University of Southern California 2VVIDI
*California Institute of Technology
"The University of Texas at Austin
Abstract: Skill chaining is promising approach for synthesizing complex behav- iors by sequentially combining previously learnd skills. Yet, a naive composition of skills fails when a policy encounters a starting state never seen during training. For successful skill chaining, prior approaches attempt to widen the policy's start- ing state distribution. However, these approaches require larger state distributions to be covered as more policies are sequenced, and thus are limited to short skill sequences. In this paper, we propose to chain multiple policies without excessively large initial state distributions by regularizing the terminal state distributions in an adversarial lcarning framework. We evaluate our approach on two complex long-horinn mogghhoizoon manipulation tasks of furniture assembly. Our results have shown that our method establishes the first model-free reinforcement learning algorithm solve these tasks; whereas prior skill chaining approaches fail. The code and videos are available at https: / http///cllaaa.ae//e www./ooooool//ooeeooe...e...e.ll wommstill-chaiinngg
Keywords: Long-Horizon Manipulation, Skill Chaining, Reinforcement Learning
1 Introduction
Deep reinforcement learning (RL) presents a promising framework for learning impressive robot behaviors [1-4]. Yet, learning complex long- long-horizon task using single control policy is still challenging mainly due to its high computational costs and the exploration burdens of RL models [5]. A more practical solution is to decompose whole task into smaller chunks of subtasks, learn policy for each subtask, and sequentially execute the subtasks to accomplish the entire task [6-9].
However, naively executing one policy after another would fail when the subtask policy encounters a starting state never seen during training [6, 7, 9J. In other words, a terminal state of one subtask may fall outside of the set of starting states that the next subtask policy can handle, and thus fail to accomplish the subtask, as illustrated in Figure 1a. Especially in robot manipulation, complex interactions between high-DoF robot and multiple objects could lead wide range of robot and object configurations, which are infcasible to be covered by a single policy [10]. Therefore, skill chaining with policies with limited capability is not trivial and requires adapting the policies make them suitable for sequential execution.
To resolve the mismatch between the terminal state distribution (i.e. termination set) of one policy and the initial state distribution (i.e. initiation set) of the next policy, prior skill chaining approaches have attempted to learn to bring an agent to suitable starting state [7], discover chain of options [11, 12), jointly fine-tune policies to accommodate larger initiation sets that encompass terminal states of the preceding policy [6], or utilize modulated skills for smooth transition between skills [13-16, 9]. Although these approaches can widen the initiation sets to smoothly sequence several subtask policies, it quickly becomes infeasible as the larger initiation set often leads to even larger termination set, which is cascaded along the chain of policies, as illustrated in Figure lb.
Instead of enlarging the initiation set to encompass termination set modelled as simple Gaussian distribution [6], we propose to keep the termination set small and near the initiation set of the next policy. This can prevent the termination sets from becoming too large to be covered by the subsequent
Work done during internship at NVIDIA
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

R4-1
144
Increase 1;
B, also increases
(b) Widen the initiation set of m. (Policy Sequencing)
1
x
x++1
4-1
B
4
l
Widen 1, to cover P
Push towards 1/1
Fall subtask
(a) Naive execution of m
(c) Update n. terminal state regularization (Ours)
Figure I: We aim to solve long-horizon task, e.g., furniture assembly, using independently trained subtask policies. (a) Each subtask policy, Te, works successfully only on its initiation set (green), Z, and results in its termination set (pink), B;; thus, it fails when performed outside of I₁ (red eurve). (b) To enable sequencing policies, a subsequent policy Ty needs to widen its initiation set to cover the termination set of the prior policy 3,-1- But this can result in an increase of its termination set 3, which makes fine-tuning of the following policy 7,+1 even more challenging. This effect exacerbated when more policies chained together. (c) During fine-tuning of policy n; regularize the terminal state distribution 3n to be close to the initiation set of the next policy I,+1- In contrast to the boundless increase of 3; in (b), our approach effectively keeps the required initiation small the chain of policies with the terminal state regularization.
policies, especially when executing long sequence of skills; thus, fine-tuning of subtask polices becomes more sample efficient. As a result, small changes in the initiation sets of subsequent policies are sufficient to successfully execute a series of subtask policies.
To this end, we devise an adversarial learning framework that learns an initiation set discriminator to distinguish the initiation set of the policy that follows from terminal states, and uses it as regularization for encouraging the terminal states of a policy to be near the initiation set of the next policy. With this terminal state regularization, the pretrained subtask policies are iteratively fine-tuned to solve the subtask with the new initiation set while keep the terminal states of policy small enough be covered by the initiation set of the subsequent policy. As a result, our model capable of chaining a sequence of closed-loop policies to accomplish collection of multi- IKEA furniture assembly tasks [17] that require high- digh--immnsional continuous control under contact-rich dynamics.
In summary, our contributions are threefold:
We propose a novel adversarial skill chaining framework with Terminal STAte Regularization, T-STAR, for learning long-bon Long-horizon and hicrarchical manipulation tasks.
We demonstrate that the proposed method can learn long-horizon manipulation task. furniture assembly of two different types of furniture. Our terminal state regularization algorithms improves the success rate the policy sequencing method from 0% to 56% for CHAIR .INGOF and from 59% to 87% for TABLE_LACK To our best knowledge, this is the first empirical result of model-free RL method that solves these furniture assembly tasks without manual engineering.
We present comprehensive comparisons with prior skill composition methods and qualitative visualizations to analyze our model performances.
2
Related Work
Deep reinforcement learning (RL) for continuous control [18-20] is an active research area. While some complex tasks can be solved based on a reward function, undesired behaviors often emerge [21] when tasks require several different primitive skills. Moreover, learning such complex tasks becomes computationally impractical as the tasks become long and complicated due to the credit assignment problem and the large exploration space.

---

Imitation learning aims to reduce this complexity of exploration and the difficulty of learning from reward signal. Behavioral cloning approaches [22-25] greedily imitate the expert policy and therefore suffer from accumulated errors, causing drift away from states seen in the demonstrations. On the other hand, inverse reinforcement learning [26-28] and adversarial imitation learning approaches [29, 30] encourage the agent to imitate expert trajectories with learned reward function, which can better handle the compounding errors. Specifically, generative adversarial imitation learning (GAIL) [29] and its variants [30, 31] show improved demonstration efficiency by training discriminator to distinguish expert versus agent transitions and using the discriminator output as a reward for policy training. Although these imitation learning methods can learn simple locomotion behaviors [29] and a handful of short-horizon manipulation tasks [32, 31], these methods easily overfit to local optima and still suffer from temporal credit assignment and accumulated enrors for long-horizer tasks.
Instead of learning an entire task using a single policy, we can tackle the task by decomposing into easier and reusable subtasks. Hierarchical reinforcement learning does this by decomposing task into a sequence of temporally extended macro actions. It often consists one high-level policy and set of low-level policies, such as in the options framework [33], in which the high-level policy decides which low-level policy to activate and the chosen low-level policy generates a sequence of atomic actions until the high-level policy switches it to another low-level policy. Options can be discovered without supervision [34-37, 11, 12], learned from data [38-40], meta-learned [41], pre-defined [7, 42 44], attained through task structure supervision [45, 10]. For long-horizon planning capability, the high-level policy can be trained using traditional planning methods [46, 47].
To synthesize complex motor skills with a set of predefined skills, Lee et al. [7] learns to find smooth transitions between subtask policies. This method assumes the subtask policies are fixed, which makes learning such transitions feasible but, at the same time, leads to failure of smooth transition when an external state has to be changed or the end state of the prior subtask policy is too far away from the initiation set of the following subtask policy. On the other hand, Clegg et al. [6] proposes to sequentially improve subtask policies to cover the terminal states of previous subtask policies. While this method fails when the termination set of the prior policy becomes extremely large, our method prevents such boundless expansion of the termination set.
Closely related to our work, prior skill chaining methods [11, 12, 38] have proposed to discover a new option that ends with an initiation set of the previous option. With newly discovered options, the agent can reach the goal from more initial states. However, these methods have a similar issue with Clegg et al. [6] that discovering new option requires large enough initiation set of the previous option, which is cascaded along the chain of options.
Furniture assembly is a challenging robotics task requiring reliable 3D perception, high-level planning, and sophisticated control. Prior works [48-50] tackle this problem by manually programming the high-level plan and learning only a subset of lovelevel controls. In this paper, we propose tackle the furniture assembly task in simulation [17] by learning all the low-level control skills and then effectively sequeneing these skills.
3 Approach
In this paper, we aim address the problem of chaining multiple policies for long-horizon complex manipulation tasks, especially in furniture assembly [17]. Sequentially executing skills often fails when a policy encounters starting state (i.e. terminal state of the preceding policy) never seen during its training. The subsequent policy can learn to address these new states, but this may require even larger states to be covered by the following policies. To chain multiple skills without requiring boundlessly large initiation sets of subtask policies, we introduce novel adversarial skill chaining framework that constrains the terminal state distribution, as illustrated in Figure 2. Our approach first learns each subtask using subtask rewards and demonstrations (Section 3.2); it then adjusts the subtask policies to effectively chain them to complete the whole task via the terminal state regularization (Section 3.3).
3.1 Preliminaries
We formulate our learning problem as Markov Decision Process [51] defined through a tuple (S, A, R, P.p.o.)) 7) for the state space S, action space A, reward function R(s, a, s'), transition
3

---

Subtask policy
Subbtakk policy
= RyReNy + AzR + RgRR
Subtask policy 3
Terminal State Regularization
Initiation Set Discriminator 2
Initiation Set Discriminator 3
Terminal State
Terminal State Buffer
Initial State Buffer 3
Initial State Buffer 2
Terminal State Buffer
Sample Initial State
Subtask rollout
Subtask 2 rollout
Subtask 3 rollout
Our adversarial skill chaining framework regularizes the terminal state distribution to be close the initiation set of the subsequent subtask. The initiation set discriminator models the initiation set distribution by discerning the initiation set and states in agent trajectories, while the policy learns reach states close the initiation set by augmenting the reward with the discriminator output, dubbed terminal state regularization. Our method jointly trains all policies and initiation set discriminators, pushing the termination close the initiation set of the next policy, which leads smaller changes required for the policies that follow, especially effective in long chain of skills.
Figure 2:
distribution P(s'|6 [ss' a), initial state distribution Po. and discount factor y. We define policy S A that maps from states actions and correspondingly moves an agent new state according to the transition probabilities. The policy is trained to maximize the expected sum of discounted
ΣnD y'R(st, ae, s(+1) where T is the episode horizon. Each policy comes with initiation set S and termination set S, where the initiation set I contains all initial states that lead to successful execution of the policy and the termination set 3 consists of all final states of successful executions. We assume the environment provides success indicator of each subtask and this can be easily inferred from the final state, e.g., two parts are aligned and the connect action activated. In addition to the reward function, we assume the learner receives fixed set of expert demonstrations, D {rf..... TN J. where a demonstration is sequence of state-action -1, ST,)
rewards [ss,a>~<
pairs, TJ
(So, Op Op.....
3.2 Learning Subtask Policies
To solve complicated long- morioooi task (e.g. assembling a table), we decompose the task into smaller subtasks (e.g. assembling table leg to a table top), learn policy л. for each subtask M,, and chain these subtask policies. Learning each subtask solely from reward signals still challenging in robot manipulation due to a huge state space of the robot and objects explore, and complex robot control and dynamics. Thus, for efficient exploration, we use an adversarial imitation learning approach, GAIL [29], which encourages the agent stay near the expert trajectories using learned reward by discerning expert and agent behaviors. Together with reinforcement learning, the policy can efficiently learn to solve the subtask even on states covered by the demonstrations.
More specifically, we train each subtask policy zे on R. following the reward formulation proposed in [52, 53], which uses the weighted sum of the environment and GAIL rewards:
R' (st,, ae St-1i Φ) R (st, a,, s(41) A A2 ac Ø), where RiAHL (St, (1,,,430 Φ) - 0.25 - tor f [53], and \ and X2 are hyperparameters that balance between the reinforcement learning and imitation learning objectives. We found this reward formulation [53] works most stable among vari- ants of GAII. [29] in our experiments thanks to its bounded reward between [0, 1]. The discriminator
(1)
1]2 is the predicted reward by the GAIL discrimina-
trained using the following objective: miny: EO (1](s) - 1)2² Essy~nt [U)(s) + 1)2

---

Algorithm T-STAR: Skill chaining via terminal state regularization
Require:
Expert demonstrations Di..... Di- subtask MDPs Mi,... Mr
f 1: Initialize subtask policies To.-..,mz =00 GAIL discriminators f3. -. initiation set discrimi-
nators D wi...y ................................ D initial state buffers B B and terminal state buffers _....., , 8
2: for each subtask = 1,... K do
3: while until convergence of z do
4: Rollout trajectories T = (So. ao, ro...., ST) with m3
5: Update f with T and
D
D. Train GAIL discriminator
6: Update Ty using Rt (st, at, s141i o) in Equation (1)
Train subtask policy
7: end while
8: end for
9: for iteration 0,,,.... M do
10: for each subtask i = I,.., K do
11: Sample so from environment or 33
12:
Rollout trajectories T = (so, au, , 8T) with T2 if T is successful then
13:
14:
B3 < B3/ U Su, B3 + B3G Usr & Collect initial and terminal states of successful trajectories end if
15:
16:
Update f' with T and T" D
Fine-tune GAIL discriminator
17:
Update D with 83 ~ B/3 and sI ~ B
P Train initiation set discriminator
18:
Update Tá using It' (st, de, S141i ii w) in Equation (3) Fine-tune subtask policy with terminal state regularization
19:
end for
20:
end for
Due to computational limitations, training a subtask policy for all possible initial states is impractical, and hence it can cause failure on states unseen during training. Instead of indefinitely increasing the initiation set, we first train the policy on a limited set of initial states (e.g. predefined initial states with small noise), and later fine-tune the policy on the set of initial states required for skill chaining as described in the following section. This pretraining of subtask policies ensures the quality of the pretrained skills and makes the fine-tuning stage of our method easy and efficient.
3.3 Skill Chaining with Terminal State Regularization
Once subtask polices are acquired, one can sequentially execute the subtask policies to complete complex tasks. However, naively executing the polices one-by-one would fail since the policies are not trained to be smoothly connected. As can be seen in Figure 1a, independently trained subtask policies only work limited range of initial states. Therefore, the execution of a policy म, fails on the terminal states of the preceding policy 3:-1 outside of its initiation set I.
For successful sequential execution of x;-1 and Tú, the terminal states of the preceding policy should be included the initiation set of the subsequent policy, 8 C. Z, This can be achieved either by widening the initiation set of the subsequent policy by shifting the terminal state distribution of the preceding policy. However, robot manipulation, the set of valid terminal states can be huge with freely located objects (e.g. robot mess up the workplace by moving throwing other parts away). This issue is cascaded along the chain of policies, which leads to the boundlessly large initiation set required for policies, illustrated in Figure lb.
Therefore, a policy needs to not only increase the initiation set (e.g. assemble the table leg in diverse configurations) but also regularize the termination set to be bounded and close to the initiation set of the subsequent policies (e.g. keep the workplace organized), as described in Figure Ic. To this end, devise adversarial framework which jointly trains initiation set discriminator, D (st). to distinguish the terminal states of the preceding policy and the initiation set of the corresponding policy, and a policy to reach the initiation set of the subsequent policy with the guidance of the initiation set discriminator. We train the initiation set discriminator for each policy to minimize the
following objective: (((w) Eer~I, [Dt 1] + E Eer [
5

---

With the initiation set discriminator, we regularize the terminal state distribution of the policy by encouraging the policy reach a terminal state close to the initiation set of the following policy. The terminal state regularization can be formulated as following:
Rn L D5
Then, we can rewrite the reward function with the terminal state regularization:
R' (st, at, St+1: w)) x1 R' (st, ag, $1+1) + A2 ae: Φ) X A
(3)
where A3 is a weighting factor for the terminal state regularization. The first two terms of this reward function guide policy to accomplish the subtask while the terminal state regularization term forces the termination set be closer to the initiation set of the following policy.
With this reward function incorporating the terminal state regularization, subtask policies and GAIL discriminators can be trained cover unseen initial states while keeping the termination set closer to the initiation set of the next policy. Once subtask policies are updated, we collect terminal states and initiation sets with the updated policies, and train the initiation set discriminators. We alternate these procedures to smoothly chain subtask policies, as summarized in Algorithm I, where changes of our algorithm with respect Clegg et al. [6] are marked in red.
4 Experiments
In this paper, we propose a skill chaining approach with the terminal state regularization, which encourages match the terminal state distribution of the prior skill with suitable starting states of the following skill. Through our experiments, we aim to verify our hypothesis that the policy sequencing fails due to unbounded terminal states, which eascaded along the sequence of skills, and show the effectiveness of our framework on learning long sequence of complex manipulation skills.
4.1 Baselines
We compare our method to the state-of-the-art prior works in reinforcement learning, imitation learning, and skill composition, which are listed below:
BC [22] fits a policy to the demonstration actions with supervised learning.
PPO [54] is model-free on-policy RL method that learns a policy from the environment reward.
GAIL [29] is an adversarial imitation learning approach with a discriminator trained to distinguish expert and agent state-action pairs (s, a).
GAIL+ PPO uses both the environment and GAIL reward (Equation (1)) to optimize a policy.
SPiRI [55] is a hierarchical RL approach that learns fixed-length skills and skill prior from the dataset, and then learns a downstream task using skill-prior- rrouiirrra-ee RI.
Policy Sequencing [6] first trains policy for each subtask independently and finetunes the policies to cover the terminal states of the previous policy.
T-STAR (Ours) learns subtask policies simultaneously. leading them to be smoothly connected using the terminal state regularization.
4.2 Tasks
We test our method and baselines with two furniture models, TABLE LACK and CHIAR INGOLF, from the IKEA furniture assembly environment [17] as illus- trated in Figure 3:
TABLE LACK: Four table legs need to be picked up and aligned to the corners of the table top.
TABLF_LACK
(b)
CHAIR INGOLF: Two chair supports and front legs need to be attached to the chair seat. Then, the chair seat needs to be attached to the chair back while avoiding collision to each other.
Two furniture assembly tasks [17] consist of four subtasks (four table legs for TABLE.LACK. two seat supports, chair seat, and front legs for CHIARIINOOLP
Figure 3:

---

In our experiments, we define a subtask as assembling one part to another; thus, we have four subtasks for each task. Subtasks are independently trained on the initial (object) states sampled from the environment with random noise ranging from [-2cm, 2cm] and [-3", 3"] in the (ar, y)plane. This subtask decomposition is given, Le., the environment can be initialized for each subtask and the agent is informed whether the subtask is completed and successful.
For the robotic agent, we use the 7-DoF Rethink Sawyer robot operated via joint velocity control. For imitation learning, we collected 200 demonstrations for each furniture part assembly with a programmatic assembly policy. Each demonstration for single-part assembly consists of around 200-900 steps long due to the long-horizon nature of the task.
The observation space includes robot observations (29 dim), object observations (35 dim), and task phase information (8 dim). The object observations contain the positions (3 dim) and quaternions (4 dim) of all five furniture pieces in the scene. Once two parts are attached, the corresponding subtask is completed and the robot arm moves back to its initial pose in the center of the workplace. Our approach can in theory solve the task without resetting the robot; however, this resetting behavior effectively reduces the gap in robot states when switching skills and is available in most robots.
4.3 Results
The results in Table show the aver- age progress of the furniture assembly tasks across 200 testing episodes for 5 different seeds. Since each task con- sists of assembling furniture parts four times, completing one furniture part assembly amounts to task progress of 0.25. Even though the environment has small noises in furniture initial- ization, both BC [22] and GAIL [29] baselines move the robot arm near furniture pieces but struggle at pick- ing up even one furniture piece. This shows the limitation of BC and GAIL in dealing with compounding errors in long-horize tasks with the large state space and continuous action space. Similarly, the hierarchical skill-based learning approach, SPiRL, also strug- gles at learning picking up single furniture piece. This can be due to the insufficient amount of data to cover a long sequence of skills the large state space, e.g., many freely located objects. Morcover, due to the difficulty of exploration, the model-free RL baseline, PPO, rarely learns to assemble one part. On the other hand, the GAIL + PPO baseline can consistently learn one-part assembly, but cannot learn to assemble further parts due to the exploration challenge and temporal credit assignment problem. These baselines are trained for 200M environment steps (5M for off policy SPiRL).
Table I: Average progress of the furniture assembly task. Each subtask amounts to 0.25 progress. Hence, 1 represents successful execution of all four subtasks while 0 means the agent does not achieve any subtask. Our method learns to complete all four subtasks in sequence and outperforms the policy sequencing baseline and standard RI. and II. methods. We report the mean and standard deviation across 5 seeds.
CHAIR TACKE_LACK
BC [22]
0.03 + 0.00
0.04 + 0.01
PPO [54]
0.09 = 0.11
0.14 0.03
GAIL [29]
0.00 = 0.00
0.00 = 0.00
GAIL+PPO [52]
0.21 0.11
0.22 0.08
SPiRL [55]
0.05 + 0.00
0.03 + 0.00
Policy Sequencing [6]
0,63 = 0.28
0.77 = 0.12
T-STAR (Ours)
0.90 = 0.07
0.89 = 0.04
By utilizing pretrained subtask policies with GAIL+ PPO (25M steps for each subtask), skill chaining approaches could achieve improved performance compared to the single-policy baselines. We train the policy sequencing baseline and our method for additional 100M steps, which requires in total 200M steps including the pretraining stage. The policy sequencing baseline [6] achieves 0.63 and 0.77 average task progress, whereas our method achieves 0.90 and 0.89 average task progress on TABLE.LACK and CHNLR_ respectively. The performance gain of our method comes from the reduced discrepancy between the termination set and the initiation set of the next subtask thanks to the terminal state regularization.
We can observe this performance gain even clearer in the success rates. Our terminal state regulariza- tion improves the success rate of policy sequencing from 0% to 56% for CHAIR CHAIR_INGOF and from 59% to 87% for LACKE We observe that with more skills to be chained, the success rate the newly chained skill decreases, especially in CHATR INGOLF; the policy sequencing baseline learns to complete the first three subtasks, but fails to learn the last subtask due to excessively large and shifted initial state distribution, i.c., terminal states of the preceding subtask.
7

---

Termina inaes roum, 304 stuus Termand cotesiss, IPS, sear dtrpii
Terminal stste (Dura, 2014 stpa)
Temihaal statarruunn 42% stand Terraw dastesgs 42M stroet
Termina instes rouns 4GM stspal
Termonal stotes IPS, 192 steal)
Terrina crates IPS, asir stepi)
(a) 36M steps
(b) 39M steps
(c) 42M steps
(d) 45M stops
Figure 4: To demonstrate the benefit of our terminal state regularization, visualize the changes in termination sets over training of the third subtask policy on CHAIR_INGOLI We plot each terminal state by projecting its object configuration into 2D space using PCA. Through 36M 45M training steps, both the policy sequencing baseline [6] and our method successfully learn to cover most terminal states from the second subtask. However, without regularization, the policy sequencing method (red) shows the increasing size of the termination set (e.g. spread over horizontally at 39M and vertically 42M steps) as more initial states are covered by the policy. In contrast, in our approach (blue), the terminal state distribution is bounded, which shows that the terminal state regularization can effectively prevent the terminal state distribution diverging. This bounded termination set makes learning of the following skills efficient, and thus helps chaining a long sequence of skills.
4.4 Qualitative Results
To analyze the effect of the proposed terminal state regularization, we visualize the changes in termination sets over training. We first collect 300 terminal states of the third subtask policy on CHAIR JINGLL both for method and the policy sequencing baseline for 36M, 39M, 42M, and 45M training steps. Then, we apply PCA on the object state information in the terminal states and use the first two principal components reduce the data dimension.
Figure 4 shows that our method effectively constrains the terminal state distribution of a subtask policy. Before 36M training steps, the policy cannot solve the third subtask due the shifted initial state distribution by the second subtask policy. With additional training, at 42M and 45M training steps, the policy learns to solve the subtask on newly added initial states for both methods. From 36M to 45M steps of training, the termination set of the policy sequencing baseline (red) spreads out horizontally after 39M training steps and vertically after 42M steps. For successful skill chaining, this wide termination set has to be covered by the following policy, requiring a large change in the policy and potentially causing an even larger termination set. In contrast, the terminal state distribution of our method (blue) does not excessively increase but actually shrinks in this experiment, which leads successful execution and efficient adaptation of subsequent subtask policies.
5 Conclusion
We propose T-STAR, a novel adversarial skill chaining framework that addresses the problem of the increasing size of initiation sets required for executing long chain of manipulation skills. To prevent excessively large initiation sets to be learned, we regularize the terminal state distribution of a subtask policy to be close to the initiation set of the following subtask policy. Through terminal state regularization, our approach jointly trains all subtask policies to ensure that the final state of one policy is a good initial state for the policy that follows. We demonstrate the effectiveness of our approach the challenging furniture assembly tasks, where prior skill chaining approaches fail. These results are promising and motivate future work on chaining more skills with diverse skill combinations tackle complex long-horizon problems. Another interesting research direction is eliminating subtask supervision required in our work and discovering subtask decomposition from large data in a unsupervised learning manner. Finally, transferring our framework to real robot systems, which involves learning a vision-based policy, improving sample efficiency, and closing conulation-to-red gaps, is our definite future work.
8

---

Acknowledgments
This work was initiated when Youngwoon Lee worked at NVIDIA Research intern. This research is also supported by the Annenberg Fellowship from USC and the Google Cloud Research Credits program with the award GCP19980904. We would like to thank Byron Boots for initial discussion, Jim Fan, De-An Huang, Christopher B. Choy, and NVIDIA AI Algorithms team for their insightful feedback, and the USC CLVR lab members for constructive feedback.
References
[1] S. Levine, C. Finn, T. Darrell, and P. Abbecl. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334- 137113333-33,,,,,,- 2016.
[2] F. Suárez-Ruiz and Q--C. Pham. A framework for fine robotic assembly. In IEEE International Conference on Robotics and Automation, pages 421 426,, 2016.
[3] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
[4] D. Jain, A. Li, S. Singhal, A. Rajeswaran, V. Kumar, and Todorov, Learning deep visuomotor policies for dexterous hand manipulation. In IEEE International Conference on Robotics and Automation, pages 3636-3643. IEEE, 2019.
[5] M. Andrychowiez, A. Raichuk, P. Stanezyk, M. Orsini, S. Girgin, R. Marinier, I. Hussenot, M. Geist, O. Pietquin, M. Michalski, S. Gelly, and O. Bachem. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021. URL. attps://openreviee
[6] A. Clegg, W. Yu, J. Tan, C. K. Liu, and G. Turk. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions Graphics (TOG), 376))11-10, 2018.
[7] Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2019. URL https:// openreview net/
X. B. Peng. M. Chang, G. Zhang, Abbeel, and S. Levine. Mep: control with multiplicative compositional policies. Neural Information Prucessing Systems, 2019.
Learning composable hierarchical
[9] Y. Lee, J. Yang. and J. J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference Learning Representations, 2020.
[10] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine. Divide-and-comquer reinforcement learning. In International Conference on Learning Representations, 2018. URL https:://penreeiieg net/ forum?id=rJwelMtbr
[11] G. Konidaris and A. Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. In Advances in Neural Information Processing Systems, volume 22, pages 1015-1023, 2009.
[12] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. International Conference on Learning Representations, 2020.
[13] P. Pastor, H. Hoffmann, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In IEEE International Conference Robotics and Automation, pages 763-766 2009.
[14] Kober, Mülling, O. Krömer. H. Lampert, B. Schölkopf, and Peters. Movement templates for learning of hitting and batting. In IEEE International Conference on Robotics and Automation, pages 853-858, 2010.
[15] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):263 279, 2013.
[16] K. Hausman, J. T. Springenberg. Z. Wang. N. Heess, and M. Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations. 2018. URL https: /ttpsr///ppnrrevio wwwp:arrwwwwoooeoooouuunuuaaaaaa net//oorumiiddrr000ZXX

---

[17] Y. Lee, E. S. Hu, and J. J. Lim. IKEA furniture assembly environment for long-horizon complex manipulation tasks. In IEEE International Conference Robotics and Automation, 2021. URL https: //clvrai. www.coo//uuuoe htt//fuunntrreeee
[18] T. P. Lillicrap. J. Hunt, A. Pritzel, N. Heess. Erez, Tassa, D. Silver, D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint (ttp:.5090009990 2015.
[19] Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.
[20] T. Haarnoja, A. 7hou, P. Abheel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1856-1865, 2018.
[21] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Van de Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing-solvingg sparse reward tasks from scratch, arXiv preprint ArXiv:1802 10567, 2018.
[22] D. A. Pomerleau. Alvinm: An autonomous land vehicle neural network. Advances in Neural Information Processing Systems, pages 305-313, 1989.
[23] S. Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, pages 1040-104 1046, 1997.
[24] C. Finn. X. Y. Tan, Y. Duan, T. Darrell. S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation, pages 512-519. IEFE, 2016.
[25] D. Pathak", P. Mahmoudich", M. Luo*, P. Agrawal*, D. Chen, F. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In International Conference Learning Representations, 2018.
[26] A. Ng and S. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, page 2, 2000.
[27] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
[28] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement Icarning. In Association for the Advancement of Artificial Intelligence, volume 8, pages 1433-1438, 2008
[29] J. Ho and S. Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pages 4565-4573, 2016.
[30] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations, 2018. URL httpss//oppnxeriee.nnff//oee id=rkHyul-A-
[31] Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor--ro Addressing sample inefficiency and reward hias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
[32] K. Zolna, S. Reed, A. Novikov, S. G. Colmenarej. D. Budden, S. Cabi, M. Denil, N. de Freitas, and Z. Wang. Task-relevant adversarial imitation learning. In Conference on Robot Learning, 2020.
[33] R. S. Sutton, D. Precup. and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial intelligence. 112(1-2)181-22 1999.
[34] J. Schmidhuber. Towards compositional learning with dynamic neural networks. Inst. für Informatik, 1990.
[35] P.-... Bacon, Harb, and D. Precup. The option-critic architecture. In Association for the Advancement Artificial Intelligence, pages 1726-1734, 2017.
[36] O. Nachum, S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neurul Information Processing Systems, pages 3303-3313, 2018.
[37] A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight. In International Conference on Learning Representations, 2019, URL https: //ppereveeee nwt ner/forum? id=ryzECoAcY7.
10

---

[38] G. Konidaris. S. Kuindersma. R. Grupen, and A. Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 3 31(3:3600-375, 2012.
[39] T. Kipf, Li, H. Dai, V. Zambaldi, A. Sanchez-Gionzaler, E. Grefenstette, P. Kohli, and P. Battaglia. Compile: Compositional imitation learning and execution. In International Conference Machine Learning, pages 3418-3428. PMLR. 2019.
[40] Y. Lu, Y. Shen, S. Zhou, Courville, J. B. Tenenbaum, and C. Gan. Learning task decomposition with ordered memory policy network. In International Conference on Learning Representations, 2021.
[41] K. Frans, J. Ho, X. Chen, P. Abbeel. and J. Schulman. Meta learning shared hierarchies. In Interna- tiomal Conference on Learming Representations, 2018. URL pestpp::/oppnree www.eeeeoooooniiieeeeeee.e.. SyXOIeWAW.
[42] T. D. Kulkarni, Narasimhan, A. Saeedi, and J. Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Neural Information Processing Systems, pages
2016.
[43] J. Oh, S. Singh, H. Lee, and P. Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Learning Representations, pages 2661-26777 2017.
[44] J. Merel, A. Ahuja, V. Pham, S. Tunyasuvunakool, S. Liu, D. Tirumala, N. Heess, and G. Wayne. Hierarchical visuomotor control of humanoids. In International Conference on Learning Representations. 2019. URL nattps//opeereviiie
[45] J. Andreas, D. Klein, and S. Levine Modular multitask reinforcement learning with policy sketches. In International Conference Machine Learning, pages 166 175, 2017.
[46] K. Kase, C. Paxton, H. Mazhar. T. Ogata, and D. Fox. Transferable task execution from pixels through deep planning domain learning. In IEEE International Conference on Robotics and Automation, pages 10459 10465, 2020.
[47] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Leaniing geometric reasoning and control for long- horizon tasks from visual input. In IEEE International Conference on Robotics and Automation, 2021.
[48] S. Nickum, S. Chitta, A. G. Barto, B. Marthi, and S. Osentoski. Incremental semantically grounded learning from demonstration. In Robotics: Science and Systems, 2013.
[49] R. A. Knepper. T. Layton, J. Romanishin, and D. Rus. Ikeabot: An autonomous multi-robot coordinated furniture assembly system. JEEE International Conference on Robotics and Automation, pages 855-862, 2013.
[50] F. Suárez-Ruiz Ruizz X. Zhou, and Q.-. Pham. Can robots assemble an ikea chair? Science Robotics, 3(17): eaat6385, 2018.
[51] R. S. Sutton. Temporal credit assignment reinforcement learning. PhD thesis, University of Mas- sachusetts Amherst, 1984.
[52] Y. Zhu, Z. Wang, Merel, A. Rusu, Erez, S. Cabi, S. Tunyasuvunkkool, Kramár, Hadsell, N. de Freitas, and N. Heess. Reinforcement and imitation learning for diverse visuomotor skills. In Robotics:
Science and Systems, 2018
[53] X. B. Peng. Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. arXiv preprint arXiv:2104.02180 2021.
[54] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint (0XXvv1707.06333 2017.
[55] K. Pertsch, Y. Lee, and J. J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning, 2020.
11


Reviews:
[{"review_id": "UflGjcEuETw", "reviewer": "Reviewer_HKpX", "summary": "The paper proposes a novel learning objective for skill learning in the context of skill chaining. \nExisting approaches to executing a sequence of skills fail when a subtask skill terminates outside a subsequent skill's initiation set. \nTo remedy this, existing approaches attempt to iteratively widen initiation sets to account for broad termination sets, but as such are limited to short skill sequences due to compounding error. \nThe paper proposes an adversarial learning objective for the termination condition such that the size of the termination set is regularized.  \nA discriminator is proposed which is trained to distinguish states from the termination set of one skill and the initiation set of the subsequent skill. \nThe discriminator is used as an additional term in the reward function to encourage subtask policies to reach the subsequent initiation set. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The adversarial formulation of learning the initiation set is clever. \n- The main evaluation against existing methods is thorough, though the main table of results needs to be run across multiple seeds. \n\tEspecially since the run reported in the main paper was (apparently) relatively lucky on the first task. \n\tMoreover, the improvement is relatively marginal. \n\n- Relevant previous work is missing; esp. the original skill chaining paper. \n- The theory is given as it pertains to skill chains, though skill trees also warrant consideration. Does the theory handle this case? \n\n- Supplementary experimentation is lacking. \n\tIt would be worth adding the initiation set to the visualization shown in Figure 4. \n\tThis would provide compelling justification to the hypothesis presented regarding the mismatch between termination and subsequent initiation. \n\tI am also curious as to how something simple like (approximate) L2 regularization would do as a baseline.  \n\n- Lacks hardware results; understandable given COVID. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'm recommending weak rejection. The idea has merit but isn't entirely original; the evaluation is a bit lacking. In particular, the main result should have been presented across multiple seeds, as in the appendix. Moreover, additional experimentation is warranted in the form of ablation to better understand the behavior of the proposed method, and the overview of existing work is lacking. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- Please consider citing the original skill chaining paper from Konidaris and Barto. Termination and subsequent initiation sets are designed to be identical in this work. \n- Are the subtask rewards distinct from the environment reward? If so, consider denoting subtask reward with R_{ENV}^i. \n- Line 18 in the algorithm may have a typo? The loss in Equation (2) takes samples from the i-th initiation set and the (i-1)-th termination set, whereas it is written here with termination samples and initiation samples from the same skill execution.  "}}, {"review_id": "HbFqS509mvk", "reviewer": "Reviewer_8Fh5", "summary": "The paper proposes a solution to long-horizon manipulation problem through chaining skills. The main contribution of the paper is an adversarial learning pipeline to bring the terminal states of one subtask close to the initial state of the upcoming one. The authors show that the proposed framework can significantly outperform IL, RL, IL+RL baselines, as well as an existing algorithm for skill chaining on two simulated IKEA furniture assemble tasks. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The presentation of the paper is very clear. The authors also provide well-made Figures to facilitate the understanding of the proposed solution, in particular, Figures 1 and 2. \n- The research problem is well-motivated. The authors not only accurately point out the limitation of existing approaches, but also validate these intuitions in the experiments (Figure 4). \n- The proposed solution of using adversarial learning is clean and neat. The algorithm proposed by the authors also seem very practical. \n- The authors compare the proposed algorithms with various baselines including IL, RL, IL+RL, and anther skill-chaining approach. The authors also provide explanations on when and how the baseline fails. \n\nWeaknesses:\n- The authors did a thorough literature survey in the field of RL, IL, hierarchical RL, and skill chaining. However, it would be great if the authors can include existing work on learning for long-horizon robotic tasks, which is also relevant to the paper. A few examples here:\n  - Driess et al. Learning Geometric Reasoning and Control for Long-Horizon Tasks from Visual Input. ICRA 2021\n  - Kase et al. Transferable task execution from pixels through deep planning domain learning. ICRA 2020\n- It would be great if the authors can expand the technical section (Section 3) a little to provide more technical details. In particular, it would be great if the authors can write out the loss function for $f_\\phi^I$ rather than referring readers to [46]. I personally think it is important to also point out that the output range of $f_\\phi^I$ is actually $[-1, 1]$ (rather than $[0,1]$ in more common formulations, including what the authors use in Eq. (2)). It actually took me a while to understand why there is a $0.25$ in for $R_{GAIL}$. \n- Related to my previous point, it is interesting that the authors choose to use different adversarial formulations for the policy (IL) and the terminal states (TSR). The ranges of the discriminators are different ($[-1,1]$ for IL and $[0,1]$ for TSR); How the rewards are related to the discriminator is also different (quadratic reshaping for IL, while linear reshaping for TSR, both of them are different from the original formulation of $\\log(\\mathcal{D})$ in the original GAN/GAIL papers). Although the authors mentioned in the appendix that they observe the quadratic reward [46] works better. It would be great if the authors can provide more thorough discussion on these design choices and/or theoretical comparison of them. \n- In the algorithm, the authors choose to pre-train the subtask policies (line 3-7) with a fixed set of initial states before the adversarial training. How does this pre-training stage help? It would be great if the authors can provide some intuitions and/or experimental results.\n- Also in the algorithm, line 17-18, why is $\\mathcal{D}_\\omega^I$ updated after the policy $\\pi_\\theta^I$?\n- Line 220, \"Subtasks are independently trained on the initial states sampled from the environment with random noise ranging from [−2cm, 2cm] and [−3°, 3°] in the (x, y)-plane.\" Are the \"states\" here referred to the states of the furnitures? Why is the perturbation only on the (x,y)-plane? \n- The fact that the authors reset the robot arm \"back to its initial pose in the center of the workplace\" after each subtask seems to shrink the set of terminal states quite a bit. Would the proposed method also generalize to when we don't reset the state of the robot after each subtask completion?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper solves a well-motivated problem for skill-chaining for long-horizon robot manipulation tasks. The authors accurately point out the limitation of the existing approaches, and also empirically validate these limitations in the experiments. The proposed solution seem technically sound and practical. The experiment validation is thorough. Though it would be great if the authors can justify some decision choices (see \"Strengths and Weaknesses\" part). ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See the \"Strengths and Weaknesses\" part. ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "BtjE0zJQK-", "reviewer": "Reviewer_bN8e", "summary": "The paper assumes that a segmentation for a long horizon task is given with individual demonstrations and reward function for each subskill, and aims to learn corresponding policies that, when executed sequentially, succeed in learning the overall task. To make sequential execution successful, the paper proposes to store for each subtask the initial and terminal states for each subskill, if it succeeded. A discriminator is then trained to classify between terminal states of the preceding subskill and successful initial states of the current subskill. The corresponding probability is provided as additional reward (on top of an RL and IL reward term) of the terminal state of the preceding subskill. The policies should thus learn to terminate in states where the subsequent skill succeeds. The approach is tested on a furniture assembly task and compared with a prior method [6] that instead of aiming to reach a good initial state for the next skill, aims to deal with the initial states of the previous skill.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n------------\n* The paper addresses an important problem, namely, how to compose lower-level policies to solve long-horizon tasks.\n\n* The presentation of the approach is mostly clear.\n\nWeaknesses\n------------------\n* The discussion of prior work is insufficient. Namely, the paper completely misses the branch of research based on skill chaining by Konidaris et al., e.g. [A], [B]. These approaches also train an initiation set classifier and use them to train subskills to reach the initiation set of the next subskill. Whereas [A] considered a reinforcement learning setting, [B] considered learning from demonstrations.\n\n* Especially, when relating the current work to these prior papers, I can not find a significant contribution. Providing additional reward to subskills for reaching initial states from which the subsequent policies succeed, seems to the main idea of the current submission, but this is idea is not novel ([A], [B]). The problem setting is a bit different from those works, because the current submission assumes that the overall task is already segmented and that subskill-specific reward functions are given, but I don't think that these simplifications increase the contribution.\n\n* There is also no theoretical contribution as the presented approach is not based on any proofs or derivations.\n\n* It is not clear how the proposed approach determines whether a subskill succeeds. Is this based on prior knowledge? Is this decision made on the respective state only, or is it maybe even non-Markovian?\n\n* The experimental evaluation seems to be based on only one seed. The appendix contains a table (I assumed based on experiments that were performed after the paper submission deadline) for three seeds, which also contains confidence intervals. However, based on these confidence intervals, T-STAR might actually perform worse than Policy Sequencing (0.86-0.09 < 0.75+0.06). Also, even with three seeds, the experimental results seem insufficient.\n\n* The \"GAIL-reward\" given in line 139 is very different from the reward used by GAIL. This difference needs to be discussed.\n\n* The paper also uses a rather uncommon \"AMP\"-discriminator objectives and also an uncommon reward function for the terminal state regularization, which makes it difficult to interpret the effect of the additional reward term. Typical AIL reward functions would aim to minimize a divergence between the terminal state of subskill $i$ and the initiation set of the next subskill. I see that the chosen reward function in the range $[0,1]$ is easier to combine with the RL and IL reward objectives, but at least a short motivation for these design choices would be very important.\n\nMinor\n-------\n* Why is the one-hot representation of the phase 8-dimensional, even though there are only four phases per task?\n\n* Wouldn't it make more sense to train the classifier between successful and unsuccessful initial states?\n\n* The paper is too repetitive regarding the difference (and benefits) between the current approach and [6]. Namely line 30-43, the caption of Fig. 1, 94-99, 153-161, are mainly repeating the same motivation. I would at least remove the paragraph 153-161.\n\n* Caption should be more descriptive.\n\n* in Figure 2, it would be good to show the \"initial state buffer\"-box also for subtask 1, and the \"Terminal state buffer\"-box also for subtask 3. For example, in the current Figure, it seems like \"initiation Set Discriminator 3\" is based on different types of data compared to \"Initation Set Discriminator 2\".\n\nReferences\n---------------\n\n[A] Konidaris, George, and Andrew Barto. \"Skill discovery in continuous reinforcement learning domains using skill chaining.\" Advances in neural information processing systems 22 (2009): 1015-1023.\n\n[B] Konidaris, George, et al. \"Robot learning from demonstration by constructing skill trees.\" The International Journal of Robotics Research 31.3 (2012): 360-375.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My recommendation is mainly based on the perceived lack of contribution. To me it seems like the paper proposes a hack (since no theoretical justification was given) for fine-tuning chained skills, that is not even significantly different from previous approaches. I also think that the discussion of prior work and experimental evaluation is insufficient.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "* The revision should discuss differences to prior work [A] and [B].\n\n* Rebuttal and revision should make clear how the approach determines whether a subskill succeeded.\n\n* Rebuttal and revision should clarify why 8 dimensions are used for encoding the phase of the skill.\n\n* Rebuttal (at least) should explain why the classifier discriminates between terminal states of the last skill and successful initial states of the current skill, instead of discriminating between unsuccessful and successful initial states of the current skill.\n\n"}}, {"review_id": "2QuWb8kjMx", "reviewer": "Reviewer_opMr", "summary": "This work proposes the algorithm T-STAR which approaches the problem of learning complex extended time-horizon skills through subtask skill chaining. T-STAR combines ideas from RL adversarial imitation learning to regularize the termination state distribution of a subtask policy to be close to the start state distribution of the next subtask. T-STAR builds on ideas in [6] by addressing some limitations in that method. The authors evaluate T-STAR on two furniture assembly tasks in simulation and show that a model-free RL method (T-STAR) is able to solve furniture assembly tasks without manual engineering.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The problem of learning complex long time-horizon skills efficiently is important and relevant to robot learning. The challenges in doing so are clearly identified and well explained. Finally the motivation for the proposed algorithm, T-STAR, is well grounded in the limitations of prior work (see section 3.3 which is especially clear).\n* The proposed algorithm T-STAR is interesting and a novel approach to the skill chaining problem as far as I am aware. It also has the benefit of not requiring any additional policies to be trained over the sub-task policies.\n* The qualitative results (section 4.4 and Figure 4) are a nice way of assessing if T-STAR is behaving as intended. It also visualizes the advantage of T-STAR over policy sequencing [6]. However I do have a question about the choice of time steps (see below)\n* Since the authors are very clear about the differences between T-STAR and PS [6], the difference between PS and T-STAR in Table 1 quantifies the effectiveness of terminal state regularization. However I have a question about the robustness of the results (see below).\n* Clarity of presentation is excellent. The paper is lucid and easy to follow. In particular, Figures 1 and 2 are very clear and do a great job of summarizing the main ideas and algorithm. The use of color highlighting in Algorithm 1 to highlight the differences with prior work is also useful.\n* Inclusion of a mostly comprehensive set of baselines covering behavioral cloning, RL, imitation learning (adversarial), hierarchical RL and policy sequencing through initial state expansion.\n* The implementation details (appendix), algorithm description (algorithm 1), and task description (section 4.2) appear to contain sufficient information for implementing T-STAR and reproducing the results.\n\n**Weaknesses**\n\nLack of robustness in results\n* Only a single score for each algorithm (both for task completion - Table 1 and success rate - lines 266-267) is presented in the main paper, and this appears to be from a single run. There is no discussion of variance in performance or why only a single result is presented.\n* The appendix does present an additional result over 2 or 3 random seeds depending on the task, however there is a discrepancy between the results presented in the main paper (Table 1) and the results presented in the appendix (Table 2) with the results in the appendix on 3 of 4 key results being lower in the appendix.\n\nTable 1:\n\nPS 0.82 | 0.7\n\nT-STAR 0.95 | 0.8\n\nTable 2\n\nPS 0.75 +/- 0.06 | 0.68 +/- 0.09\n\nT-STAR 0.86 +/- 0.09 | 0.87 +/- 0.07\n\n* It is not clear if Table 2 shows results with new seeds different to the seed used in Table 1 or if Table 2 includes the results in Table 1.\n* It is also not clear why only 2 seeds are selected for TABLE_LACK but 3 for CHAIR_INGOLF\n* There is a discrepancy between the caption for Table 2 (which implies that 3 seeds have been used for all tasks) and the text.\nLooking at the results across Table 1 and Table 2, it appears that T-STAR performs significantly better on CHAIR_INGOLF but not on TABLE_LACK. However for the reasons described above this is difficult to assess.\n* To address these robustness issues I suggest\n    1. Unifying Table 1 from the main paper and Table 2 from the appendix and presenting the mean and std deviation across all random seeds in a single table in the main paper.\n     2. Presenting the mean and standard deviation of the task success rate across all random seeds.\n     3. Being clear about how many seeds were used in total for each task and reporting results averaged over all of these.\n     4. Using the same number of seeds for both TABLE_LACK and CHAIR_INGOLF\n     5. Not bolding results if the distributions overlap (e.g. Table 2: TABLE_LACK).\n\nSingle policy baselines do not appear to be a fair comparison with PS and T-STAR\n* The single policy baselines BC, PPO, GAIL, GAIL+PPO are trained for 100M environment interactions, sPiRL is trained for 5M, whilst the PS and T-STAR results are trained for at least 200M interactions (not including the sub-task pre-training).\n\nWhilst there is good coverage in baselines, two alternative approaches that are highlighted and cited by the authors as relevant are not included as baselines. One baseline is an important omission, the other is more minor.\n* Transition policies from [7]. The authors reference this work frequently and it would be valuable to know how T-STAR compares to this approach. This approach directly addresses the policy sequencing problem, so without the inclusion of this baseline it is hard to assess the advantages of T-STAR over this method.\n    1. The authors note that [7] is brittle to external state changes and can fail if the end state of a prior subtask policy is too far away from the initiation set of the following task. It would be valuable to see this assessed and compared with T-STAR experimentally on the furniture assembly task. \n    2. [minor] The second approach is perhaps less directly relevant, so it is understandable why it is not included. However, a comparison with one modulated skills method e.g. one from [11 - 14, 9] would be interesting to see.\n\nMinor\n* Figure 4\n     1. Why were timesteps 36M, 39M, and 42M steps chosen? Without any explanation it seems like these could have been cherry picked.\n     2. It would be useful to see the same visualization evenly spaced throughout the fine tuning part of training, from 0 - 50M steps inclusive.\n* Algorithm 1\n    1. Lines 3 - 7: Is this repeated until convergence for each subtask or only carried out once?\n     2. Line 5: Are the expert demonstrations (sampled from D^e) subtask specific or demonstrate the whole task. If the former then it appears that the subtask index i is missing. If the latter, then how are the relevant parts of the demonstration identified per subtask? \n    3. Line10: What is the sampling ratio and how was it chosen?\n* Table 1: Average progress over how many trials?\n* Typos\n    1. Line 28: “requires to adapting the policies to be suitable” -> “requires adapting the policies to make them suitable”\n    2. Line 200: “an” -> “a”\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Thank you for this submission. T-STAR is an interesting approach to an important problem in robot learning. T-STAR is well motivated, the paper is very clearly written and the main points are easy to understand. As-is however, it is not clear that T-STAR represents a significant improvement over PS or over the other relevant baseline from [7]. If the questions around robustness and baselines (described above) are addressed this would make a strong submission. In the future it would also be great to see results on a physical robot.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* Lack of robustness in results.\n* Fair comparison of single policy baselines.\n* Missing baseline from [7] \n* Please also address the minor issues\n", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the problem of long-horizon skill changing for manipulation tasks and proposes an adversarial learning algorithm - T-STAR - which brings the terminal states of one subtask close to the initial state of the upcoming one (avoiding the need for large initial state distributions required for baseline methods). The method is demonstrated to outperform various RL and IL baselines on two simulated IKEA furniture assembly tasks.

During the rebuttal phase, the authors greatly improved the quality of exposition and addressed the majority of reviewer concerns. All missing citations were added. The addition of std estimates (thanks to 5 seeded runs) shows that the empirical improvements are statistically significant. The experimental details were also made clearer. A majority of reviewers increased their recommended score and I agree with this sentiment. The paper is an interesting contribution to the thread of skill chaining in robot manipulation problems with strong empirical results and would be of benefit to the CoRL community.
Decision: Accept (Poster)