SCaR: Refining Skill Chaining for Long-Horizon Robotic Manipulation via Dual Regularization
Authors: Zixuan Chen
Ze Ji
Jing Huo
Yang Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: long-horizon manipulation
sub-task
skill chaining
robot learning
reinforcement learning
imitation learning
Primary area: robotics

Abstract: Long-horizon robotic manipulation tasks typically involve a series of interrelated sub-tasks spanning multiple execution stages. Skill chaining offers a feasible solution for these tasks by pre-training the skills for each sub-task and linking them sequentially. However, imperfections in skill learning or disturbances during execution can lead to the accumulation of errors in skill chaining process, resulting in execution failures. In this paper, we investigate how to achieve stable and smooth skill chaining for long-horizon robotic manipulation tasks. Specifically, we propose a novel skill chaining framework called Skill Chaining via Dual Regularization (SCaR). This framework applies dual regularization to sub-task skill pre-training and fine-tuning, which not only enhances the intra-skill dependencies within each sub-task skill but also reinforces the inter-skill dependencies between sequential sub-task skills, thus ensuring smooth skill chaining and stable long-horizon execution. We evaluate the SCaR framework on two representative long-horizon robotic manipulation simulation benchmarks: IKEA furniture assembly and kitchen organization. Additionally, we conduct a simple real-world validation in tabletop robot pick-and-place tasks. The experimental results show that, with the support of SCaR, the robot achieves a higher success rate in long-horizon tasks compared to relevant baselines and demonstrates greater robustness to perturbations.

SCaR: Refining Skill Chaining for Long-Horizon Robotic Manipulation via Dual Regularization
Zixuan Chen'
Ze JI Jing Huo
Yang Gao'
State Key Laboratory for Novel Software Technology, Nanjing University, China, 2Schoo of Engineering, Cardiff University, UK.
chenzxenju. edu. en, jiz1@cardiff. ac.uk, huojingenju.edde edu. cn, gaoyōnju edu.
Abstract
Long-horizon robotic manipulation tasks typically involve a series of interrelated sub-tasks spanning multiple execution stages. Skill chaining offers feasible solution for these tasks by pre-training the skills for each sub-task and linking them sequentially. However, imperfections in skill learning or disturbances during execution can lead the accumulation of errors in skill chaining process, resulting in execution failures. In this paper, we investigate how to achieve stable and smooth skill chaining for long-horizon robotic manipulation tasks. Specifically, we propose novel skill chaining framework called Skill Chaining via Dual Regularization (SCaR). This framework applics dual regularization to sub-task skill pre-trace pre-training and fine-tuning, which not only enhances the intra-skill dependencies within each sub-task skill but also reinforces the inter-skill dependencies between sequential sub-task skills, thus ensuring smooth skill chaining and stable long-horizon ecution. We evaluate the SCaR framework on two representative long-howe horizon robotic manipulation simulation benchmarks: IKEA furniture assembly and kitchen organization. Additionally, we conduct a simple real- world validation in tabletop robot pick-and-place tasks. The experimental results show that, with the support of SCaR, the robot achieves a higher success rate in long-horizon tasks compared to relevant baselines and demonstrates greater robustness perturbations.
1 Introduction
Long-horizon robotic manipulation tasks are characterized by sequences of diverse and interdependent sub-tasks, which makes it crucial to maintain the stability of multi-stage sequential exccution. For instance, in the robotic assembly of a stool (Fig. [ involving two sub-tasks of leg installation, overall success evaluated based on both the sequential installation success and factors affecting the assembly within environmental constraints. Although recent advances in deep reinforcement learning (RL) and
imitation learning (IL) show promise in training robots for such complex tasks π 2
3 5
6
managing long-horizon tasks with scratch RL or IL policy remains challenging due to computational demands, extensive exploration, and intricate step dependencies
9. Skill chaining, which involves decomposing Long-horizon tasks into smaller sub-tasks, pre-training skills for each, and executing them sequentially. offers a practical solution [10, [T]. However, as shown Fig. Tb)(,,, such methods tend to fail when sub-task skills are insufficiently trained or unexpected states arise due to disturbances, especially when applied to high-degree-of-frrcee of-fredomm robots performing contact-rich, long- hong-horizon tasks. [12 13
14
15
16 I7
In this paper, we argue that the coordination and enhancing of dependencies within and between sub- task skills is necessary for stable and smooth skill chaining of long-horizon robotic manipulation [10
Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Failed Pre-training Sub-task skills
Failed sub-fask
Failed sub-task 2
intra-skill dependency
intra-skill dependency
leg
leg
(b) Failed Skill Chaining due to Disturbance
mis-aligned
Impact on chair cushion
Goal
Initial state
Sub-fask Fail
inter-skill
deperdency
(c) Skill Chaining via Dual Regularization (SCaR)
Stable and Smooth Stool Assembly
inter-skill
Initial state
Sub-tank I: Assembling stool leg
dependency
Sub-task 2: Assembling unother stool leg
intraskill dependency
intra-kill depenéency
Figure 1: Illustration of the problem setting and the motivation of SCaR, using the example of stool assembly task with two sub-tasks. Best viewed when zoomed in.
For instance, as depicted in Fig. (a)(b,, the robot must consider following two points ensure the overall task is accomplished: 1) ensuring the gripper consistently grasps and installs the stool leg stably within each sub-task skill range. and 2) ensuring the terminal state of previous skill aligns with the initial state of next skill for smooth skill chaining. We define the above two points as intra-skill dependencies between sequential actions within each sub-task skill and inter-skill dependencies between sequential sub-task skills, respectively. In this context, we propose novel robotic skill chaining framework, Skill Chaining via Dual Regularization (SCaR). This framework enhances the aforementioned dependencies alternately through dual regularization during sub-task skill learning and chaining, aiming to provide stability for the execution of long-horizon robotic manipulation.
Specifically, in the pre-training phase of each sub-task skill, we propose the adaptive sub-task skill learning sheeme, which employs a two-part policy learning objective that focuses on what sub-tasks the robot should perform (via RL) and how the robot should perform that task (via IL), and utilizes novel adaptive equilibrium scheduling (AES) regularization the robot's ' learning progress. This process aims to reinforce the intra-skill dependencies, ensuring a coherent sequence of actions in each sub-task skill. Subsequently, bi-directional adversarial learning is introduced in the fine-tuning phase of SCaR for better chaining sequential sub-task skills. This mechanism uses bi-directional regularization to bring the terminal state of the current skill close the initial state of its successor, and also to bring the initial state of the successor close to the terminal state of the current skill. This bi-directional alignment aims to reinforce robust inter-skill dependencies between sequential skills. Through the two innovative designs described, SCaR ensures coordination between the intra-skill and inter-skill dependencies, provides dual constraints for skill learning and skill chaining. described in Fig. (c), leading smooth skill chaining from the inside (within the sub-task skills) to the outside (between sub-task skills). Experimental results show that compared to crratch-training and skill chaining baselines, SCaR provides better task execution performance and stronger robustness to environmental perturbations in various long-horizon and contact-rich robotic manipulation simulation tasks. In addition, conduct simple validation in real-world tabletop robot pick-and- place tasks, and the results show that SCaR achieves a higher task success rate compared to previous skill-chaining methods.
balance these two parts based on
The principal contributions of our work are delineated as follows: 1) We propose a novel robotic skill chaining framework via dual regularization, SCaR, for smoothly executing long- horizon manipulation

---

tasks. 2) We introduce an adaptive sub-task skill learning scheme that acts as a regularization to enhance intra-skit dependencies between sequential actions within each sub-task skill. 3) We develop bi-directional adversarial learning mechanism that serves as regularization for reinforcing inter- skill dependencies between sequential sub-task skills. In all eight simulated long- long-horizon robotic manipulation tasks and simple real-world pick-and-place tasks, SCaR demonstrates significantly better performance than scratch- -tanning and skill-chaining baselines. Video demonstrations are available at: htttps://sitet googtteesggoggeee...... google. com wiww//iwwwwe// /caar8997
2 Related Work
2.1 Long-horizon Robotic Manipulation
Training robots from scratch for complex, long-horizon tasks using reinforcement learning (RL) and imitation learning (IL) is challenging due to computational demands and distributional errors. Solutions involve decomposing tasks into reusable sub-tasks [18]. Typically, such algorithms con- sist of a set of bolicpois that can be obtained through various methods, such as unsupervised
21 22 23], learning from demonstrations [5] 6 25], and predefined mea- sures [26] 27 28 29 14]. Despite the merits of each of these approaches, they do not address well the challenges of long-horizon robot manipulation in environments that object-rich, contact-rich, and characterized by multi-stage tasks (28 29 14]. Thus, even when pre-trained skills are provided,
exploration [19 20
ensuring a smooth connection between manipulation policies remains formidable challenge.
2.2 Skill Chaining for Long-horizon Tasks
Previous skill chaining methods for long-horin long-horizon tasks mainly focus on updating each sub-task policy to encompass the terminal state of the previous policy [1] 14 30), implementing option chains [T] 32 to forge logical skill sequences, or utilizing modulated skills to facilitate smoother
transitions 33 35 36 [4 [66]. However, these methods, while effective, often lead to broad range of skill start and end states, a challenge in complex robotic manipulation tasks. T-STAR (15) is closely related to work, addressing this by regularizing the learning process with discriminator to control the expansion of the terminal state space. However, it focuses only on uni-directional dependencies between skills and ignores intra-skill dependencies within sub-task skills under long- horizon goals. Sequential Dexterity [17] centers on dexterous hand manipulation, introducing an optimization process to backpropagate long-term rewards across a policy chain. However, its scope still primarily emphasizes strengthening the dependencies between sub-task skills. GSC [7 attempts to solve skill chaining by employing diffusion models. It trains and chains primitive skills (pick, place, push, pull) through a Transformer based skill diffusion model. However, due to the use of Transformer-basse
techniques, GSC requires high computational resources and cannot scale well to task environments with object-rich and contact-rich conditions. Our method instead employs simple and intuitive dual regularization constraints based on the lightweight policy network. By coordinating the dependencies within and between skills, we achieve refinement within sub-task policies and bi-directional alignment between them. This allows for stable skill chaining while also being scalable to various long- mng-horizon manipulation tasks.
3 Preliminaries
Among several related works on skill chaining, we consider a challenging yet practical problem setting that deals with long-horizon manipulation tasks through combination of reinforcement learning (RL) and imitation learning (IL). In each sub-task in the horizooiioon task, we consider robotic agents acting within a finite-horizon Markov Decision Process (38) (S, A, P.r, r, dz, T). where S is the state space, A is the action space, P(s'|s [s, a) is the transition function, r(8, a, s') is the reward function, 7 is the discount factor, dz is the initial state distribution, and T is the episode horizon of sub-task. We define a policy π : S - A that maps states to actions and correspondingly moves the robotic agent a new state according the transition probabilities. This sub-task policy is trained to maximize the expected sum of discounted rewards E(s,a)~= (Et=a>><E'== 7'1(s1, at, 8t-1)]. We assume that each sub-task policy has an initial state set I E S and a terminal state set 3 E S, where the initial set I contains all the initial states that lead to the successful execution of the policy and the terminal state set 3 contains all the final states of the successful execution. The environment
3

---

Example performing SCaR table Jack sub-tasks)
Phase 1: Adaptive Sub-task Skill Lcaming
Phase 2: Bi-directional Adversania rial Learning
Mosine
bakake progres
Environment
Environment
Demonstration
Demunsiration
aggres
Sub-1okk
9ub-laa
Bub-took
9ul--aak
f2
Misitoe
Sase
Figure 2: The Pipeline of Skill Chaining via Dual Regularization (SCaR). (Left) Phase 1: Sub-task skill pre-training ( ees merges environmental feedback and expert guidance, using adaptive cquilibrium scheduling (AES) regularization balance learning, which enhances intra-skill depen- dencies within skills. (Middle) Phase 2: Bi-directional discriminators coupled with AES to fine-tune pre-trained sub-task skills, as regularization for reinforcing inter-skill dependencies. (Right) Evaluation: Evaluation of SCaR long-horizon manipulation.
provides the environmental feedback for each step taken by the agent and success metries for each sub-task, derived from the terminal states of sub-task policy. For instance, as shown in Fig. [c). the alignment the back and legs of the stool triggers the connect action and the realization of the sub-task goal, which indicates the successful completion of the sub-task. Additionally, we posit that during each sub-task policy learning, the agent receives a set of pre-defined expert demonstrations, DE [T .... ) ). facilitate the IL process. Here, N represents the number of episodes, and
each demonstration comprises a sequence of state-action pairs, 7 (81, a........ -T-1, aT (0T-111 8T)).
4 Method
we present the pipeline the SCaR framework. Sections 4.2 and 4.3 provide further elaboration on the key design elements.
In Section 4.1
4.1 Overall Pipeline
As illustrated in Fig. 2 the SCaR framework has two phases: (a) pre-training (adaptive sub-task skill learning) and (b) fine-tuning (bi-directional adversarial learning). In the pre-training phase, the agent co-learns sub-task skills by integrating environmental feedback and expert demonstrations. In the fine-tuning phase, it refines these skills through bi-directional adversarial learning, enabling sequential integration of sub-task skills. After fine-tuning, SCaR can smoothly chain sub-task skills to complete long-horizon manipulation tasks. Specific modules and mechanisms for these phases are detailed Sections 4.2 and
4.2 Adaptive Sub-task Skill Learning
Weighted Reward Function To learn sub-task skills better, we combine goal-conditional RL and generative adversarial imitation learning (GAIL) (3), pre-train skills that enable the agent to perform challenging sub-tasks in desired expert behavioral style 44 15]. More specifically, we consider the weighted reward function that is used train each sub-task policy x consists of two components specifying: what sub-task the agent should perform learning from environmental feedback, and 2) how the agent should perform that task learning from expert demonstrations:
r(si, at St+1:0) =>RL. (St, aes 8t41,9) g) Predr,reddss oc: Ø).
As shown in Eq. [ the first component is represented by a task- reward r. F a, St+1. which defines general objectives that the agent should satisfy to fulfiil a given sub-task goal y for current MDP M (e.g. assembling stool leg). The second component is represented through a learned task-agnostic predict-reward y PPed (st, dr; ()), which specifies manipulation details of the behaviors that the agent should adopt when performing the sub-task (c.g., the expert way to grab stool leg

---

and attach it), and prProd (st, ag; Φ) is the predicted reward by least-square GAIL discriminator f
40 [5]. which is more stable than the standard GAIL objective using the sigmoid cross- entropy loss function. Therefore, the predicted reward is:
r: Pred (st; Ge; Φ) = max (0, 1 0.25 . [f (s6, az) - 1]7].
(2)
We adopt the training objective of the eaat-squares GAIL discriminator 41 with a gradient penalty
term 42 43 This penalty term mitigates the instability of the training dynamics due to the interplay between the discriminator and the policy [40) as follows:
()))) ((f((s) + +) TAD
argming; /3 ( (ff(() 1)
2 (())-==IVVe fa))(
where 178P is manually-speciin works are set by fixed weights and linearly combined into the final reward function [40] [5]. This could lead to the agent rigidly imitating experts and curbing self- self-exploration, finding it difficult to adjust intra-skill dependencies and adapt to dynamic task perturbations. We propose a principle to counter this: If the agent fails to imitate the expert's 's demonstration well, it should shift focus to self-learning from the environment. Conversely, effective imitation should continue, focusing on the expert to mitigate low sample efficiency in reinforcement learning. Accordingly, we extend the automatic discount scheduling (ADS) solution [9] to our problem setting, and propose adaptive equilibrium scheduling (AES) to regularize the scales of y ,av and Pred in sub-task skill learning for adaptive scheduling the focus of reinforcement and imitation learning, as shown in Fig. 3
coefficient. The scales of y Fnv and Pred in previous related
Adaptive Equilibrium Scheduling (AES) Regularization Specifically, AES balances the scales of y Env and pred during the learning process of each skill through adaptive scheduling of ARL and ÅIL. according to how well the agent imitates the expert's 's demonstration. To capture the agent's imitation progress, AES refers to the solution in ADS [9] and uses the imitation identifier + to continuously monitor whether the agent is imitating the expert demonstration well enough.
At the beginning of training, the agent assigned two initial balance factors ART. = ax Aur. exponent a E [o, 1]. We set a 0.5 in the experiments and the agent is assigned two identical balance factors
where base
Pred
Expert demas
ARL = \yL = 0 indicating that at the beginning of learning, the agent imitates the expert's 's behavior with the same weight as the behavior of environment exploration according to the task goal. As training progresses, the imitation progress recognizer & is queried periodically to monitor the progress of the agent's imitation of the expert's behavior. &b receives the agent's collected trajectories and infers the agent's current imitation progress p E [0, T), where in an integer and T' is the step of the entire episode.
Scheduling
Monitor
AES
Scheduling
Am
Environment
Env
Action
The construction of 4, with reference to ADS, first requires the construction of a sequence Q(g...... gr), where qi argmin,c(ss ((s,, sff) is the index of the near- est neighbor of s; in r², c is the cosine similarity. The progress alignment between and rf is measured as the length of the longest increasing subsequence (LIS) in Q, denoted LIS(T, D). Specifically, the agent's imitation progress p is increased by 1 if the following inequality holds:
Figure 3: AES regularization for sub- task skill learning.
max LIS(TIP41: F
EE 1 2 px min LIS A1p
AE
(4)
where f of fE the subscript 1 p + denotes the first + 1 steps of the trajectory, and E [o. 1] controls the strietness of the imitation progress monitoring. This suggests that the similarity of the agent trajectory to its best matching expert trajectory at time step p+ exceeds the minimal similarity criterion within the expert demonstration. See Appendix B for detailed explanation of AES.
After obtaining the current imitation progress of the agent, AES then adopts a mapping function xpx (p) to schedule the two new balance discount factors ARL and ÅLL Straightforward idea of setting
We further explore what effect different would have in the Ablation Experiments.
5

---

(p) is that If p is larger and reaches certain threshold, i.e. the agent is able to imitate the expert behavior well, then the more the agent tends imitate the expert's behavior in subsequent training, and vice versa. Therefore, we set the threshold as T If p E [0, 7), we
2
propose <x(p) 1 c E); if p E 122 T), we propose (x(())
where k is used to
flatten the curve of the mapping function. Then ARL and AIL are scheduled to be
{ ARL = (xx(p), An = =1-apx00)
p E [0, F) AXI. - (FF(P)) ARI. = -1-a))) if E (T,
(5)
Consequently, the RL and IL components of sub-task skill learning can be adaptively scheduled and regularized through AES, effectively enhancing intra-skill dependencies between sequential actions. The pseudo-code of adaptive sub-task skill learning is outlined in Algorithm in Appendix AA.
4.3 Bi-directional Adversarial Learning for Skill Chaining
Executing pre-trained sub-task skills sequentially without considering inter-skill dependencies may lead to failure. To address this, we propose bi-directional adversarial learning to further refine and better integrate sequential sub-task skills. The pseudo-code of bi-directional adversarial learning is outlined in Algorithm 2 in Appendix A.2
Bi-directional Regularization In contrast to previous uni-directional regularization schemes that only augment the initial state set Ij or regularize the terminal state set 33 [ [5], we impose the bi-directional constraints (C2,C2) on inter-skill dependencies, facilitating smooth skill chaining, as shown in Fig With the bi-directional constraint, we implement the bi-directional adversarial learning, centered on the joint training of a bi-directional discriminator, denoted by S which is adept at distinguishing between the terminal state set of the preceding policy and the initial state set of the subsequent policy. The bi-directional constraints Cr.C2 are defined as Eq. TO)
next initial + current terminal:
C = =EsT~Tu4 Tu+ IShn (ST) - 1] + Esr
BR ISús, (ST)]²
(6)
previous terminal - current initial: C2 = Es Eg==Errp
Sig (ST) 11 + EsgNT, Isi [sT]]²
Sa, and She are two separate networks, each used to minimize the adversarial learning process in two differ- ent directions, and the parameters of the two networks are averaged and combined into Ç In summary, C is
uni-directional
"pull" to presrious sarts
"pushh" nest sets
trained for each policy minimize the objective func- (i) = t #C,+1C2. Guided by C the bi-directional adversarial learning not only steers the terminal state set of the current policy towards the initial state set of the subsequent policy, but also ensures alignment of the ini- tial state set of the subsequent policy with the terminal state set of current policy. This dual alignment estab- lishes a balanced mapping between the initial and ter- minal states of sequential skills to reinforce inter-skill dependencies, ensure consistency and stability in multi- stage tasks, and guarantee smooth transitions between sequential skills. Accordingly, the bi-directional regular- ization can be added to the overall objective function of policy learning in the form of the following reward term: ri (s;w) L (s) + LI
bi-directional (ours)
"pull" previous sats
"push" neot sets
"push" next
"pull" previous sets
Figure 4: Bi-directional regularization for sub-task skill chaining.
Overall Objective Function So far, the objective function via dual regularization, i.e., AES regularization and bi-directional regularization, to pre-train, fine-tune and chain sub-task skills can be rewritten as a weighted sum of the individual reward terms:
rs(5t, at, St+1: Φ) = >ALL Env (st, at, 8141,9) + Au Pred (st, ar; ④)
+ Bi (SL+1;W) (7)
AES regularization
bi-directinnal regularization
We explore the impact of different scales of Ci and C2 in Appendix D.3

---

Sob-task
Sub-tack
Sub-task
Sub-task
Sub-taik
Sub-tank
Sub-task
Sub-taska
Sub-task
100
Success rate (%)
80
Success rate (%)
60
60
40
20
(BOCAM
caciett toae))
Sub-tasks in chair Ingolf
Sub-tasks Extended Kitchen
m9
CAL
Fowd-ALL AL-.
Adaptive Learmng (5Caf)
PPO
GAL
Rull
Adaptive Skill Leamnng (SCal)
(a) chair ingolf
(b) Extended Kitchen
Figure 5: Evaluation Performance of Sub-task Skill Learning. Best viewed zoomed.
where ARe is the weighting factor of the bi-directional regularization. The objective function features AES regularization and bi-directional regularization enhance intra- and inter-skill dependencies. It enables the agent to adaptively pre-train skills that can solve different sub-tasks well through envirommental feedback and expert guidance, and further tune them through the bi-directional discriminator to achieve dual aligmment between sequential skills. At the same time, the fine-tuned sub-task skills help to collect terminal and initial states to refine the bi-directional discriminator. This iterative process ensures smooth long-horiag mogghorizon task skill chaining.
Experiments
5.1 Experiment Setup
We conduct simulation experiments on six IKEA furniture assembly tasks and two kitchen organi-
zation tasks, and also perform long-horizon pick-and-place experiments on the real Sagittarius K robot. Please refer to the Appendix for more detailed simulation experiment setup (Appendix G
network architecture (Appendix H training details ( (Appendix[], more quantitative (Appendix D) and
qualitative results (Appendix of the simulation tasks, and the real-robot experiments (Appendix
Furniture Assembly We conduct experiments in six IKEA furniture assembly tasks in chair Lagnr_
chair_ingol
baolelack,, toy toy_lable, and bookk_doccstra..
1) chair_agne: Two stool legs need to be picked up and aligned with the cross notches on the stool back. 2) chair bernhard: The two chair supports need to be taken and aligned with the slots the bottom the chair surface. 3) chair ingolf: Two chair supports and front legs need to be attached to chair_ingolf
agne:
the chair seat, which must then be secured to the chair back while avoiding collision with each other. 4) table lack: The four table legs need to be picked up and aligned with the corners of the tabletop. 5) toy table: The four table legs need to be picked up and aimed and inserted with the four notches the table back. 6) table_ _lockstrrrr: After supporting the two bases with table leg, the table top needs to be mounted while preventing collision. For each assembly task, define the assembly of individual parts as sub-tasks. We collect 200 demonstrations per sub-task using a procedural assembly policy for imitation learning. Each demonstration consists of 150 steps.
Kitchen Organization We use the Franka Kitchen tasks in D4RL 45 and collect 200 demonstra- tions per sub-task for imitation learning. Specifically, we refer to the kitchen task in 44 and further extend the task sequence: in the Kitchen task, the 7-DoF Franka Emika Panda arm needs to perform 4 sequential sub-tasks namely Turn the microwave Move the kettle Turn the stove Turn the light. In the Extended Kitchen task, the robot needs to perform 5 sequential sub-tasks: Turn on the microwave- Turn on the stove Turn the light Slide the cabinel the right- Open the cabinet, in which the sub-tasks have a lower probability of switching and is more challenging.
Baselines
We compare SCaR with the following two types of baselines:
7

---

Table 1: Long-horizon tasks execution performance (varies by sub-task completion progress); *tasks with sub-tasks progress by 0.5 per sub-task, "tasks with 4 sub-tasks by 0.25, *tasks with 5 sub-tasks by 0.2, and table_ doocktta with 3 sub-tasks by 0.3, where 0.9 indicates completion of all tasks. Best viewed zoomed.
Furaiture A Avacublly
Kitchen Organization
Method
choir gane
chair Jornhard
clatr Jngoly
fabll Jack
fable
soble dockate
All
Kitchen
E-Kithhen
All
PPO (Scratch RL)
0.5
0.00
0.3
0.25
0.13+
c.03.cco
0.0
GAIL. (Serstch
0.3
0:23-
0.00+ coe
0.00+000
0.00+.mm
0.21-
0:12+am
0.00-oro
0.00-cao
0.00-mm
Fiaed-RI-I1.
0:5
0.23-am
0:3
0.1 S
0.4--re
0:37+ams
0.38-
0.18-ca
0.26-om
SliMo
0.75 caw
cas
0.47 a.0
0.34 4.s
0.56 anu
0.57 000
1121 LCM
Pullcy Sequeneinn
0.89 cie
0.82 1:00
0.77 -.15
0.63 -0.5
0:45 mea
0.61 -036
0,90 +ais
0.53 -n.1
0.36 c.
0.44 c:
T-STAR
0.92 co
0.90 1:4
0.89 sM
0.71 p:21
0.77=cmm 0000
0,55 zom
0.68 20.1
0:46 cM
0.58 2010
SCaR m/o Bi
0.93 co
0.91 am
0:93 Love
0.50 2:00
0.79 co02
0.88 oous
0.75 2008
0.572144
SCaR we ALS
0.95
0.94..4
0.931 ezz
0,95
0.85.
0.80_oxo
0,911
0.77..00
0.61.c13
0.74_
SCaR (Ours)
0.98. caz
0.96
0.95: 6:0
0,97.o00
0.92.
0,88_
0.94 Lciz (12% 1)
0.54....5
0.73.c17
0.78 a12 (18%
Scratch Training: PPO is model-free RL algorithm [47 that utilizes environmental rewards to Iearn tasks from scratch. 2) GAIL [39] is an adversarial imitation learning method to learn tasks from scratch, with trained discriminator for distinguishing state-action distributions of experts and agents. 3) Fixed-RL.-II RL 44) uses fixed-weight environmental rewards and GAIL rewards to train policies from scratch. 4) SkiMo [46] is a model-based hierarchical RL approach that learns dynamic skill models for predicting outcomes in downstream tasks, which is used to test if modularly skill chaining method can surpass model- seratch- method on nong-horizon tasks.
Skill Chaining: 1) Policy Sequencing [2] focuses on sequentially expanding the initial sets in skill chaining. 2) T-STAR [15] incorporates a discriminator to uni-directionally regularize the terminal states of sub-skills in a skill chaining. 3) SCaR w/o Bi reference to T-STAR during the fine-tuning phase, only uni-directional regularization of the terminal state set is performed to verify the validity of the proposed bi-directional regularization. 4) SCaR w/o AES fixes the scales of the two reward terms at 0.5 at all times to verify the effectiveness of the proposed AES regularization.
5.2 Quantitative Results
Sub-task Skill Learning Performance First, we evaluate the proposed adaptive sub-task skill learning scheme in the sub-tasks of furniture assembly and kitchen organization. Specifically, we treat each sub-task policy tested in the reset sub-task as the criterion. All methods are trained in each sub-task with random seeds, 150 million environment steps, and evaluated with the average success rate over 100 testing episodes. As shown in the Fig. 5 in chair_ingol and Extended Kitchen tasks, even with the increase of objects in the environment and the increase of unpredictable perturbations, our proposed adaptive skill learning learns good sub-task skills and consistently maintains a task success rate of more than 85% in all stages of the sub-task. In contrast, the PPO (only RL rewards). GAIL (only IL rewards), and Fixed-RL-IL (fixed RL and IL reward weights) baselines fail to maintain good sub-task success rates as the number of sub-task stages increases. This result well validates that our proposed adaptive weighted reward function based on AES regularization enhances intra-skill dependencies for multi-stage sub-task learning and brings effectiveness and stability.
separate task for policy learning and take the success rate of the trained
Long-horizon Execution Performance We then demonstrate the performance of SCaR in perform- ing 8 long-horizon tasks in IKEA furniture assembly and kitchen organization. Table T shows the mean and standard deviation for these 8 tasks across 200 testing episodes with 5 different seeds. The PPO and GAIL baselines show minimal success on tasks with 4 and 5 sub-tasks, indicating the diffi- culty of learning complex multi-stage tasks solely from reward signals or expert demonstrations. The fixed RL-IL bascline, although improved compared PPO and GAIL, mostly completed only one sub-task, which highlights the limitations of using fixed RL and IL reward weights in long- horizon tasks. While SkiMo achieves better success rates than model free methods by building dynamic skill models, its performance remains inconsistent on long-horizon tasks due to its scratch learning nature. The performance of these scratch baselines demonstrates the importance of effective staged sub-task learning for long-horizon tasks. The results in Table [] further highlight the superiority of the SCaR framework. By reinforcing intra- and inter-skill dependencies, task success rates are considerably higher than previous skill chaining approaches such Policy Sequencing and T-STAR, which primar-
8

---

ily address uni-directional inter-skill dependencies. Compared to T-STAR, SCaR increases average success rates by more than 12% on six furniture assembly tasks and 18% on kitchen tasks D
5.3 Robustness to Perturbations
Perturbation tests are conducted to evaluate the robustness of skill chaining for fur- niture assembly tasks. As shown in the top figure of Table 2 for the chair_ berrhard task, the perturbation involves applying ex- ternal joint torque the robotic arm, mov- ing the chair back before assembling the second support. For the chair chair_ingolf task, the perturbation is applied by exerting ex- ternal torque on the robotic arms, causing them to move slightly before mounting the assembled chair seat to the chair back. The results in Table 2 highlight the detrimen- tal impact of environmental perturbations the success rates of baseline methods during the execution of multiple sub-task skills. Methods like Policy Sequencing and T-STAR, which focus solcly inter-skill dependencies through uni-directional regularization, struggle to complete tasks after perturbations. In contrast, SCaR, demonstrates more robust performance under unseen perturbations. These results further support the advantages of our proposed dual regularization for stable skill chaining on long-horizon manipulation tasks.
Table 2: Comparison of the robustness of skill chaining in perturbed environments.
Perturbation
Porturbation
clair bernhond
cbair, uagoly
Method
No Perturb
Perturb
No Perturb
Perturb
Policy Sequencing
0,82
0.51
0,77
0.50
T-STAR
0,90
0.60 2m
0.89
0.59
SCaR wio Bi
0.92 om
0.65 am
0.91 am0
0.63 2:05
SCaR wio ARS
0.94 am
0.74 mm
0.93 g.
0.71 gore
SCal (Ours)
0.96 c.
0.85 ..
0.95 400
0.80 a.
5.4 Ablations and Analysis
We perform ablation studics to explore the important factors that affect the performance SCaR.
Offects of scales and reward items
Sequerding
Success rate (%6)
Success nate (%)
dhet Aggz
dhiihhja""
Dase exponent a (a)
Demonddrationss taskes)
Figure 6: Ablation experiments.
Modular Ablation We investigate how the adaptive sub-task skill learning and bi-directional adversarial learning impact skill chaining through SCaR w/o Bi and SCaR w/o AES. As shown in Table [ without bi-directional regularization, SCaR w/o Bi experiences significant performance drops in tasks with more than two sub-tasks but still outperforms T-STAR. This is because SCaR w/o Bi maintains the adaptive scheduling of AES during sub-task skill learning, underscoring the importance of focusing the intra-skill dependencies between successive actions. Similarly, the absence of AES regularization reduces SCaR w/o AES's performance, though it still maintains stable outcomes. This underscores the importance of reinforeing inter-skill dependencies on long-horizon tasks and reaffirms the contribution of bi-directional regularization. As shown in Table 2 SCaR w/o
'The overall increase is somewhat modest due to averaging the success rates of the 2, 3, and 4 sub-tasks and the and sub-tar tasks, respectively.

---

Bi, though slightly more robust than T-STAR due to the presence of AES, still faces challenges in adapting to perturbations and maintaining stable skill chaining because of its uni-directional fine- tuning limitations. SCaR w/o AES manages to maintain certain level of performance stability under perturbations, thanks to bi-directional regularization, which ensures the bi-directional alignment of initial and terminal states between skills. The results show that the pre-trained skills via AES exhibit enhanced intra-skill dependencies within sub-tasks, and bi-directional regularization ensures stable long-horizon execution, even in the presence of perturbations, by reinforcing inter-skill dependencies.
Parametric Ablation We further investigate the impact of different scales of RL and IL reward terms, as well as the size of expert demonstration datasets. The effect of varying the base exponent on task success rates is tested across four tasks: chair chair_agne, chair _ingolf. table dockstra, and extend kitchen. As depicted in Fig. a(),, SCaR achieves the highest success rates in all four tasks when a 0.5, indicating a balance between RI. and II. at the beginning of learning. When a becomes smaller, emphasizing II. at the start, performance decreases more steeply. Conversely, as becomes larger, giving more weight to RI,, performance also declines but at a slower rate. We also evaluate the impact of different sizes of expert datasets three skill chaining methods: Policy Sequencing, T-STAR, and SCaR, specifically in the chair_ _ingo_ task. We vary the overall task expert data size from 80, 120, 200, 400, 600, to 800 demos. As shown in Fig. bb), the results indicate significant performance improvement when increasing the dataset size from 400 to 800 demos, while the improvement is less pronounced when going from 80 to 120 demos. This demonstrates the importance of the demo dataset size in the effectiveness of data-driven approaches like skill chaining.
6 Discussion
Limitation and future directions The primary limitation of work is that the sub-task division for long-horizon tasks is predefined and does not incorporate visual or semantic processing of objects. Expanding our framework handle longer-horizon visual manipulation tasks is direction we aim to explore in future research. For example, integrating a more scalable architecture [48] and performing large-scale pre-training on extensive datasets 449 50 are promising directions. Another avenue
worth exploring is applying our framework to real-world robotic furniture assembly tasks, rather than only staged pick-and-place tasks. Constructing a deployment environment for real-world furniture assembly and ensuring the complete insertion of each furniture module presents significant challenges. We discuss additional limitations and potential solutions in further detail in Appendix
Conclusion In this paper, we introduce SCaR, a novel skill chaining framework that ensures smooth and stable execution of long-horizon robotic manipulation tasks via dual regularization within and between sub-task skills. Extensive experiments demonstrate that the SCaR framework achieves better task success rates than the baseline methods in both simulated and real-robot manipulation tasks, while being robust against perturbations. We hope this work will inspire future research to further explore the potential of skill chaining for long-horizon robotic manipulation.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Science and Technology Innovation 2030 New Generation Artificial Intelligence Major Project under Grant 111201113333 in part by the National Natural Science Foundation of China under Grant 62192783, Grant 62276128; in part by the Collaborative Innovation Center of Novel Software Technology and Industrialization; in part by the Fundamental Research Funds for the Central Universities under Grant 143801 28; in part by the Postgraduate Research & Practice Innovation Program of Jiangsu Province under Grant K KYCX240022)
References
[I] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1): -3733, 2016.
[2] Francisco Suárez-Ruiz and Quang-Cuoe Pham. A framework for fine robotic assembly. In
2016 IEEE international conference on robotics and automation (ICRA), pages 421-426. IEEF, 2016.
10

---

[3] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. Robotics: Science and Systems X/V, 2018.
[4] Divye Jain, Andrew Li, Shivam Singhal, Aravind Rajeswaran, Vikash Kumar, and Emanuel Todorov. Learning deep visuomotor policies for dexterous hand manipulation. In 2019 interna- lional conference on robotics and automation (ICRA), pages 3636-3643. IEEE, 2019.
[5] George Konidaris, Scott Kuindersma, Roderic Grupen, and Andrew Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, 31)33360-375, 2012.
[6] Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefenstette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3428, PMLR, 2019.
[7] Chen Wang, Linxi Fan, Jiankai Sun, Ruohan Zhang, Fei-Fei, Danfei Xu, Yuke Zhu, and Anima Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. arXiv preprint arXiv:2302. 12422, 2023.
[8] Marcin Andrychowicz, Anton Raichuk, Piotr Stariczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations, 2020.
[9] Yuyang Liu, Weijun Dong, Yingdong Hu, Chuan Wen, Zhao-Heng Yin, Chongjie Zhang, and Yang Gao. Imitation learning from observation with automatic discount scheduling. arXiv preprint 2rX10:2310.07433, 2023.
[10] Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Integrated task and motion planning. Annual review of control, robotics, and autonomous systems, 4:265-299 2021.
[11] George Konidaris and Andrew Barto. Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 22, 2009.
[12] Alexander Clegg, Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Learning to dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions on Graphics (TOG), 37(6):1-10, 2018.
[13] Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph J Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2018
[14] Youngwoon Lee, Jingyun Yang, and Joseph J Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International conference on learning representations, 2019.
[15] Youngwoon Lee, Joseph J Lim, Anima Anandkumar, and Yuke Zhu. Adversarial skill chaining for long-horizon robot manipulation via terminal state regularization. In Conference on Robot Learning (CoRL. 2022), pages 406-416. PMLR, 2022.
[16] Jiayuan Gu, Devendra Singh Chaplot, Hao Su, and Jitendra Malik. Multi-skill mobile ma- nipulation for object rearrangement. In The Eleventh International Conference on Learning Representations, 2022.
[17] Yuanpei Chen, Chen Wang, Li Fei-Fei, and Karen Liu. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation. In Conference Robot Learning, pages 3809-3829. PMLR, 2023.
[18] Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 112(1- 2):181-211, 1999.
11

---

[19] Jürgen Schmidhuber. Towards compositional learning with dynamic neural networks. Inst. für Informatik, 1990.
[20] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. Proceedings of the AAAI conference artificial intelligence, volume 2017.
[21] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[22] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierar- chies with hindsight. arXiv preprint arXiv:1712.00948, 2017.
[23] Visak CV Kumar, Sehoon Ha, and Karen Liu. Expanding motor skills using relay networks. In Conference on Robot Learning, pages 744-756. PMLR, 2018.
[24] Yuchen Lu, Yikang Shen, Siyuan Zhou, Aaron Courville, Joshua B Tenenbaum, and Chuang Gan. Learning task decomposition with ordered memory policy network. In International Conference Learning Representations, 2020.
[25] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318 18.. PMLR, 2023.
[26] Tejas D Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[27] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. In International Conference on Machine Learning, pages 2661-26700 PMLR, 2017.
(28] Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In International Conference on Learning Representations, 2018
[29] Chen Wang, Danfei Xu, and Li Fei-Fei. Generalizable task planning through representation pretraining. IEEE Robotics and Automation Letters, 7(3):8299-88) --006,88666000 2022.
[30] Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mcp: Learning composable hicrarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 32, 2019.
[31] Akhil Bagaria and George Konidaris. Option discovery using deep skill chaining. In Interna- tional Conference on Learning Representations, 2019.
[32] Zixuan Chen, Ze Ji, Shuyang Liu, Jing Huo, Yiyu Chen, and Yang Gao. Cognizing and imitating robotic skills via a dual cognition-action architecture. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 2204-2206, 2024.
[33] Peter Pastor, Heiko Hoffmann, Tamim Asfour, and Stefan Schaal. Learning and generalization of motor skills by learning from demonstration. In 2009 IEEE International Conference on Robotics and Automation, pages 763-768. IEEE, 2009.
[34] Jens Kober, Jan Peters, Jens Kober, and Jan Peters. Movement templates for learning of hitting and batting. Learning Motor Skills: From Algorithms to Robot Experiments, pages 69-82, 2014.
[35] Katharina Mülling, Jens Kober, Oliver Kroemer, and Jan Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Roboties Research, 32(3):263-279, 2013.
[36] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. International Conference on Learning Representations, 2018.
12

---

[37] Utkarsh Aashu Mishra, Shangjie Xue, Yongxin Chen, and Danfei Xu. Generative skill chaining: Long-horizon skill planning with diffusion models. In Conference on Robot Learning, pages 2905-2925. PMLR, 2023.
[38] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey, Journal of artificial intelligence research, 4:237-285, 1996.
[39] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[40] Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (ToG), 4004) 2021.
[41] Xudong Mao, Qing Li, Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley, Least squares generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2794-2802, 2017.
[42] Naveen Kodali, Jacob Abernethy, James Hays, and Zsolt Kira. How to train your dragan. arXiv preprint 1rX5.0:705.072/5 2(4), 2017.
[43] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? In International conference on machine learning. pages 3481-3490, PMLR, 2018.
[44] Youngwoon Lee, Edward S Hu, and Joseph J Lim. Ikea furniture assembly environment for long-horizon complex manipulation tasks. In 2021 ieee international conference on robotics and automation (icra), pages 0343-6349) IEEE, 2021.
[45] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[46] Lucy Xiaoyang Shi, Joseph J Lim, and Youngwoon Lee. Skill-based model-based reinforcement learning. In Conference on Robot Learning, pages 2262-2272. PMLR, 2023.
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1007.06347, 2017.
[48] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, 2021.
[49] Sudeep Dasari, Frederik Fbert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning, 2019.
[50] Abhishek Padalkar, Acom Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and models. arXiv preprint arXiv:2310.0080
2023.
[51] Siddhant Haldar, Vaibhav Mathur, Denis Yarats, and Lerrel Pinto. Watch and match: Super- charging imitation with regularized optimal transport. In Conference on Robot Learning, pages 32 43. PMLR, 2023.
[52] Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conference on Robot Learning, pages 1025-1037. PMLR, 2020.
[53] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito,, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[54] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference machine learning, pages 9118-9147, PMLR, 2022.
13

---

[55] Yingdong Hu, Fanqi Lin. Tong Zhang, Li Yi, and Yang Gao. Look before you leap: Unveiling the power of gpt-4v in robotic vision- laiiuae planning. arXiv preprint arXiv:2311.17842. 2023.
[56] Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, and Yang Gao. Copa: General robotic manipulation through spatial constraints of parts with foundation models. arXiv preprint 203.022403 2024.
[57] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive rein- forcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2/06.05091, 2021.
[58] Younggyo Seo, Junsu Kim, Stephen James, Kimin Lee, Jinwoo Shin, and Pieter Abbeel. Multi- view masked world models for visual robotic manipulation. In International Conference on Machine Learning, pages 30613-3000 0011330632... PMLR, 2023.
[59] Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. Croco: Self-supervised pre-training for 3d vision tasks by cross-view completion. Advances Neural Information Processing Systems, 35:3502-3516, 2022.
[60] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang. Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, et al. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. arXiv preprint arXiv:2311.09217 2023.
[61] Yicong Hong, Kai Zhang, Jiuxiang Gu. Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint Stti::2311.04000 2023.
[62] Dhruv Shah, Blažej Osiáski, Sergey Levine, et Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference robot learning, pages 492-504. PMLR, 2023.
[63] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 2022.
[64] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.033378 2023.
14

---

Technical Appendix
A
Pseudo-code
Pscudo-code for adaptive sub-task skill learning and bi-directional adversarial learning are shown in Algorithm I and Algorithm (eeorithmmrespectivvlly We highlight the key differences between method and the most relevant T-STAR with a gray background.
A.1 Adaptive Sub-task Skill Learning
As shown in Algorithm| ! the innovation of the sub-task skill learning scheme we propose, compared to previous methods, consists of two parts: 1) We use more stable weighted reward function for policy learning of sub-task skills, as shown in Eq. [ and Eq. 3 in the main paper. 2) We introduce AES regularization constraints into this weighted reward function periodically adaptively schedule the scale of the two reward terms, as shown in line 11-14 of Algorithm [ allowing the robot to fully explore and learn from both the environment and the expert behaviors.
Algorithm 1 Adaptive Sub-task Skill Learning.
Key differences to T-STAR [15] in gray.
1: Require: expert demonstrations DD DE K. sub-task MDPs Mi,...,MK
2: Initialize sub-task policies m) ..... "
K least-squares GAIL discriminator fa.... f 3: Initialize imitation progress recognizer d with D balance discount factor ARL t a, AIL t
4: for each sub-task = 1,.....k do
5: for episode 2. do
6: Rollout trajectories r (81 01, Env
,...,,8)) ST) with Ti
7:
// WEIGHTED REWARD FUNCTION
8:
Compute balanced reward {r....., -T + ARI Env AII
9:
Update f with 7 and - Dz using Eq.
10:
Update Tà with the rewarded trajectories (31 01 T1 TI...., // ADAPTIVE EQUILIBRIUM SCHEDULING REGULARIZATION
11:
12:
Update imitation progress recognizer & with and T ~ D Query o about the current imitation progress p
13:
14:
Update balance discount factor ARL, AnL < (p)
15:
end for
16:
end for
A.2 Bi-directional Adversarial Learning
As shown in Algorithm [ the innovation the bi-directional adversarial learning mechanism consists of two parts: 1) We propose a bi-directional regularization which is trained by two balanced bi- directional constraints to better chain sequential skills, shown in line 16-17 of Algorithm 2 2) We also employ the adaptive sub-skill learning scheme during the bi-directional adversarial learning process order ensure inter-skill alignment while enabling the sub-task skills to be adaptively adjusted to task changes during fine-tuning as well, as shown in line 10-12 of Algorithm
B More Details on AES Regularization
Automatic Discount Scheduling (ADS) 9 is mechanism for allocating more appropriate reward sig- nals in s Imitation Learning from Observation (ILfO), based on the concept of Optimal Transport [51] and further introducing the characteristic of process dependency across tasks. Based on this, ADS focuses on adjusting the discount factor during reinforcement learning training in ILfO. Following the mechanism in ADS, our AES also employs an imitation progress recognizer to monitor the extent to which the agent has assimilated the expert's behaviors. The main idea is to assess the closeness of the pair of trajectories by evaluating the agent-collected trajectory T = (sD....,8T) and the expert trajectory , = {s ..) e) through monotonic state-by-state alignment.
15

---

Algorithm 2 Bi-directional Adversarial Learning
Key differences T-STAR [S] in gray.
1: Require: expert demonstrations Df.... ...... Dr tab-task MDPs Mi....,MMx pre-trained sub- task policies T2. ....... F pre-trained GAIL discriminator for ....... fø.
2: Initialize bi-directional discriminator Cu:.... cK imitation identifier & with D balance dis- count factor ÀRI. +- a, ÅnI. +- 1
3: Initialize initial state buffers , B and terminal state buffers
4: for iteration m 1, 1..... ... .M do
5: for each sub-task i ...... K do
6: Sample so from environment or 3 B-11
7: Rollout trajectories = (31, ....... ST) with pre-trained m is successful then B₁ + Bj U 81, B t B ST
8:
9:
ADAPTIVE EQUILIBRIUM SCHEDULING
11:
Update imitation identifier & with т Query about the current imitation progress p
13:
end if
14:
Update balance discount factor ARL, An. + (2(P) Fine-tune l with T and E ~ TDf
15:
16:
I/ TRAIN RI-DIRECTIONAL DISCRIMINATOR
17:
Update t with 8g --1 and 85 ~ Bi with L,(w) = C1 C2
18:
FINE-TUNE WITH DUAL REGULARIZATION
19:
Update T9 with (st, ag, $1+1; ø, w) using Eq.
20:
end for
21:
end for
Agent Traj
Traj
(1,233,455
(1,3,2,5,4)
Expert
Expert
Traj
(a)
(b)
Figure 7: Visualization of the construction of the sequence Q. To be more intuitive, we directly represent the minimum cosine similarity with double arrows.
To be specific, & receives the agent's collected trajectories (line 12 in Algorithm 1 and infers the agent's current imitation progress p, p [0, 7)) (line in Algorithm 1b. The construction of Φ with reference to ADS, first requires the construction of sequence Qq ..n))....)).... where q. = arrgmiin c(s,, s5) is the index of the nearest neighbor of s; in F e is the cosine similarity. As shown in Fig. 7 If T and nF are exactly the same, then Q becomes strictly increasing sequence and 7 characterize some different behaviors, there are some unordered sequences in Q ((b))
(Fig ((a) On the contrary,
After constructing Q, the progress alignment between and 7 , is measured the length of the longest increasing subsequence (LIS) in Q, denoted as LTS(TT FB). For instance, Q (1,3,2, 4, in Fig 7(b) then its LIS can be {1, 5), (1,2,5), {1,3, 4] or (1,2, The LIS measurement concentrates on the consistency of the macroscopic trends in these trajectories, thereby preventing overfitting the microscopic features in the observation 9
Further, if the following inequality Eq. g holds, this indicates that at this time step, the agent's imitation of the expert's 's action is equivalent to the level of the expert's 's performance, then the agent's imitation progress will increase by
max I. F
+E # 2 px
min #e F
LIS
1E "1.p+1)
(8)
16

---

1.9
=1 exp( -
exp(- - Threshold p
9.(0) expl
0.5
0.4
20 40
50
100
120 140
p
Figure 8: Visualization of the mapping function (p(2). In this example, we assume that T = 150.
where f * f.. the subscript 1 : + denotes the first + 1 steps of the extracted trajectory, and p E [0, controls the stringency of the imitation progress monitoring.
After obtaining the current imitation progress of the agent, AES then adopts a mapping function (p) schedule the two new balance discount factors ARL and ÅIL. Straightforward idea of setting
(p) is that y reaches a certain threshold, i.e., the agent is able to imitate the expert's behavior well, then the more the agent tends to imitate the expert's behavior in subsequent training, and
vice versa. Therefore, we set the threshold as T. If p E [0, )) we propose (p) = 1-el
el (-xx;;;
2
E 1 T), propose sx mapping function. The mapping function shown in Fig. 8 where 150. In our experiments, we use different flatten factors for the two stages, where kl 10 and 1:2 30.
where k1 and 1: are used to flatten the curve of the
Then dRL and dy are scheduled be
ARL -a--cl tt)
AL 1 a--c(-a)))
if E (0, F)
(9)
AyL a²
ARL = - a
if E (,)
As shown by the trend of function (x()) in Fig 9 when p [0, 7), the scale of ARa:
al-el +) is scheduled be larger than An.
Plats of for Offferent Values o'
1 1- l 1-el f but this gap gets smaller and smaller as p gets larger. When p T).
0.5
the scale of ALL a
is scheduled to be
larger than ARL: 1 -
while the scale
of ÅyL increases as the agent imitates better.
Thus, if p is larger and reaches a threshold step, i.e., the agent is able to imitate the ex- pert's behavior well, then the more the agent tends to imitate the expert's behavior sub- sequent training, and vice versa. The entire process is adaptively scheduled based on pe- riodic monitoring of the agent's 's imitation process. Consequently, the RL and IL components of
Figure 9: (x(x)) based on the variation of different sizes in (x(p) E [0, 1]. We use 0.5 as the base in our experiments.
17

---

tas- skill learning can be adaptively scheduled and regularized through AES, effectively enhanc- ing intra-skill dependencies between sequential actions.
C Sub-task Skills
In our simulation experiments, we use sequences of sub-tasks defined internally by the environ- ment 44 45 as task decomposition sub-tasks. Here list these sequential skills to emphasize the difficulty of long-horizon tasks. Each skill takes a 3D position as the input 9.-
IKEA Furniture Assembly:
Chair_agne (2 sub-task skills): Assemble stool leg target position g Assemble stool leg 1 to target position g.
Chair bernhard (2 sub-task skills): Assemble support leg 0 to target position gf Assemble support leg to target position 9!
Table_ dockserokktraaeace (3 sub-task skills): Assemble table leg 0 to target position df - Assemble table leg to target position g + Assemble table top to target position g:
Chair ingolf (4 sub-task skills): Assemble chair support O to target position g + Assemble chair support target position 9¹ Assemble front leg 0 target position g Assemble front leg 1
to target position
Table lack (4 sub-task skills): Assemble table leg 0 to target position oft - Assemble table leg 1 to target position g. > Assemble table leg to target position g² Assemble table leg 3 to target position g.
(4 sub-task skills): Assemble table leg insert to target position gf + Assemble table leg I insert to target position nd - Assemble table leg 2 insert to target position g - Assemble table leg 3 insert to target position g
Kitchen Organization:
Kitchen (4 sub-task skills): Turn on the microwave to target position df - Move the kettle to target position g! + Turn on the stove (rotate the stove button to target position g²) -> Turn on the light (rotate the light button to target position g.)
Extended Kitchen (5 sub-task skills): Turn on the microwave to target position 9f + Turn on stove (rotate the stove button to target position g Turn on the light (rotate the light button to target position gÊ) > Slide the cabinet to the right target position g - Open the cabinet to target position g
D More Quantitative Results
We present the training curves with different skill learning methods for sub-task skills in chair_ingol, task, and we further present the evaluation performance of the pre-trained skills with different methods across sub-tasks in the other 6 long- morioooi simulation tasks. Also, we the algorithms trained from scratch in the presence of perturbations to further illustrate the importance of the execution of sub-tasks on long-horizon tasks.
Additionally, the main paper does not delve into the loss function (w) concerning the different scales of the bi-directional constraints in bi-directional adversarial training. Therefore, conduct further ablation experiments to examine the impact of different scales of the two constraints in the bi-directional discriminator.
D.1 Sub-task Skill Learning Performance
D.1.1 Training performance
Fig. 10 shows the sub-task skill training curves IKEA furniture assembly tasks. All methods are trained in each sub-task with 5 random seeds, 15M environment steps. As can be seen, the sub-task skill training based on PPO (learning only from environmental feedhack), GAIL. (learning only from expert demonstrations) and Fixed-RL- RL-IL (learning from fixed scale of environmental feedback
18

---

Marderd Phoiiin
chair_agne
Adnativ 26B Learnng
Fsce-/UL4
(b) chair bernhard
Sub-bas
51b- 2
Subrcadk
(c) table dockstra
of Taare
Sumber of Thanirg
Sumberef
number Tarigg
Mieriie learnig
(d) table_lace
Sub-ask
Learnn
(c) toy_table
Sab-oock
Subtikk
Sub-aak
(1 chair ingolf
Figure 10: Training curves for sub-task skills in IKEA furniture assembly tasks. The y-axis represents the success rate of the sub-task.
19

---

Success rate (%)
Success rate (%)
80
50
60
20
20
Sub-tasks chair agne
Sub-tasks chair bernhard
GALL
Rood RL
Aduptre skill Learnin (scal)
PP
GNL
Read ALIL
Asapoive Skil Learring (SCaR)
(a) chair_agne agne
(b) chair bernhad
100
100
Success rate (%)
80
Success rate (9)
40
20
sub tasc
Sub-task2
Sub bab- 3
sub tast
Sub-tasks table lack
Sub-tasks in toy. table
PPO
ONL
Foodd ALA.
Adoptive Skil Learnng 1SCaR)
(c) table_lack
(d) toy_tablc
Success (%)
30
Success rate (96
800
20
20
Sub-tasks table dockstra
Sub-tasks in Kitchen
Fowd Adaptive Sell Learning (Scan)
GAIL
Rand-R--L
Ateegiic skill Learring [SCAR]
(e) table dockstra
(I) Kitchen
Figure 11: Evaluation Performance Comparison of Sub-task Skill Learning.
and expert demonstration) cannot maintain stability and exhibits significant training performance degradation as the sub-task stage increases. contrast, sub-task skill training process using our proposed adaptive sub-skill learning scheme has always been relatively stable and better performing.
D.1.2 More evaluation performance
As shown in Fig. [T in shair_agne, chair bernhar, table laccl_lcctto__oo table dockstra, and Kitchen tasks, even with the increase of objects in the environment and the increase of unpredictable perturbations our proposed adaptive skill learning learns better sub-task skills. In contrast, the PPO, GAIL, and Fixed-RL-IL baselines fail to maintain well-leaming sub-task skills.
These results further corroborate that proposed AES regularization reinforce inter-step dependencies to the sequential actions within each sub-task skill, and thus pre-train hetter sub-task skills for long-horizon tasks.
D.2 Robustness to Perturbations
test the algorithms trained from scratch in the presence of perturbations. As shown in Table 3 algorithms trained from scratch fail to successfully complete the task when environment perturbations occur during execution. This further illustrates the importance of dividing sub-tasks for multi-stage exccution on Long-horizon manipulation tasks that contact-rich and subject to unanticipated
20

---

Table 3:
Success rates of completing the two sub-tasks chair_ bernhard and four sub-tasks chair _ingolf in stationary and perturbed environments.
bernhad
chair_inged
Method
No Perturb
Perturb
No Perturb
Perturb
PPO (Seratch RL)
0.42 0.12
0.01 0.00
0.14 1 0.03
0.00 0:000
GAIL (Scratch IL)
0.23 0.0
0.00 0,00
0.00 0.00
0.00 0:000
Fixed-RL-IL
0.53 i 0.07
0.05
= 0.00
0.22 = 0.00
0.00 0:00
SkiMo
0.62 ous
0.10
0.47
0.00 0.00
Policy Sequencing
0.82 +: 0.09
0.51 = 0.04
0.77 = 0.12
0.50 + 0.10
T-STAR
0.90 + 0.44
0.60 0.08
0.89 0.04
0.59
: 0.00
SCaR w/o Bi
0.92 + 0.2
0.65 =
0.91 0.01
0.63
+ 0.05
SCaR w/o AES
0.94 + 0.33
0.74 = 0.00
0.93 0.02
0.71
+ 0.07
SCaR (Ours)
0.96 + 0.04
0.85
0.95
0.80 + 0.33
Effects of scales of dual state adversarial training items
90-
Success rate (%)
85-
diair Lann char ingelf bble dodestre
85
01
63
0.9
Base exponent
Figure 12: Impact on skill chaining performance of different scales of bi-directional constraints in SCaR.
perturbations, It also supports the significance of our work long-horizon robotic manipulation tasks.
D.3 Further Ablation
We set the loss function for the bi-directional discriminator in the main paper as Li(w) = ]] + C2.. where the bi-directional constraints C1,C₂ are defined as:
next initial +
previous terminal: C1 Esz.r, (zz) 112 + Eer (S)
(10)
previous terminal next initial: =2= (8T) 1)2 Bez 1S (sT))
The first constraint C trains the policy to have the initial states approach the terminal states of the previous policy, while the second constraint C2 trains the policy to have the terminal states close to the initial states of the next policy. In the experiments, these two constraints have the same scale in the training process of the bi-directional discriminator.
We wonder whether different scales of these two terms would lead to different performances, and for this reason, we conduct further parametric ablation experiments to explore this. Specifically, we define the scale parameter of the first term C₁ as d1, and the second term C2 as d2 and set 0.1, 0.3, 0.5, 0.7, 0.9 for dj respectively for comparison experiments. We test the effect of different scales of bi-directional adversarial training items di and d2 on the success rate SCaR in each of the four tasks: chair_ _agne, chair chuir_ingolf, mable_dockstru and extend kitchen. As shown in Fig. 12 the experimental result also line with our intuition that when the ratio of the two terms initial previous terminal and terminal + next initial is the same, the performance is the best
d1.
21

---

among the four tasks, whereas when the more imbalanced the scale of the two terms is, the worse the performance is.
This ablation result further demonstrate our statement in Sec. 4.3 in the main paper: The purpose of the bi-directional discriminator is to establish balanced mapping relationship between the initial states and terminal states to ensure the coherence and stability of the policy. If the constraint in one direction (e.g.. from initial states terminal states) is stronger than the constraint in the other direction (e.g., from terminal states to initial states), the information transmission becomes asymmetrie. This asymmetry results in better training in one direction and insufficient training the other, thereby affecting overall performance.
D.4 Impact of Different Sub-task Divisions
To explore the impact of different sub-task divisions, we conduct more experimental validation using the chair_ _iiggoff task. The original sub-tasks in this task are divided as follows: "Assemble chair support 0 to target position" "Assemble chair support target position" "Assemble front leg 0 to target position" + "Assemble front leg to target position". " We have re-divided the sub-tasks into two alternative settings:
"Assemble chair support O and chair support to target positions" + "Assemble front leg 0 to target position" + "Assemble front leg to target position".
"Assemble chair support 0 to target position" + "Assemble chair support target position" "Assemble front leg 0 and leg 1 to target positions".
Table 4: The impact of different sub-task divisions on SCaR performance.
chair jingol (setup 1)
chair iitgllnngfff (setup 2)
SCaR
0.68
0.74
is worth noting that, since the re-division of the sub-tasks results in only three sub-tasks, we set 90% the success metric for all three sub-tasks being successfully executed. As seen in Table 4 compared to SCaR's 's success rate of about 95% with the original four sub-task divisions, the success rate for completing the first sub-task and then executing the remaining two sub-tasks is significantly reduced. This decrease is due to the increased difficulty of the first sub-task in setup (which requires assembling both chair support) and the last sub-task in setup 2. These changes result in lower overall success rate for the task. This result suggests that a reasonable division of sub-tasks in long-horizon tasks is crucial for the success rate of overall task completion.
D.5 Impact of the Number of Sub-tasks
To explore the performance of skill-chaining methods as the number of sub-tasks increases, we add a sub-task to the Extended Kitchen task to evaluate SCaR's performance in manipulation tasks with longer horizons, involving sub-tasks. The modified task, Longer Extended Kitchen includes: 1. Turn on the mierowave; 2. Turn the stove; 3. Turn the light; 4. Slide the cabinet to the right target position; 5. Open the cabinet to the target position; 6. Move the kettle to the target position.
Table 5: Performance Comparison on Longer Extended Kitchen Task.
Method Longer Extended Kitchen Task
T-STAR SCaR
0.33
0.61
As shown in T the addition of an extra sub-task increases the complexity and difficulty of skill chaining in long-horizon tasks. Nonetheless, SCaR achieves a significantly higher overall task execution success rate, surpassing T-STAR by 28%. Although there is still ample room for improvement, we believe our approach establishes strong baseline for future research on skill- chaining methods for long- horizon manipulation tasks.
22

---

(a) SCaR Successful
(b) T-STAR FFiled
Figure 13: Qualitative results of successful skill chaining performance with SCaR and failed skill
chaining performance with T-STAR, More qualitative results can be found on our project website http:///iiieesgoogll.ocooooo///a
23

---

E
More Qualitative Results
Fig 13 shows the qualitative comparison of skill chaining methods. Their animated versions can be found on our project website.
F
Real-world Validation via Sim-to-Real Transfer
Table 6: Skill chaining performance of real-world long-horizon robotic manipulation tasks.
Method
Success rate
T-STAR
70% sub-tasks) / 50% (3 sub-tasks)
SCaR 90% (2 sub-tasks) 70% (3 sub-tasks)
Real-robot Experiment Setup We also evaluate the skill chaining performance of real-robot for solving simple yet intuitive real-world long-horizon manipulation. We set up two types of desktop- level long-horizon manipulation tasks. The robotic arm needs to pick-and-place 2 and 3 blue squares in sequence.
We built the corresponding task enviromment using the gazebo simulation that accompanies the Kl robo and collect 50 demonstrations of grasping skills for each square for training. With camera calibration,
deploy agents trained under simulation in real robot desktop task to solve 2-square as well as 3-square pick-and-place tasks without the need for adaptation processes. We conduct experiments with the Sagittarius K1 and use Movelt2 library based ROS framework for controlling the arm. We use RGB observations from RealSense D435i camera on the wrist of the robotic arm.
Figure 14: Visualization of the successful skill chaining in the 2-blue Salue-square pick-and pick-and-place tasks using SCaR.
Figure 15: Visualization of the successful skill chaining in the square aeck-and-place tasks using SCaR.
For evaluation, measure the success rate across 10 randomized square positions for each task. As shown in Table. 6 SCaR can solve the two long-horizon tasks and outperforms T-STAR baseline. Fig. 14 and Fig. 15 show the qualitative results of successful skill chaining in the 2 and 3-blue-square pick-and-place tasks using SCaR.
Results
http::///gaabbboo/////////ttataa

---

chair agne
agnir_agne chair bernhard
chair ingolf
table lack
toy_table
table_ dlocstaa
Figure 16: IKEA Furniture Assembly Environment for Long- Complex Manipulation Tasks.
G Environment Details
G.1 IKEA Furniture Assembly
We choose six tasks, chair_agre ble dockstra from the IKEA furniture assembly environment [44] as the focal points of our experi- ments, as shown in Fig. 17 Our chosen robotic platform is the 7-DoF Rethink Sawyer robot, and we control using joint velocity commands.
berrhabe chair ingolf, table_lcck, toy_table, and ta-
Observation Space The observation space comprises three key components: robot observations (29 dimensions), object observations (35 dimensions), and task phase information (8 dimensions). Robot observations encompass robot joint angles (7 dimensions), joint velocities (7 dimensions), gripper state (2 dimensions), gripper position (3 dimensions), gripper quaternion (4 dimensions), gripper velocity dimensions), and gripper angular velocity dimensions). Object observations include the positions (3 dimensions) and quaternions (4 dimensions) of all five furniture pieces in the scene. Task information, an 8-dimensional one-hot encoding, represents the current phase, including actions like reaching, grasping, lifting, moving, and aligning.
Action space The action space includes arm movement, gripper control, and the connect action, which can vary based different control modes: 6D end-effector space control using inverse Kinematics, joint velocity control, and joint torque control.
In the context of reinforcement learning (RL), we utilize heavily shaped multi Pelti-ppase dense reward obtained from the IKEA Furniture Assembly Environment 44
Environmental Reward Function The IKEA furniture assembly environmental reward function is a multi-phase reward defined with respect to a pair furniture parts to attach (e.g., a table leg and table top) and the corresponding manually annotated way-points, such target gripping point for each part. The reward function for pair furniture parts consists of eight different phases as follows:
The robot has to reconfigure its arm pose to an appropriate pose Pinit for grasping new furniture part. The reward is proportional to the negative distance between the end-effector Per and Pain-
Initial phase:
Reach phase: The robot reaches above target furniture part. The reward is proportional to the negative distance between the end-effector Par and point Preach 5 em above the gripping point
Lower phase:
The gripper is lowered onto the target part. The phase reward is proportional the negative distance between Per and the target gripping points.
Grasp phase: The robot learns to grasp the target part. The reward is given if the gripper contacts the part, and is proportional to the force exerted by the grippers.
Lift phase: The robot lifts the gripped part up to Pin- The reward is proportional to the negative distance between the gripped part ppan and the target point Pun-
Align phase: The robot roughly rotates the gripped part before moving it. The reward is proportional to the cosine similarity between up vectors u,A Up and forward vectors fA, fp of the connectors.
25

---

Kitchen
Extended Kitchen
Figure 17: Kitchen Organization Environment for Long-Horiag Complex Manipulation Tasks.
Move phase:
The robot moves and aligns the gripped part to another part. The reward is proportional to the negative distance between the connector of the gripped part and point Pmove_to 5 cm above the connector of another part, and the cosine similarity between two connector up vectors, U and ug, and forward vectors fA and fg. Note that all connectors are labeled with aligned up vectors and forward vectors.
Fine-grained move phase: The robot must finely align two connectors until attached. The reward is used as the move phase with higher coefficient, making the reward more sensitive to small changes. In addition, when the part is connectable, a reward is provided based on the activation of the connect action a [[connect].
Upon completion of each phase, completion rewards are given encourage the agent to move on to the next phase. In addition to stage-based rewards, control penalties, stabilizing wrist pose rewards, and grasping rewards (i.e., opening the grasping hand only during the initial, arrival, and lower stages) are provided throughout the process. If the robot releases the grasped object, the phase ends early and negative reward is provided. Phase completion depends on the robot and part configurations satisfying distance and angle constraints with respect to the goal configuration. After all stages are completed, stage resets to the initial stage. This process repeats until all parts are connected.
Demonstration Collection For imitation learning (IL), we gathered 200 demonstrations for each furniture part assembly using a programmatic assembly policy. Each demonstration for single-part assembly typically spans 150 steps, reflecting the overall task's ' inherently long-horizon nature.
Sub-tasks In our experiments, we define sub-task as the process of assembling one part to another. Thus, the chair_ape aanne and three distinct sub-tasks, and chair chair_ingolf. table_lace and toy_ toy_table have four distinct sub-tasks. These sub-tasks trained independently, with their initial state sampled from the environment and random noise introduced in the [-2cm, 2cm] and [-3%, 3"] ranges of the (x, y) plane. Importantly, the decomposition of the sub-tasks is pre-determined, which means that the environment initialized for each sub-task, and the agent receives a notification when a sub-task is successfully completed. Once the two components firmly connected, the corresponding sub-tast tab-tssk is considered completed and the robotic arm is guided back to its initial pose, i.c., at the center of the workspace.
tasks have two distinct sub-tasks, table suble_dockstra has
Assembly Difficulty The difficulty of modeling furniture depends largely the shape of the furniture. For example, the poy_table task with cylindrical legs is more difficult to grasp. whereas the table_lac lack task with rectangular legs is easier to grasp. Chairs are generally more difficult to assemble because of their irregular shape (e.g., seat and back). This is the reason why the success rates of the toy payte andchair ingolf tasks are lower than the success rates of table Jack.
G.2 Kitchen Organization
We use the Franka Kitchen tasks in D4RL 45 and refer to the experimental setup in SkiMo (6) for the sub-task extensions. Including the following two tasks: Kitchen task and Extended Kitchen task, as shown in Fig. I7
26

---

Kitchen The 7-DoF Franka Emika Panda robot arm is tasked with performing four sequential sub-tasks: Turn on the microwave Move the kettle Turn on the stove Turn on the lights.
Extended Kitchen The environment and task-agnostic data used in this experiment are consistent with those employed in the Kitchen scenario. However, we introduce a different set of tasks for this experiment, namely: Turn the microwave- Turn the stove Turn the lights Slide the cabinets to the right Open the cabinets, as depicted in Fig. [7 (right). It's worth noting that this sequence of tasks is not aligned with the sub-task transition probabilities observed in the task-agnostic data, posing a challenge for exploration based on prior data.
Observation Space The agent operates within 30-dimensional observation space, which includes an 1 --ddmenniionl robot proprioceptive state and 19-dimensional object states. This modified observation space removes constant c0-dimensional goal state found in the original environment.
Action Space The agent's action space consists of 9 dimensions, encompassing 7-dimensional joint velocity control and 2-dimensional gripper velocity control.
Environmental Reward Function In terms of the environmental rewards, the agent receives reward of +1 for each completed sub-task. The total episode length is set to 280 steps, and an episode concludes once all sub-tasks are successfully accomplished. The initial state initialized with slight noise introduced in each state dimension.
Demonstration Collection For imitation learning, we collect 200 demonstrations per sub-task with reference to the dataset in [52] that obtained through teleoperation. This dataset covers interactions with all seven manipulatable objects within the environment.
H Network Architecture
For a fair comparison, our method and the benchmark methods use the same network structure. The policy network and the critic network consist of two layers of 128 and 256 hidden units fully connected with ReLU nonlinear properties, respectively. The output layer the actor network outputs an action distribution, which consists of the mean and standard deviation of Gaussian distribution. The critic network outputs only one critic value. The discriminator of GAIL [39 and the bi-directional discriminator of our proposed approach a two-layer fully connected network with 256 hidden units. The outputs these discriminators are clipped between [0, 1], following the least-square GAIL, proposed by [40]
I Training Details
L1 Computing Resources
Our method and all baselines were implemented using PoTch [33] All experiments were conducted on workstations equipped with Intel(R) Xeon(R) Gold 5218 CPUs and dual NVIDIA GeForce RTX 3080 GPUs In the SCaR framework, the pre-training phase for each sub-task skill policy, spanning 150M time steps, took approximately 10 hours with dual regularization, compared to about 8 hours without it (Fixed-RL-IL),, leading to an added computational overhead of roughly 2 hours. The full testing and evaluation process of skill chaining for complete long-horizon task required an additional 10 to 15 hours, depending on the task's 's complexity. For comparison, training the skill dynamics model in SkiMo [46] took approximately 24 hours (100M steps). Training baselines such as PPO (7). GAIL [3]. and Fixed-RL-IL took longer, requiring about 48 hours each, as these methods train the entire cong-horizon task from scratch, with 450M time steps for each complete task. In our evaluation, we used 5 seeds, each tested over 200 episodes, resulting in an average real-time execution time of about 36-54 seconds per single long-horizon task.
I.2 Algorithm Implementation Details
We report the hyperparameters used in our experiments in Table 7
27

---

Table 7: Hyperparameters used in our experiments.
Hyperparameter
Value
Rollout Size
1024
Learning Rate
0.0003
Learning Rate Decay
Linear decay
Mini-batch Size
128
Discount Factor
0.99
Entropy Coefficient
0.003
Reward Scale
0.05
State Normalization
True
Discriminator learning rate
le-4
Sub-task training steps
150000000
# Workers
20
Epochs per Update
10
Base exponent for balancing a
0.5
k (used flatten the mapping function during p E [0,
2
l (used flatten the mapping function during p T))
Weighting factor Ani
10000
p (for imitation progress recognizer +)
0.9
Penalty coefficient 178P
10
Table 8: Comparison to prior work and ablated methods.
Method
Model-based
Skill-based
Scratch training
Joint training
PPO 47
and GAIL (39
Fixed-RL-II 40
SkiMo 46
Policy Sequencing
X
T-STAR [
X
SCaR (Ours) and SCaR w/o Bi and SCaR w/o AES
X
For the baseline implementations, we use the official code for PPO 47 GAIL 33
Fixed-RL-IL. 40
SkiMo [46), Policy Sequencing [12] and T-STAR (15). The table below (Table compares key components of SCaR with model-based, model-free and skill-based basclines and ablated methods, where joint training indicates whether or not reinforcement learning combined with imitation learning is used for training.
PPO [47]
Any reinforcement learning algorithm can be used for policy optimization, in this paper we choose to use Proximal Policy Optimization (PPO) and use the default hyperparameters of PPO 47
In this paper we choose to use Generative Adversarial Imitation Learning (GAIL) as the learning algorithm for imitation learning and use the default hyperparameters of GAIL (39 We specifically use an agent states s to discriminate agent and expert trajectories, instead state-action pairs (s, a).
GAIL 39
Fixed-RL-IL [12] We adopt the AMP 40 solution combining environmental rewards and least square GAIL with \RL. = Au 0.5. For implementation details of least square GAIL training and GAIL rewards, see original paper 40
SkiMo [46] We use the official implementation of the original paper and use the hyperparameters suggested in the official implementation.
Policy Sequencing
employ the official implementation and the hyperparameters provided
by [15).
28

---

T-STAR 15
We use the official implementation of the original paper and use the hyperparameters suggested in the official implementation [5].
SCaR (ours) We refer to T-STAR and use ARE = 10000 for bi-directional regularization. We take 50% of the initial state samples from the start environment of each policy, 50% of the terminal state samples at the end, and 50% of the initial state buffer and 50% of the terminal state buffer from the previous skill, respectively.
Limitations and Potential Solutions
While SCaR demonstrates strong capabilities, it does have certain limitations. In the following sections, we outline these limitations and propose potential solutions.
Limited Task Generalization SCaR primarily focuses on predefined, structured environments to validate mechanism for chaining pre-trained sub-task skills within long-horizon tasks. Con- sequently, our current study does not address SCaR's ' adaptability across varied or novel task en- vironments. While we demonstrate SCaR's robustness to unknown perturbations within a single environment (e.g.. unexpected forces applied to the robot arm joints in specific sub-tasks), the system's s ability to generalize to entirely new or unfamiliar environments remains unexplored. A potential solution is leverage foundational models to expand SCaR's s applicability. In Long-horizon tasks where sub-task definitions are unclear or missing, foundational models can use their powerful task-planning capabilities to divide tasks into logical sub-tas tub-tasks [54 55 56]. When unexpected sub- task demands or changes in overall task goals arise, these models can re-plan sub-tasks accordingly. Another approach involves buman-in-the-lopp learning, incorporating human guidance in the training pipeline, such as using human priors for sub-task division (32 or employing methods to manage the subjectivity of human-labeled rewards through preference learning [57
Extensive Retraining for New Tasks SCaR faces limitations in adapting to diverse manipulation tasks different robotic setups, as it requires extensive retraining of sub-task skills for each new task. This reliance on full retraining restricts SCaR's 's efficiency and scalability when addressing evolving or variable task requirements, particularly in complex, real-world environments. A potential solution is to integrate online reinforcement learning into SCaR's 's skill-leaning process, allowing it to adapt to task variations, such different table designs, without full retraining, enabling more efficient adaptation.
Lack of visual input handling SCaR currently lacks the capability to process encoded image and semantic state inputs, limiting its applicability in tasks that rely on visual or semantic information for effective performance. The current setup largely dependent on environments with direct access to state information, as both simulated environments are based on state representations. While effective in controlled setups, enabling SCaR to learn from image encodings as states would enhance its robustness and applicability in tasks requiring nuanced visual and semantic processing. Given recent advancements in learning multi-view representations [58 59 and generating 3D models [60] [1], incorporating these improvements and 3D priors into the encoder design could enhance the sample efficiency of SCaR.
K
Potential negative impacts
Since method is currently limited to applications in simulated environments and simple desktop- level robot manipulation, it is not expected to have a significant negative impact on society. However, privacy concerns may arise if our method is applied to real-world long time-series tasks with mobility, as imitation learning agents used in applications such as autonomous driving [62] or real-time
control [63] 64 require large amounts of data that often contain controversial information. In addition, the imitation learning policy is a challenge because it imitates a specified demonstration that may include bad behavior. the expert demonstration includes some nefarious behaviors (e.g., training data for mobile manipulation task includes behaviors that may be violent towards pedestrians), then the policy may have significant negative impact on the user. To address this issue, future directions should focus on developing agents with safety adaptations in addition to improving performance.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose new skill chaining framework for long time-series robotic manipulation tasks that improves overall task completion performance by providing dual regularization for intra- and inter-skill dependencies. We hope this work will inspire future research to further explore the potential of skill chaining for long-horizon robotic manipulation.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification: We discuss limitations in the last section of the main paper: limitations mainly exist in that 1) the sub-tasks our framework are predefined, 2) we did not test our method more challenging real robot furniture assembly task due to limited hardware.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
30

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide further explanation in the Appendix to explain the assumptions presented in the main paper.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We further describe the network architecture, training details, dataset, and the open source codebase on which the method is based in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
31

---

Our experiments perform means and standard deviations for the five seed In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [NA]
Justification: All simulation environments, datasets, and open-source code libraries used to implement our method are described in the Appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guides/ mubllc//guildee/Dodeuuaaaaaassi
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.ccppubii.. wttpcc/ppppppppu..eeeeeeeooopppp guwdee/deeeeee CodeSuumi ssioiooicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
We describe partial details in the main paper and provide further details in the Appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
results.
Guidelines:
The answer NA means that the paper does not include experiments.
32

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We illustrate the computational resources and the time required for the experi- ments in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////...eepe.cppo//olllccttt
Answer: [Yes]
Justification: Yes, the research conducted in the paper conforms in every respect with the
NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: We elaborate these in the final section of the Appendix.
33

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper does not release data or models with high risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We have cited provided URLs to all the code, data, and models used in the paper.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
34

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not do crowdsourcing experiments and research on human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
35

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "NWvNSKnInF", "reviewer": "Reviewer_9LLM", "summary": "The paper introduces the Skill Chaining via Dual Regularization (SCaR) framework, designed to enhance skill chaining in long-horizon robotic manipulation tasks by applying dual regularization techniques during skill learning and chaining. The framework addresses the error accumulation issue in traditional skill chaining by focusing on both intra-skill and inter-skill dependencies, ensuring a more reliable execution over complex tasks like IKEA furniture assembly and kitchen organization. The effectiveness of SCaR is demonstrated through higher success rates and robustness in simulation and real-world tests compared to existing methods.", "questions": "1. How does SCaR handle scenarios where sub-tasks are not clearly defined or when unexpected sub-tasks arise due to changes in the environment?\n2. Can you elaborate on the computational overhead introduced by the dual regularization process and its impact on real-time task execution?\n3. Is there a strategy within SCaR to simplify the fine-tuning process, especially in adapting to different robots or task specifics without extensive retraining?", "limitations": "Although SCaR shows improvements in task execution, the scalability of this approach to even longer-horizon tasks involving more complex interactions and environmental variability remains less explored.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Innovative Dual Regularization**: The dual regularization approach is a significant innovation that stabilizes the execution of chained skills by balancing intra-skill coherence and inter-skill alignment, addressing a common shortfall in existing methods.\n **Comprehensive Evaluation**: The paper provides extensive experimental results, including simulations and real-world tasks, which thoroughly demonstrate the effectiveness of SCaR compared to baseline methods.\n **Practical Impact**: The application to real-world tasks like furniture assembly and kitchen organization showcases the practical relevance and potential of SCaR in improving robotic automation efficiency and reliability.", "weaknesses": "**Limited Task Generalization Discussion**: The paper could expand on how SCaR adapts to varying task environments or tasks with different complexity levels, as most examples are confined to pre-defined scenarios with structured environments.\n **Dependency on Accurate Sub-task Definition**: The success of SCaR heavily relies on the precise definition and segmentation of sub-tasks, which might not be straightforward in more dynamic or less structured environments.  The paper mentions that the sub-task division is predefined and does not involve visual and semantic processing of objects. This could limit the framework's applicability in more complex, real-world scenarios where tasks might not be easily decomposable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank  you for acknowledging the strength of our work, e.g., the innovative dual regularization, comprehensive evaluation and practical impact. We address the comments and questions below.\n\n**Regarding the discussion of limited task generalizaiton.**\n\n* Thank you for pointing this out. The SCaR experimental setup in the original paper focused primarily on predefined structured environments. We chose these scenarios to validate the core mechanism and performance of SCaR, i.e., that it requires pre-training of predefined sub-task skills in a long-horizon task, and then stringing these skills together to perform the overall task. So the original paper does not discuss SCaR's ability to adapt to different task environments.\n\n  We have experimented and discussed the ability of SCaR to adapt in the face of unknown perturbations in the same environment. In Section 5.3 of the original paper, we tested the robustness of SCaR to perform a long line-of-sight task in the face of an unknown perturbation in that environment by adding an unexpected force to the robot arm joints during the execution phase of the sub-tasks of chair_bernhard and chair_ingolf. The experimental results show that our SCaR is robust to such perturbations compared to the comparison methods. More detailed experimental results are presented in Appendix D.2. We will discuss more about the limitations of SCaR's adaptability to different task environments in the next version of the paper in response to the reviewers' comments.\n\n**Regarding the dependency on accurate sub-task definition.**\n\n* Thank you for pointing this out. The current performance of SCaR does depend heavily on the rationality of sub-task division in long-horizon tasks, and we have done a simple experiment to illustrate this in our response to reviewer ysj9's question 1. Since the core innovation mechanism of our SCaR is mainly concerned with the learning of sub-task skills and skill chaining in long-horizon tasks, the sub-tasks of the long-horizon task in the experimental setting used in the current version of the paper are artificially divided and do not involve the model's ability to process visual and semantic objects.\n\n  We are actively exploring how to divide long-horizon tasks into sub-tasks without human intervention using large language models, and how to integrate visual and semantic processing techniques to extract key frames in the task to achieve a more rational division of sub-tasks that is more adaptable to real-world long-horizon tasks. We appreciate your insights and will discuss these future directions in the revised paper.\n\n**Regarding Question 1**. \n\n* Thank you for your question. Currently, SCaR's ability to optimize long-horizon task skill chaining is in the context of predefined sub-tasks, and we agree that enabling SCaR to handle scenarios where sub-tasks are not clearly defined or where unanticipated sub-tasks arise is an important direction to extend the boundaries of its capabilities.\n\n  We currently plan to utilize the capability of large models to extend the capability boundary of SCaR. Specifically, when facing long-horizon task scenarios with unclear or no sub-task definitions, we are trying to utilize the powerful task planning capability of the large model to divide reasonable sub-tasks for long-horizon tasks. When confronted with unexpected sub-tasks in the environment (e.g., the overall task goal is found to change), we are planning to utilize the large model for re-planning the sub-tasks. In this way, we will then train sub-task skills and realize skill chaining for long-horizon tasks through SCaR to extend its ability to handle various types of task scenarios.\n\n  We appreciate your feedback and will discuss these future directions in a revised paper to better elucidate this important aspect.\n\n**Regarding Question 2**. \n\n* Thank you for your question regarding computational overhead. We have actually reported the associated computational overhead and resource consumption in Appendix I.2 of the original paper.\n  The dual regularization process mainly increases the computational load of the offline pre-training phase of the skills, which takes about 10 hours per skill, while the pre-training of the skills without dual regularization (fixed-RL-IL) takes roughly more than 8 hours. The added computational overhead of the dual regularization process is within 2 hours.\n  And the process of SCaR chaining pre-trained skills (i.e., testing and evaluation) takes 10 to 15 hours, depending on the difficulty of the task. In the evaluation phase, we evaluated 5 seeds, each tested over 200 episodes, which translates to a real-time execution time of about 36-54 seconds for a single long-horizon task.\n  We will describe the computational overhead of each method at each stage in more detail in the updated version of the paper, and promise to make the code publicly available for further validation of computational efficiency after acceptance.\n\n**Regarding Question 3**. \n\n* Thank you for your question. To enable SCaR to adapt to different long-horizon manipulation tasks or robots (such as assembling a four-legged table, assembling a chair with three components, or organizing a kitchen), it indeed requires retraining sub-task skills.\n When SCaR needs to adapt to different scenarios within the same task distribution, such as assembling a four-legged table where only the table’s design changes, we believe that integrating techniques like online learning into the skill learning process can help. By using online gradient descent to gradually update model parameters, we can fine-tune the skills for assembling tables of different designs without extensive retraining, allowing for rapid task adaptation. \nThank you again for your insightful comments. We will incorporate this discussion and follow-up work plan into the updated paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mixLsSmXio", "reviewer": "Reviewer_uZgV", "summary": "The paper proposes regularization strategies to enhance inter-skill and intra-skill consistency in skill chaining. Skill chaining is the problem of learning a sequence of sub-task policies chained together to achieve the goal, given expert demonstrations and rewards. The sub-task partition is defined manually before learning.\n\nThe paper builds its skill chaining framework on T-STAR [1]. Specifically:\n- To regularize individual skill learning, the authors use Adaptive Equilibrium Scheduling (AES) strategy to adaptively balance the IL and RL objectives. The strategy is adapted from [2].\n- To enhance inter-skill consistence, the authors extend the uni-directional reularization in [1] to bi-directional to push the initial set of successor skill and the termination set of previous skill close to each other.\n\nThe proposed method is evaluated on IKEA furniture assembly and Franka Kitchen benchmarks. It demonstrates improved performance against several skill chaining and RL / IL baselines, including T-STAR [1]. The paper further ablates on the two regularization strategies to demonstrate their necessity.\n\n[1] Lee, Youngwoon, et al. \"Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization.\" 5th Annual Conference on Robot Learning.\n\n[2] Liu, Yuyang, et al. \"Imitation Learning from Observation with Automatic Discount Scheduling.\" The Twelfth International Conference on Learning Representations.", "questions": "- I'm not sure how the bi-directional regularization works. Could you clarify on what are the classifiers trained for, and how they are trained (in equation 6)?", "limitations": "See Weakness", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Skill chaining is an important problem for robots to execute long-horizon tasks.\n- The proposed regularization strategies are intuitive from a high-level.\n- The experiments demonstrate the performance gain introduced by the proposed regularization strategies.", "weaknesses": "- While the skill chaining method outperforms baselines in experiments, it is merely a technical integration of existing methods.\n  - The overall framework is built heavily on T-STAR [1]. It follows T-STAR exactly to formulate skill chaining as individual skill pretraining and joint fine-tuning, tackling skill pretraining with a combination of RL loss and GAIL loss, and aligning adjacent skills by training a discriminator.\n  - The proposed AES strategy is adopted from [2].\n  - The proposed bi-directional inter-skill regularization is merely a naive extension of the uni-directional one in [1].\n\n- The explanation of proposed AES and bi-directional regularization doesn't seem to be clear enough for readers to understand without reading the related papers [1, 2].\n  - In particular, I'm not sure how the bi-directional regularization works exactly. The key equation (6) of bi-directional regularization seems to be problematic - it learns a single classifier to classify both the initial set and termination set of the same skill.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback and thank you for acknowledging the importance of the skill chaining, the high level of our proposed regularization strategies, and our experiments! We address your comments and questions below.\n\n**Regarding the reliance on existing methods:**\n- We appreciate the concerns raised by the reviewer but cannot fully agree with them. Our work goes beyond mere integration and introduces significant innovations and improvements. Below are the differences and innovations between SCaR and related methods:\n  - **Differences with T-STAR:** Our work focuses on skill-chaining approaches for long-horizon manipulation tasks. The basic premise is that sub-task skills for each phase of a long-horizon task must be pre-trained and then integrated, as discussed in the original paper(L27-29) and related work [1,3,4,5]. While our work draws on the T-STAR framework, T-STAR focuses only on the chaining process between skills. In contrast, SCaR enhances intra- and inter-skill dependencies by introducing Adaptive Equilibrium Scheduling (AES) Regularization and Bi-directional Regularization, two simple but effective schemes. Additionally, T-STAR uses a simple combination of RL loss and standard GAIL loss for skill pre-training, whereas SCaR incorporates a gradient penalty term in GAIL loss, improving training stability (Equation (3) of the original paper).\n  - **Differences with ADS:** Our proposed AES is inspired by ADS but fundamentally differs from it. ADS focuses on adjusting the discount factor during reinforcement learning training in Imitation Learning from Observation (ILfO), specifically adjusting $\\gamma$ in $E_\\pi[\\sum_{t=1}^{T-1} \\gamma^{t-1} r_t]$, which is essentially an inverse reinforcement learning (IRL) process.  Our proposed AES extends the concept of \"discounted scheduling\" in ADS to skill learning with both RL and IL processes, focusing on the scheduling of weight factors $\\lambda_{\\text{RL}}$ and $\\lambda_{\\text{IL}}$  for RL loss and GAIL loss around the sum of the environment feedback and predicted reward weights $\\lambda_{\\text{RL}} r_i^{\\text{Env}}(s_t,a_t,s_{t+1},g) + \\lambda_{\\text{IL}} r^{\\text{Pred}}_i(s_t,a_t; \\phi)$, as described in Eq. (5) of the original paper.\n  - **Bi-directional and Uni-directional Regularization:** Bi-directional inter-skill regularization is a heuristic innovation we propose. While it may seem like a simple extension of uni-directional regularization, it effectively ensures better alignment and consistency between successive skills by simultaneously considering bi-directional constraints. Although simple, we hope our approach serves as a robust baseline for skill-chaining methods and informs future research on long-horizon manipulation tasks.\n\n**Regarding the clarity of methodology explanation.**\n* Thank you for pointing that out. We will include a more detailed  background on [1,2] in the appendix of the updated version to provide  readers with a better understanding of our methods.\n\n**Regarding the bi-directional regularization.**\n* To implement bi-directional regularization, we train a bi-directional discriminator, $\\zeta_\\omega$. We start by modeling each sub-task of the long-horizon task as a Markov Decision Process (MDP) and initialize 2-layer fully connected networks for each MDP as the initial dual discriminator $\\zeta^k_\\omega$. It is important to note that the dual discriminator is trained only after the corresponding skill policy has been pre-trained on each sub-task MDP. We initialize two buffers, $I_k$ and $\\beta_k$, for each sub-task to store the initial and termination states of the skill success trajectory, respectively.\n\n  The pre-trained skill is then used to execute the corresponding sub-task and generate the trajectory. If sub-task $k$ is successfully completed, the initial state of the current trajectory is added to $I_k$, and the termination state is added to $\\beta_k$.\n\n  Bi-directional regularization focuses on chaining at both ends, so we update the dual discriminator $\\zeta^i_\\omega$ using $\\beta_{k-1}$, $I_k$, $\\beta_k$, and $I_{k+1}$ after the trajectory of the $(k-1)$, $k$, and $(k+1)$th sub-task skills are executed. In our code implementation, we use two fully connected networks to minimize the training separately: one to make the terminal states of skill $k$ converge to the initial states of skill $k+1$, and the other to make the initial states of skill $k$ converge to the terminal states of skill $k-1$, as described in Equation (6) of the original paper. Finally, the parameters of the two networks are averaged and combined into $\\zeta^k_\\omega$.  We will further specify the different network subscripts in Eq. (6) to avoid confusion. Thank you for raising this issue.\n\n  Additionally, we have provided pseudo-code for bi-directional adversarial training in Algorithm 2 in Appendix A.2 of the original paper. We are currently refining our code to ensure clarity and ease of use. Once finalized, we commit to making it publicly accessible. We apologize for some typographical errors in the current Algorithm 2 (e.g., \"dual set discriminator\" should be \"bi-directional discriminator\"). We will correct these errors in the future update.\n\n**Regarding Question 1.**\n* Thank you for your question. Please refer to the response regarding **bi-directional regularization** for more details.\n\n[1] Lee, Youngwoon, et al.  Adversarial Skill Chaining for  Long-Horizon Robot Manipulation via Terminal State Regularization. \n\n[2] Liu, Yuyang, et al. Imitation Learning from Observation with  Automatic Discount Scheduling.\n\n[3] George Konidaris, et al.  Skill discovery in continuous reinforcement learning domains using skill chaining\n\n[4]  Lee, Youngwoon, et al.  Learning to coordinate manipulation skills via skill behavior diversification\n\n[5] Chen, Yuanpei, et al. Sequential dexterity: Chaining dexterous policies for long-horizon manipulation", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DByUkbPawS", "reviewer": "Reviewer_2GPv", "summary": "In this work, the authors present SCaR or Skill Chaining via Dual Regularization, an algorithm for both learning a set of subtasks as well as how to sequence them well. The method consists of two major components, AES regularization for learning the skills from demonstrations and real world interaction, as well as a bi-directional adversatial regularization that ensures that there is a good overlap between one skill's ending states and another skill's beginning states.\n\nThe authors experimentally evaluate the method with experiments in two simulated environments and one real world setup. In the experiments, the proposed method outperform the baselines. The authors also show ablation experiments where they remove both the AES module and the bi-directionality module, and show that they both are important components.", "questions": "- Why is the sub-task learning so slow even with demonstrations?\n- Why use this particular method of learning from sub-tasks rather than using more efficient techniques?\n- Could the discriminators be initialized by learning from the demo data?\n- Are there works that talk about the Least-Squares GAIL more? The authors presented it without a citation, but it seems like it's more well known.", "limitations": "- The work relies on having access to states, which is hard in the real world and incredibly brittle.\n- The number of training steps seems astronomically large, especially given demonstrations and for state-based environments.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The bi-directional regularization is a natural idea, and it is good to have confirmation that this works well in practice.\n+ The bi-directional objective design also makes sense, and it is good to see that the natural setup works.\n+ The experimental setup are interesting and and demonstrate the point of the authors re: the algorithm well.", "weaknesses": "- The AES component seems convoluted and overly complicated, and since it is completely unrelated to any of the skill chaining part, I am curious as to why all of the extra regularization was required.\n- This point sticks out especially sorely because there are already good IL+RL methods out there, like [1], while it seems like the authors are struggling to fit a policy to their demo data+environment. I feel like I understand why this process is so difficult especially given that this has nothing to do with sequencing, and all about inverse RL.\n- There are certain steps that seem particularly brittle, like equation 4, which is measuring progress, and it seems like it would be hard to scale to arbitrary environments with such a criteria.\n- The setup seems very much directed towards setup with access to state. The simulated environments are both based on state, and the real robot experiment is also just learned in a simulation and zero-shot deployed in the real world (happy to be corrected if I am wrong). Something that can learn in the real world with image encodings as states would be much more robust.\n- Training curves are not shown. In the appendix, the authors mention that they train sub-tasks for 150000000 (150 million!) steps, which seems practically out of the world to me as someone working on real robot. Especially given the existence of methods like [1], why are there so many steps needed to learn the sub-tasks?\n\n[1] Haldar, Siddhant, et al. \"Watch and match: Supercharging imitation with regularized optimal transport.\" Conference on Robot Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for acknowledging the strength of our work, e.g.,  the natural bi-directional regularization idea, interesting experimental setups, and effective experimental validation. We address the comments  and questions below.\n\n**Regarding the AES component.**\n- We introduced AES regularization in order to better pre-train the sub-task skills in the long-horizon task. Because our skill learning scheme combines RL and IL, this combined learning process not only allows the robot to explore the environment through the RL process to minimize possible suboptimality of learning from demonstrations alone, but also IL mitigates the over-exploration problem inherent in RL (as described in lines 44-48 of the original paper). The role of the AES, on the other hand, is to balance the RL and IL learning processes by monitoring the robot's skill learning process: if the robot struggles to imitate the expert demonstration effectively, it should focus more on self-learning from the environment. Conversely, if imitation is successful, it should continue to focus on the expert to improve the efficiency of sampling RL, as described in lines 156-158. This balance is consistent with how humans learn: if the teacher is ineffective, we learn independently; if the teacher is effective, we learn more from them. We will explain the AES regularity in more detail in an upcoming update of the paper.\n\n**Regarding the RL+IL setting.**\n- Please let us know if we have misunderstood your question. We will explain why we use the RL+IL setup and how our approach differs from the method mentioned in [1]. In the SCaR setting, the environment already includes a predefined reward function, allowing skills to be learned through agent interaction with the environment (RL) as well as learning from demonstrations (IL). In contrast, the approach in [1] requires inferring the reward function from demonstrations for IRL, which is fundamentally different from SCaR’s skill learning scheme.  We appreciate the reviewer's reminder, as we found that the reward inference process in ADS[2], which inspired our AES regularization, exploits the Optimal Transport concept from [1]. We will cite [1] and provide further discussion in the related work of future versions of this paper.\n\n**Regarding the imitation progress equation.**\n- The purpose of Equation 4 is to measure progress, primarily for use in AES regularization, to monitor the robot's learning process and to assess how well the robot imitates the expert. We have actually described this process in detail in Appendix B of the original paper. Admittedly, while we have mathematically modeled this process, it still has some limitations: this progress measure is only applicable to tasks where expert demonstrations are present in the environment and where trajectory similarity can be computed. We will discuss these limitations further in upcoming paper updates.\n\n**Regarding the state input.**\n- You are correct that SCaR does not currently handle encoded image inputs. We acknowledged this as a limitation and potential area for future development in the original paper. Our ongoing work is focused on enabling SCaR to process encoded image and semantic state inputs.\n\n**Regarding the training setting.**\n- We actually have shown the training curve in Fig.10 in the Appendix, where the x-axis represents the number of training steps and the y-axis shows the success rate of the sub-task. We chose 150M steps to train the sub-tasks to ensure that the skills are well-learned and  robust across various conditions and scenarios, which is critical for tasks that are content-rich and require high precision. And the benefits of high-quality policy learning in a simulated environment can often offset the computational costs, thus facilitating zero-shot  deployments in real-world applications.\n\n**Regarding Question 1.**\n- Thank you for your question. In fact, as shown in the training curves in Fig.10, our adaptive skill learning with AES regularization converges quickly and exhibits more stable training performance compared to other methods. In contrast, the comparison methods—PPO, GAIL, and fixed-RL-IL—do not achieve fast convergence or stable training performance, even with 150M training steps. This comparison highlights the  effectiveness of our proposed method.\n\n**Regarding Question 2.**\n- Thank you for your question. We chose the method of learning from sub-tasks because it is better suited to the long-horizon manipulation problem we are focusing on. The long-horizon tasks can be better accomplished by breaking them down into easier-to-learn sub-tasks, learning the corresponding skills for the sub-tasks, and then integrating the skills sequentially to execute the complete skill chaining. Learning the entire long-horizon task from scratch, without decomposing it into sub-tasks, using RL or IL is often challenging, as shown in the comparison results in Table 1 of the original paper.\n\n**Regarding Question 3.**\n- Please correct us if we misunderstand your question. If you are referring to the discriminator used in the GAIL part of IL in adaptive skill learning, the answer is yes. However, if you are referring to the bi-directional discriminator for the inter-skill chaining process, the answer is no. The initialization and learning of the bi-directional discriminator are not based on the demonstration data but instead rely solely on the set of successful initial and terminal states collected from the pre-trained skills.\n\n**Regarding Question 4.**\n- Thank you for pointing it out! The loss function for the least-squares discriminator was first proposed by [3], and we will include a citation for it in the updated version of the paper.\n\n[1] Haldar, Siddhant, et al. Watch and match: Supercharging  imitation with regularized optimal transport.\n\n[2] Liu, Yuyang, et al. Imitation Learning from Observation with Automatic Discount Scheduling.\n\n[2] X. Mao, et.al. Least Squares Generative Adversarial Networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQ7csxSBFg", "reviewer": "Reviewer_ysj9", "summary": "This paper focuses on skill chaining for long-horizon robotic manipulation tasks. A dual regularization method is proposed to tackle this problem, including an adaptive sub-task learning scheme that enhance intra-skill dependencies, and a bi-directional adversarial learning mechanism for reinforcing inter-skill dependencies. The proposed framework is evaluated on two long-horizon robotic manipulation simulation benchmarks and real-world robot pick-and-place tasks. Results show that the proposed method outperforms benchmarks on success rate and is more robust to perturbations.", "questions": "How is the sub-task division of the long-horizon task defined and how will different divisions affect performance?\n\nHow well can the method scale to longer-horizon manipulation tasks? How would the method perform when the number of sub-tasks increases?", "limitations": "The authors have adequately addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Applying dual regularization to skill chaining for long-horizon manipulation tasks is well-motivated by the two common failure cases--failed pre-training of sub-task skills and failed skill chaining due to disturbance.\n\nExtensive experiments in simulation are performed to demonstrate the effectiveness and robustness of the proposed method. Detailed ablations are performed to validate the effectiveness of each component of the approach.", "weaknesses": "Real-world evaluation is only done on desktop pick and place task, which is a well-studied task that has many effective and robust solutions, and does not necessarily require intra-skill dependency and inter-skill dependency. The method would have better soundness if it is evaluated on more useful real-world robotic tasks.\n\nThe writing of the paper can be improved, especially the figures in the paper are hard to see--the robot execution frames are too small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for acknowledging the strength of our work, e.g., reasonable idea of dual regularization and extensive experiments. We address the comments and questions below.\n\n**Regarding real-world robotics experiments.**\n\n* Thank you for pointing this out. Due to hardware limitations, we are  currently using simple long-horizon pick-and-place tasks to validate  SCaR's skill-chaining performance in real-world robotics. This is the  same limitation and future direction discussed in the \"Limitations and  Future Directions\" section of the original paper. We are in the process  of procuring a Franka robotic arm, which we expect will allow us to  validate SCaR on more complex and extensive long-range robot  manipulation tasks. We hope that our solution will serve as a solid  baseline for future research on long-horizon manipulation tasks.\n\n**Regarding writing and figure sizes.**\n\n* Thank you for pointing out these issues. We will improve the writing in subsequent updates of the paper based on your comments. We improve the resolution and size of the robot execution frames in the figure (as shown in the uploaded pdf) and will add it to the revised paper.\n\n**Regarding Question 1.**\n\n* **The Sub-task Division:** At the macro level, the division of sub-tasks for long-horizon tasks is determined by analyzing the task's complexity and identifying logical, discrete components that can be learned independently. This division is typically guided by the natural segmentation of the task into sequential steps or phases, each requiring a different skill.\n\n  In our experimental setup, the division of sub-tasks was predefined according to the stages of task execution. For example, in the long-horizon task of \"assembling a table,\" each sub-task was defined as \"installing table leg 1\" through \"installing table leg 4.\" Similarly, in the \"kitchen organization\" task, the order of sub-tasks was predefined as \"turn on the microwave,\" \"move the kettle,\" \"turn on the stove,\" and \"turn on the light.\" The sub-tasks for each long-horizon task in our experiments are described in detail in Appendix C of the original paper.\n\n* **Impact of Different Sub-task Divisions:** Thank you for your insightful question. To address your query, we conducted experimental validation using the chair_ingolf task. The original sub-tasks in this task are divided as follows: \"Assemble chair support 0 to target position\" → \"Assemble chair support 1 to target position\" → \"Assemble front leg 0 to target position\" → \"Assemble front leg 1 to target position.\" We have re-divided the sub-tasks into two alternative settings:\n\n  1. \"Assemble chair support 0 and chair support 1 to target positions\" → \"Assemble front leg 0 to target position\" → \"Assemble front leg 1 to target position.\"\n  2. \"Assemble chair support 0 to target position\" → \"Assemble chair support 1 to target position\" → \"Assemble front leg 0 and leg 1 to target positions.\"\n\n  Due to time constraints during the rebuttal phase, we were only able to run one test seed. The test results are as follows:\n\n* |      | chair_ingolf  (setup 1) | chair_ingolf  (setup 2) |\n  | :--: | :---------------------: | :---------------------: |\n  | SCaR |          0.68           |          0.74           |\n\nIt is worth noting that, since the re-division of the sub-tasks results in only three sub-tasks, we set 90% as the success metric for all three sub-tasks being successfully executed. As seen in the table above, compared to SCaR's success rate of about 95% with the original four sub-task divisions, the success rate for completing the first sub-task and then executing the remaining two sub-tasks is significantly reduced. This decrease is due to the increased difficulty of the first sub-task in setup 1 (which requires assembling both chair support) and the last sub-task in setup 2. These changes result in a lower overall success rate for the task.\n\nThis result suggests that a reasonable division of sub-tasks in long-horizon tasks is crucial for the success rate of overall task completion. We will add this experimental result and related discussion to an updated version of the paper.\n\n**Regarding Question 2.**\n\n* Thank you for the question. In the original experiment, we demonstrated the performance of SCaR for long-horizon manipulation tasks with up to 5 sub-tasks, specifically in the \"Extended Kitchen\" task, which includes the following sub-tasks:\n\n  1. Turn on the microwave\n  2. Turn on the stove\n  3. Turn on the light\n  4. Slide the cabinet to the right target position\n  5. Open the cabinet to the target position\n\n  To further answer your question, we added a sub-task to the Extended Kitchen task to evaluate SCaR's performance in manipulation tasks with longer horizons, involving 6 sub-tasks. The modified task, \"Longer Extended Kitchen,\" includes:\n\n  1. Turn on the microwave \n  2. Turn on the stove\n  3. Turn on the light\n  4. Slide the cabinet to the right target position\n  5. Open the cabinet to the target position\n  6. **Move the kettle to the target position**\n\n  Due to the time constraints of the rebuttal period, we were only able to compare SCaR with T-STAR and run one test seed. The comparison test results are as follows:\n\n|        | Longer Extended kitchen |\n| :----: | :---------------------: |\n| T-STAR |          0.33           |\n|  SCaR  |          0.61           |\n\nAs can be seen from the results, adding an additional sub-task increases the complexity and difficulty of the long-horizon task. Despite this,  SCaR still achieves a higher overall task execution success rate, with 28% higher compared to T-STAR. Although there is significant room  for further enhancement, we believe our solution offers a promising  baseline for future research in skill chaining methods for long-horizon  manipulation tasks. We will include this experimental result and related discussion in the updated version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Global response**

Dear Reviewers,

Thank you very much again for your helpful comments. We appreciate your recognization of our work and would like to engage with you in our responses to your questions/comments. We hope that our approach will provide a simple and solid baseline for future research on skill chaining methods for long-horizon manipulation tasks.

 If you have any questions about our work or our response, we would be happy to discuss them further.

Best Regards,

the Authors.

Meta-review: This paper proposes an advanced skill chaining algorithm, addressing long-horizon tasks. While the proposed method builds upon the existing T-STAR framework, it introduces two key innovations: adaptive equilibrium scheduling and bi-directional regularization, which significantly improve the robustness of skill chaining. The empirical evaluations are comprehensive, demonstrating superior performance on complex tasks, such as furniture assembly and kitchen organization.

The reviewers initially had concerns, but after the rebuttal phase, they reached a consensus to accept this paper, contingent on additional experimental results and clarifications regarding the methodology and experimental details. 

The authors have committed to enhance the paper by (1) improving the overall writing and figure quality, (2) incorporating discussion points, particularly regarding the sub-task decomposition, (3) addressing more challenging tasks in both simulated and real-world environments.
Decision: Accept (poster)