Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models
Authors: Utkarsh Aashu Mishra
Shangjie Xue
Yongxin Chen
Danfei Xu
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation Planning
Diffusion Models
Task and Motion Planning

Abstract: Long-horizon tasks, usually characterized by complex subtask dependencies, present a significant challenge in manipulation planning. Skill chaining is a practical approach to solving unseen tasks by combining learned skill priors. However, such methods are myopic if sequenced greedily and face scalability issues with search-based planning strategy. To address these challenges, we introduce Generative Skill Chaining (GSC), a probabilistic framework that learns skill-centric diffusion models and composes their learned distributions to generate long-horizon plans during inference. GSC samples from all skill models in parallel to efficiently solve unseen tasks while enforcing geometric constraints. We evaluate the method on various long-horizon tasks and demonstrate its capability in reasoning about action dependencies, constraint handling, and generalization, along with its ability to replan in the face of perturbations. We show results in simulation and on real robot to validate the efficiency and scalability of GSC, highlighting its potential for advancing long-horizon task planning. More details are available at: https://generative-skill-chaining.github.io/

Generative Skill Chaining: Long-Horizon Skill Planning with Diffusion Models
Utkarsh A. Mishra, Shangjje Xue, Yongxin Chen, Danfei Xu Georgia Institute of Technology Email: (umishra31 xxj, yongchen, danfei Beaeeieegaaachh..c edu
Long-horizon tasks, usually characterized by complex subtask depen- dencies, present significant challenge in manipulation planning. Skill chaining is a practical approach to solving unseen tasks by combining learned skill pri- ors. However, such methods are myopic if sequenced greedily and face scalability issues with search-based planning strategy. To address these challenges, troduce Generative Skill Chaining (GSC), a probabilistic framework that learns skill-centric diffusion models and composes their learned distributions to generate long-horizon plans during inference. GSC samples from all skill models in par- allel to efficiently solve unseen tasks while enforcing geometric constraints. We evaluate the method on various long-horizon tasks and demonstrate its capability in reasoning about action dependencies, constraint handling, and generalization. along with its ability to replan in the face of perturbations. We show results in simulation and real robot to validate the efficiency and scalability of GSC, highlighting its potential for advancing long-horizon task planning. More details are available at: sttps://generative--te ceaiiiinhaining.. github.
Abstract:
in-
Keywords: Manipulation Planning. Diffusion Models, Task and Motion Planning
1 Introduction
Long-horizon reasoning is erucial solving real-world manipulation tasks that involve complex inter-step dependencies. An illustrative example is shown in Figure 1(bottom), where a robot must reason about the long-term effect of each action choice, such as the placement pose of the object and how to grasp and use the tool, in order to devise a plan that will satisfy various environment constraints and the final task goal (place object under rack). However, finding a valid solution often requires searching in a prohibitively large planning space that expands exponentially with the task length. Task and Motion Planning (TAMP) methods address such problems by jointly searching for a sequence of primitive skills (e.g., pick, place, and push) and their low-level control parameters. While effective, these methods require knowing the underlying system state and the kinodynamic models of the environment, making them less practical in real-world applications. This work seeks to develop learning-based skill planning approach to tackle long-horizon manipulation problems.
Prior learning approaches that focus on long-horizon tasks often adopt the options framework [1, 2] and train meta-policies with primitive skill policies as their temporally-extended action space [3-8]. However, the resulting meta policies are rask-specific and have limited generalizability beyond the training tasks. A number of recent works turn to skill-level models that can be composed solve new tasks via test-time optimization [9-15]. Key their successes skill-chaining functions that can determine whether each parameterized skill can lead to states that satisfy the preconditions of the next skills in plan, and eventually the success of the overall task. However, these methods are discriminative, meaning that they can only estimate the feasibility of a given plan and requires an exhaustive search process to solve task. This bottleneck poses a severe scalability issue when dealing with increasingly complex and long skill sequences.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

I(O,I)
1(0.1)
yxl
4x
AT
a1
22
3
44
81
82
$3
84
85
Transition Feasibility
Skill AlTurdabiiity
0
Place
State
Hock
Box Using Hook
Hock
Box
Box
Hook
Box Using Hook
Figure 1: (Top) Generative Skill Chaining (GSC) aims to solve long-horizon task for a given se- quence of skills by using linear probabilistic chains to parallelly sample from a joint distribution of multiple skill-specific transitions qn (s, a, s') learned using diffusion models. The framework implic- itly considers transition feasibility and subsequent skill affordability while demonstrating flexible constraint- handling abilities. (Bottom) An example of a nong-horizon TAMP problem composed of multiple skills. Such task necessitates reasoning inler-dependencies between actions.
In this paper, we propose generative and compositional framework that allows direct sampling of valid skill chains given plan skeleton. Key to our method is skill-level probabilistic generative models that capture the joint distribution of precondition skill parameter effect of each skill. Sampling a valid skill chain boils down to, for each skill in a plan, conditionally generating skill parameters and post-condition states that satisfy the pre-condition of the next skill, constrained by the starting state and final goal. The critical technical challenge is to ensure that the sequence of skill parameters achievable from the initial state (forward flow) satisfy the long-horizon goal (backward flow) and account for additional constraints.
To this end, we introduce Generative Skill Chaining (GSC), framework to train individual skill diffusion models and combine them according to given unseen task skeletons with arbitrary con- straints at test time. Each skill is trained as an unconditional diffusion model, and the learned distri- butions are linearly chained to solve for an unseen morizoo goal during evaluation. Further, we employ classifier-based guidance satisfy any specified constraints. GSC brings a paradigm shift in the approaches used to solve TAMP problems to date and uses probabilistic models to establish compositionality and reason for long-horizon dependencies without being trained on any such task. We demonstrate the efficiency of GSC on three challenging manipulation task domains, explore its constraint handling advantages, and deploy a closed-loop version on a physical robot hardware to show generalization capabilities and robustness.
2 Related Work
Task and Motion Planning. Understanding inter-dependencies between sequential actions is fun- damental challenge in solving long-horizon problems. The key idea for solving them is to break the planning problem into symbolically feasible sequence of smaller subtasks [1-3, 7, 8] and characterize their solutions with primitive actions or skills [4-6]. Such approaches rely on formu- lating folly-observable conditions and accurate system dynamics forecasting [16, 17] to realize the precondition of applying skill (affordability) and its effect respectively [18-24]. In this direction, logic-geometric programming [25, 26) and hierarchical [27] frameworks have solved for the sym- bolic feasibility [18, 28, 29] of sequence of skills sufficient to reach a goal condition from the initial state. While such methods are exhaustive, their strong assumptions limit their practical applications and scalability. To overcome this, we opt for learning- based framework [11, 30).

---

Learning to solve long-horizon tasks. Skill-chaining methods model pre- and post-conditions of pre-defined skills to search for feasible goal-reaching plans, but most methods have focused on single-task settings [9-15]. Recent methods have investigated composable skill models to learn multi-task planner [11, 27, 30j. However, such methods are discriminative and require exhaustive searches. Moreover, their auto-regressive nature leads to cascading errors and large exploration spaces as the tasks become long. Recently, Agia al. [11] proposed CEM-based skill-chaining strategy that maximizes the success of individual actions in the sequence by training individual skills as RL agents. Their method is still limited to trained policy (deterministic) priors, lack finding multi-modal solutions, and cannot account for additional planning constraints.
Generative models for planning. Generative models have been widely used for planning with Gaussian processes [31] and adversarial networks [32, 33]. With recent advancements in diffusion model-based generative strategies for planning in roboties, such approaches have been adopted for imitation learning [34-38] and offline reinforcement learning [39, 40] settings. Most relevant to this work are Diffuser [39], Decision Diffuser [40] and Diffusion Policy [37]. While they can synthesize sequences of "states" (and/or "actions") as plans and optionally account for constraints [40], such approaches still focus on capturing the distribution of training task solutions and cannot easily solve unseen new tasks. In this work, we introduce compositional [41, 42] generative planning method that can flexibly combine skill-level generative model and generalize to new tasks and constraints introduced during inference time. The proposed framework is inspired by the recent work on generating high-quality extended image chains with parallel diffusion [43] and scalable diffusion models with transformers [44].
3 Preliminaries
Problem formulation. We formulate the skill chaining problem by considering a given symboli- cally feasible skeleton Φ = {(T1L 01), (T2, o2),..., (TK, ok)} of skills from a pre-defined library TiK E II and the set of objects o on which the skill operates. Each skill π E II is parameterized by a continuous set of feasible skill parameters a E Ar governing the desired motion while executing the skill from a state s in the state space S. The goal to find optimal skill parameters such that the skeleton is geometrically feasible and the effect of each skill satisfies the precondition of the following skills while the goal condition is satisfied.
Environment setup. We use expert policies to solve individual skills and collect state-action-state transitions between current (s E S) and next state (s' E S) resulting from the execution of a skill π with parameters a Ar by following the transition model Ta S x A,, - S. Further, following previous work [11], we consider a selection of basic objects like hook, rack, and boxes of various dimensions construct the environmental setup. The state of the environment consists of fully observable poses and sizes of each of the present objects.
Diffusion models. The diffusion model is a parameterized model (x())) that estimates an unknown distribution (xxo) using the samples Xg ~ (Xo). It consists of two diffusson processes: a forward noising and a reverse denoising process. The forward process progressively injects i.i.d. Gaussian noise samples from go and leads to a family of noised distributions qe (Xx). The distribution of X; conditioned the clean data No is also Gaussian: qoe q0e (xx|xo)
Nx(x;; No, IfI,, where at defines fixed series of noise levels monotonically increasing w.r.t. forward diffusion The reverse denoising process recovers clean data by iteratively removing the added noise by a process represented as the following stochastic differential equation (SDE) [45-47]:
dx = - V, log qe (xx)dt + / 2o
(I)
where Vx log qe (x:) is referred to as the score function of the noised distribution and w, a standard Wiener process. We follow DDPM [48] sampling strategy in continuous settings [49]. The score function allows recovery of the minimum mean squared error estimator of xo given x, [50, 51]:
x = E|xo|x,] Xt + oi log qi (xx),
(2)
3

---

where we can treat Xo as "denoised" version of X2 at timestep t. In practice, the unknown score function is estimated using a neural network <g xx,,,,) () by minimizing the denoising score match- ing [49] objective x, o Vx, log qoe (x (x)] - cx(x,, t)]l where A(t) is a time-dependent weight. Diffusion models are scalable, and the learned distributions represent all positive samples satisfying the distribution heuristic, thus multi-modal. Further, their simple probabilistie represen- tation allows wide range of flexible sampling strategies [47, 48, 52, 53] combined with constraint- handling abilities [40, 54-56].
4 Methodology
Generative Skill Chaining (GSC) offers a new paradigm for approaching long-horizon planning with a given skeleton of skills. The primary objective of GSC is to determine the optimal skill parameters for an unseen task skeleton, such that executing the plan achieves long-horizon goal while satisfying task-specific constraints. It introduces probabilistic chaining of distributions of short-horizon transitions to sample from long-horizon trajectory distribution. GSC uses skill-level diffusion models to represent each skill's joint distribution of precondition, control parameters, and effect. Further, the framework composes individual skills at inference time to form sequence- level trajectory distribution, which can be sampled via parallel diffusion to generate feasible skill parameter sequences as planning solutions. This is different from the widely used auto-regressive heuristic-search-basee approach [17, 57-59] used in prior works [11, 30J.
We consider given skeleton + of skills (and relevant objects) which satisfies the symbolic feasibility of the sequence in the environment. The primary goal is to generate the sequence of states and skill parameters (as shown in Figure 2(a)) such that the final state (here sy s(2) satisfies a goal condition and leads to the successful execution the last skill.
X2
(0)
M
Action primitives as diffusion models. We characterize the individual skills by the nature of state-action-stuu transitions observed while executing it in the environment. The opera- tion of each skill π can then be represented by an uncondi- tional distribution 9r (S,, at, s(). Such a representation simul- taneously captures the skill policy P5 and the transition dy- namies distribution Tr and ensures their consistency. For each train diffusion model score
V
(b)
Figure 2: (a) A linear chain graph for a long sequence of transitions and (b) adding an additional con- straint node.
skill in the skill library, function e (St, a Si, f) with transformer backbone shown in Figure 3 (right) using provided per-skill demonstration data. We represent the set of objects of interest by an order which denotes the relevance of the objects in the scene w.r.t. the skill' [11]. We also denote the masked sampling score model for the states and action as Ex (Sc. t), e (s}, t) and (a,, t) respectively.
Sequencing skill diffusion models. To solve objective of finding sequence of suitable skill transitions which satisfies + an auto-regressive approach primarily used in prior works follows:
(s (0:2) (0:)) = P.. (a (0) (0)) T (s(1) (0)) a (0)) Pay (a (a(1s)))) (s)(s())ss)) a
However, such formulations are myopic and can only be rolled out in the forward direction with- out feedback from the final task goal. This limits long-howe horizon reasoning, and prior methods have leveraged random [30] or CEM-based rollouts [11] to sample from such a distribution. overcome the above limitations, we transform the unconditional skill diffusion models into forward and backward conditional distribution,
())) (1), (1)) Jm. (s (0) ((1) )qn, (s (())
pa (s (0:2) a (0:1) s (0)) x qu
(s (0) a (0)
9mz (s)
For example, if there is hook (1), a box (2) and rack (3) in the environment, then the object order corresponding to tasks are: (a) pick the box: [2, 1,3). (b) Place Box on Rack: [2, 3, 1].

---

(Hook, Table)
Pick
(a,,)
co (a.,
c (slit)
Place (Red Rack)
Unconditional Skill Score Function ey (,(
(Hock, Blue Box)
#2
Push (Hook, Blue Box,
Time
Positional Embedding
Skil Object Order
Rack)
Embedding
Figure 3: Left The primitive skills and their executions are shown with the objects of interest. Right Transformer-base
skill diffusion model. We use the noisy state-action-state distribution X ~ {sr, ag, } at diffusion step f to obtain the corresponding Eg during sampling. The skill object order depends on the objects of interest and is represented as a collection of one-hot vectors.
MTo (s(1) a (1) s22) 4n. (s(0) (0) s()) 14 (s(),
(2)
Po (s (0:2) (0:1) 8 (22 xx qm (s a (0)
(s(1)
In both equations above, the relations implicitly give rise to the notion of skill affordability and transition feasibility, i.e. the resulting state from the one skill must lie in the initial state distribution of the next skill and vice-versa. Now, if we transform the probabilities into their respective score functions (Vx log ((x)) for a particular reverse diffusion sampling step L, we obtain:
60 (s (0)
a
1)) cm, (s
() + Emg (St
(1)
16 (2)
Exg (1)
(3)
(s
a
= Ex
t) + Emg (St
(2) Exi (s)
12)
(4)
respectively. Finally, we linearly combine the score functions from the forward and backward dis- tributions weighted by a dependency factor 7:
(s () 7m, cx, (s , (1 7n, Cx2
(5)
Here, y E [0, 1] is a decision variable that balances the influence of the state in the transition of the skill w.r.t. the subsequent skill and the goal condition. This an important aspect that governs the behavior of the skills in the sequence and the choice of their respective parameters.
Classifier-based guidance for constraint satisfaction. Besides the final task goal, constraints play an important role in governing the feasibility of actions in the environment and specifying task- specific conditions, such as maximizing/miniiiiing the distance between two objects in an interme- diate or final state. Our diffusion moddel-aaeed formation allows GSC to easily incorporate additional constraints as implicit (in- (in-painting) or explicit (classifier-bases based) guidance. Here we present a flex- ible sampling strategy in the presence of several planning constraints. In principle, the additional constraints v can be appended as additional terms in the target sampling distribution:
pr. p (s (0:2)) x pa (0:2) (0)) h(fs, a)w)
where h(-)) is the likelihood of the constraint acting on set of state-action nodes given by (s, a. The corresponding diffusion score function with the added constraints becomes
(s a
= (s,
,t) + c (( a)),t)
(6)
where eq ((s,, }. t) V as } log (s,,,, ar ov). Consider the example shown in Figure 2(b) where the constraint depends on the nodes a (0) (1) and s22), Suppose the constraint is chosen be binary indicator (i.e. success 1) of satisfaction and is defined for the denoised samples (a. aa and (2)) at 0. In such situation, the likelihood is defined as the exponential of the constraint satisfaction such that
he (a) 20
(0) (1) (22)) exp a (a (0) 20
(1) (2)
5

---

It is worth noting that while the constraint is a function of the denoised samples, the gradients must be calculated w.r.t. the noised samples. We calculate this by first obtaining the denoised sample x for the diffusion step t from Equation and then modifying the corresponding nodes in (0)
(s,
(2) (2) based on the weight factor o, as
c (0) (1) (2) Cop
(0)
(2) (1 (1 D0 ())))
(0) (1) 1/
(8)
Summary. To summarize, the proposed framework GSC is divided into three segments: (1) train individual skill diffusion models with the proposed architecture without any knowledge about other skills, (2) chain skill diffusion models according to an unseen task skeleton during inference using probabilistic linear chaining of the individually learned distributions with a dependency factor, and (3) incorporate classifier-based guidance for any unseen planning constraint added while inference. Following standard reverse denoising, we consider parallelly sampling from all individual models instead of one and hence our proposed approach is both task-skeleton and skeleton-length agnostic. Further, the dependency factor helps in making flexible design choices for satisfying the desired goal condition. While constant value of skill. In addition to the above, we also collect failure data to train a success probability prediction module (ss, s') S S S + [0, 1] for each skill which a measure of the successful execution of the skill given the current and the transitioned state. Such model is used to consider the best parameter sequence from the sampled candidate solutions. We illustrate the overall algorithm in Appendix A.
0.5 is sufficient, it can be fine-tuned for every
Results
We conduct experiments to validate the efficacy of GSC in (1) long-horizon planning for unseen tasks of arbitrary lengths, (2) constraint handling and satisfaction, and (3)
demain
genemited combiration (fixed initial state)
(a)
maximizing action-dependency horizon and finally (4) gen- eralization to perturbations. First, show the composi- tional and constraint-handliigg performance of GSC in a toy domain. Second, we evaluate the performance of the chain- ing trained skill diffusion models on nine standard TAMP tasks introduced by previous work [11]. These tasks en- compass wide range of skeleton lengths and challenge the method on various levels of long-horizon dependency.
learned skill models
w/ construints for left segment (final state)
(d)
GSC
(b)
wi fixed final state
Finally, we discuss the response of GSC to perturbations, followed by the importance of dependency factor y in the success of GSC.
(c)
Baselines and metrics. the context of skill chaining, we primarily consider search-based methods with CEM op- timization strategy. Our main baselines CEM meth- ods with uniform priors (Random CEM) and learned pol- icy priors (STAP). Further, to show improvement in per- formance as compared to training on task sequences and expecting generalization to new ones, we add DAF's [30] performance following Agia et al. [11]. Another potential baseline is using diffusion for states only and inverse dynamies model for actions based on Decision Diffuser [40]. However, due considerable distribution shift and cascading error from state pre- dictions, such method does not perform well (refer to Appendix B). The success rate of satisfying the goal condition in 100 random environment executions is used as the comparison metric?.
Figure 4: Toy Domaïn: We model four distributions of states and seg- ment one of them into left and right segments. The figure illustrates dif- fusion model composition using GSC with fixed start state and unit-vector actions followed by the addition of soft and hard constraint guidance.
Toy domain. The 2D domain consists of states as the s the action, [r, e 2R2,, as the direction of the unit vector (0) and magnitude (r). A transition
(x, position of a point sample and
We consider the same results for baselines demonstrated in previous work [11].

---

defined by s+ |r cos(f), r sin( To replicate the skill distributions, we create multiple state distributions Figure 4(a) and collect transition data between them to represent individual skills. We illustrate the performance of GSC in chaining trained skill models Figure 4(b) to sample transitions between sequence of distributions Figure 4(c) with fixed initial state and unit action. Finally, we validate the constraint satisfaction performance by first constraining final states to the left segment and second, keeping the goal state fixed Figure 4(d)(e), respectively. The success in the above tasks validates the framework's ability to compose skills and handle arbitrary constraints directly while evaluation. Further clarification for the toy domain is provided in Appendix C.
Long-horizon manipulation. We consider TAMP problems in a PyBullet [60] environment and follow the tasks proposed by STAP [11]. namely: (i) Hook Reach: The hook is used to pull a box followed by executing other skills the box. (ii) Constrained Packing: Multiple boxes should be placed the rack that all of them
be accommodated without interference. (iii) Rearrange- ment Push: A sequence of placement and push using hook is executed to bring box below the rack. The proposed three skeletons for each category above are solved using our method and com- pared against the previous work and their baselines w.r.t. the success rate is shown in Table 1. These tasks are unseen while training, have varying skeleton lengths, and demand reasoning long-horizon action dependencies. The performance of GSC implies that is better than (or as good as) search- based methods, along with other advantageous abilities. The details about the target skeletons and the desired transitions further explained in Appendix E.
Imposing additional constraints. In addition to the skeletons evaluated for the long- long-horizon prob- lems and extending the experiments conducted for the toy domain, we impose certain planning constraints on the final state, intermediate states, and actions. This is done by adding an objective of maximizing the distance between all the "place" skill action parameters in the skeleton (this ad- dition is still task agnostic) for the constrained packing task. The resulting sequence of states is shown in Figure 5. Further, we compare cumulative task completion and constraint satisfaction success rate with previous approaches in Table 2. This qualitatively and quantitatively demonstrates that the framework can handle such unseen constraints in test time.
Table 1:
The success rate of the proposed GSC algorithm is shown and compared with relevant search-bare based baselines (CEM strategy). All results are calculated from 100 trials for each task.
Hook Reach
Rearrangement Push
Methods
Task
Task 2
Task 3
Task
Task 2
Task 3
Random CEM
0.54
0.40
0.30
0.30
0.10
0.02
DAF (Generalization)
0.32
0.05
0.0
0.0
0.08
0.0
STAP (Policy CEM)
0.88
0.82
0.76
0.40
0.52
0.18
GSC (Ours)
0.84
0.84
0.76
0.68
0.60
0.18
Task Length
4
5
5
4
8
Figure 5: For a constrained packing task of picking-and and-alacing all the boxes on the rack: Task agnostic secondary placement objectives help in realizing accurate and consistent state-action sequences in Top Simulation and Bottom open loop hardware rollouts.
Generalizing to unseen sequences. We next highlight the generalization ability of GSC by evalu- ating on more complex unseen task plans. First, we increase the maximum horizon dependency of action by changing the goal to position not only the box but also the hook in such fashion which complies with the success of the "push" skill in the rearrangement push domain. The resulting
7

---

Table 2:
The success rate of GSC algorithm with and without task-agnostic constraint handling is shown for Constrained Packing task and compared with relevant baselines.
Constrained Packing
Methods
Task
Task 2
Task 3
Random CEM
0.45
0.45
0.10
DAF (Generalization)
0.45
0.70
0.0
STAP (Policy CEM)
0.65
0.68
0.20
GSC (Ours w/o constraint guidance)
0.70
0.80
0.50
GSC (Ours w/ constraint guidance)
0.75
1.0
1.0
state sequence is shown in Figure 1 (bottom). Secondly, we execute algorithm in closed-loop fashion and use the skill success classifier to indicate sub-task completion. The sampled plan is executed and the resulting state is checked for subsequent skill feasibility. In case, the state satisfies the pre-condition for a future or previous skill, the plan resampled from that skill to complete the task. We demonstrate this perturbation experiment on a Franka Panda arm (video attached to supplementary, hardware execution details in Appendix H).
Importance of the forward and backward dependency. The dependency variable governs the flow of information from the initial state (forward) and the goal (backward) during the diffusion process. We provide qualitative study of this feature in Figure 6. In the case where the forward flow information is weak (7 0), the model tends to hallucinate and predict states that are inconsistent with the initial state. When the backward flow is weak (7 1), the model becomes myopic and fails to solve the task. We empirically found that y 0.5 achieves balanced performance.
Figure 6: Example of rearrangement push task rollout with different dependency factor y: Top: Hallucination (y 0) With weak forward signal, the framework fails to realize correct position of objects at the final stage. Bottom: Myopic (? 1) A weak backward information behaves like policy shooting and fails to understand long-horizon dependency.
Limitations
The proposed framework considers planning with given skeleton. While we do not solve the complete TAMP problem, our method is compatible with any skeleton-plamiing method and hence crucial segment of unified framework for solving TAMP problems. Further, only validated on a fully observable environment setup with no degree of partial observability and operates on low- dimensional state space of the system i.e. 6-DoF poses of the objects. We use fixed set of primitive skills, and thus, the framework requires either expert data to train models or the pre-trained models to perform compositional planning. This can be extended by incorporating skill discovery frameworks.
7 Conclusion
We introduced GSC, a new paradigm to solve TAMP tasks with given skeletons using skill-centric diffusion models. GSC trains high-quality skill diffusion models using a transformer backbone and composes skeleton-specific distributions for unseen skeletons by chaining trained individual skill distribution. Such -keleton-specifie distributions are then used generate long-horizon param- terized skill plan sequences. The framework scalable and flexible and shows better constraint- handling capacities, and generalizes well to new scenarios, including perturbations and replanning.

---

References
[1] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tem- poral abstraction in reinforcement learning. Artificial intelligence, 122(1-2)1811-11 1999.
[2] P.-.. Bacon, J. Harb, and D. Precup. The option-crite architecture. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[3] D. Shah, P. Xu, Y. Lu, T. Xiao, A. Toshev, S. Levine, and B. Ichter. Value function spaces: Skill-centric state abstractions for long-horizon reasoning. arXiv preprint arXiv:2111.03189 2021.
[4] S. Nasiriany, H. Liu, and Y. Zhu. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA), pages 7477-7484. IEEE, 2022.
[5] N. Vuong, H. Pham, and Q.-.. Pham. Learning sequences of manipulation primitives for robotic assembly. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 4086-4092. IEEE, 2021.
[6] R. Chitnis, S. Tulsiani, Gupta, and A. Gupta. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages
IEEE, 2020.
[7] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. Advances in neural information processing systems, 31, 2018.
[8] W. Masson, P. Ranchod, and G. Konidaris. Reinforcement learning with parameterized actions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[9] Y. Lee, J. Lim, A. Anandkumar, and Y. Zhu. Adversarial skill chaining for long-hou horizon robot manipulation via terminal state regularization. In Sth Annual Conference on Robot Learning, 2021. URL https: /twp///eeeeeeeeeee///eee
[10] Y. Lee, S.-H. Sun, Somasundaram, learning transition policies. In International Conference on Learning Representations, 2019.
Hu, and J. J. Lim. Composing complex skills by
[11] C. Agia, T. Migimatsu, J. Wu, and J. Bohg. Taps: preprint arXiv:2210. 2022.
Task-agnostic policy sequencing. arXiv
[12] G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robotics Research, (3)33663-37 2012.
[13] A. Bagaria and G. Konidaris. Option discovery using deep skill chaining. In International Conference Learning Representations, 2020.
[14] G. Konidaris and A. Barto, Skill discovery in continuous reinforcement learning domains using skill chaining. Advances in neural information processing systems, 22, 2009.
[15] A. Clegg, W. Yu, J. Tan, C. K. Liu, and G. Turk. Learning dress: Synthesizing human dressing motion via deep reinforcement learning. ACM Transactions Graphics (TOG), 37 (6):1-10 2018.
[16] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[17] K. Chua, R. Calandra, R. McAllister, and Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.

---

[18] L. P. Kaelbling and T. Lozano-Pérez. Learning composable models of parameterized skills. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 886-893. IEEE, 2017.
[19] J. Liang, M. Sharma, A. LaGrassa, S. Vats, S. Saxena, and O. Kroemer. Search-based task planning with learned skill effect models for lifelong robotic manipulation. In 2022 Interna- tional Conference on Robotics and Automation (ICRA), pages 6351-6357. IEEE, 2022.
[20] Silver, R. Chitmis, Kumar, W. McClinton, T. Dozano-Pereez L. Kaelbling, and Tenen- baum. Inventing relational state and action abstractions for effective and efficient bilevel plan- ning. arXiv preprint 2oXix:2203009344 2022.
[21] G. Konidaris, L. P. Kaelbling, and T. Jazano-Perez. From skills to symbols: Learning symbolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61: 215-289, 2018.
[22] V. Xia, Z. Wang. and I.. P. Kaelbling, Learning sparse relational transition models. arXiv preprint arXiv: 1810.11177 2018.
[23] H. M. Pasula, L. S. Zettlemoyer, and L. P. Kaelbling. Learning symbolic models of stochastic domains. Journal of Artificial Intelligence Research, 29:309-352, 2007.
[24] Z. Wang, C. R. Garrett, L. P. Kaelbling, and T. Lozano-Pérez. Active model learning and diverse action sampling for task and motion planning. 2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 4107-4114. IEEE, 2018.
[25] M. Toussaint. Logic-geometric programming: An optimization- approach to combined task and motion planning. In IJCAI, pages 1930-1936, 2015.
[26] D. Driess, O. Oguz, and M. Toussaint. Hierarchical task and motion planning using logic- geometric programming (hlgp). In RSS Workshop Robust Task and Motion Planning, 2019.
[27] D. Driess, J.-S. Ha, and M. Toussaint. Learning solve sequential physical reasoning prob- lems from scene image. The International Journal of Robotics Research, 40(12-14);1435- 1466, 2021.
[28] B. Ames, A. Thackston, and G. Konidaris. Learning symbolic representations for planning with parameterized skills. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 526-533. IEEE, 2018.
[29] T. Migimatsu, W. Lian, J. Bohg, and S. Schaal. Symbolic state estimation with predicates for contact-rich manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA), pages 1702-1709. IEEE, 2022.
[30] D. Xu, A. Mandlekar, R. Martín-Martin Y. Zhu, S. Savarese, and L. Fei-Fei. Deep affor- dance foresight: Planning through what can be done the future. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6206-6213 IEEE, 2021.
[31] Z. Wang, C. R. Garrett, P. Kaelbling, and T. Lozano- Pérez. Learning compositional models of robot skills for task and motion planning. The International Journal of Roboties Research, 40(6-7):866889) 2021.
[32] T. Kurutach, A. Tamar, G. Yang, S. J. Russell, and P. Abbeel. Learning plannable representa- tions with causal infogan. Advances in Neural Information Processing Systems, 31, 2018.
[33] J. Ho and S. Ermon. Generative adversarial imitation learning, Advances in neural information processing systems, 29, 2016.
10

---

[34] T. Yu, T. Xiao. A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, J. Peralta, B. Ichter, et al. Scaling robot learning with semantically imagined experience. arXiv preprint arXiv:2302. / 1550, 2023.
[35] I. Kapelyukh, V. Vosylius, and E. Johns. Dall-e-bot: Introducing web-scale diffusion models to robotics. IEEE Robotics and Automation Letters, 2023.
[36] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S, V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677 2023.
[37] C. Chi, S. Feng, Y. Du, Z. Xu, E. A. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. ArXiv, Jbs/2300..4133,202 2023.
[38] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, B. Tenenbaum, D. Schuurmans, and P. Abbeel. Learning universal policies via text-guided video generation. ArXiv, abs/2302.00111. 2023.
[39] M. Janner, Du, J. Tenenbaum, and Levine. Planning with diffusion for flexible behavior synthesis. International Conference on Machine Learning, 2022.
[40] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and Agrawal. Is conditional generative modeling all you need for meaini? arXiv preprint arXiv:2211.1565 2022.
[41] N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423-439. Springer, 2022.
[42] Y. Du, C. Durkan, R. Strudel, J. B. Tenenbaum, S. Dieleman, R. Fergus, J. Sohl-Dickstein, A. Doucet, and W. S. Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. In International Conference on Machine Learning. pages 1489-8510, PMLR, 2023.
[43] Q. Zhang, J. Song, X. Huang, Y. Chen, and M.-Y. Liu. Diffcollage: Parallel generation of large content with diffusion models. ArXiv, abs/2303. 2023.
[44] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv preprint arXiv:2212.09748, 2022.
[45] T. Karras, M. Aittala, T. Aila, and S. Laine. Elucidating the design space of diffusion-based generative models. arXiv preprint arXiv:2206.00366 2022.
[46] S. Särkkä and A. Solin. Applied stochastic differential equations, volume 10. Cambridge University Press, 2019.
[47] Q Zhang, M. Tao, and Y. Chen. gddim: Generalized denoising diffusion implicit models. arXiv preprint vrXiv:2206.05564 2022.
[48] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-683 2020.
[49] Y. Song. J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole. Score-based gen- erative modeling through stochastic differential equations. arXiv preprint urXiv:2011.13456, 2020.
[50] S. Saremi and A. Hyvarinen. Neural empirical bayes. arXiv preprint urXiv:1903.023:: 2019.
[51] C. M. Stein. Estimation of the mean of multivariate normal distribution. The annals of Statistics, pages 1135-1151, 1981.
[52] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:0010.02502, 2020.

---

[53] Q. Zhang and Y. Chen. Fast sampling of diffusion models with exponential integrator. arXiv preprint arXiv:2204. 13902, 2022.
[54] Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
[55] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
[56] A. Lugmayr, M. Danelljan, A. Romero, F. Yu, R. Timofte, and L. Van Gool. Repaint: In- painting using denoising diffusion probabilistic models. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11461-11471, 2022.
[57] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, and E. A. Theodorou. Information theoretic mpe for model-based reinforcement learning. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 1714-1721, 2017. doi: 10.1109/CRAA2217..999992
[58] Pourchot and Sigaud. CEM-RL: Combining evolutionary and gradient-based methods for pol- icy search. In International Conference on Learning Representations, 2019. URL https: /ppenreview wwt.:eeotrrr.dd-keelllll Jdctt
[59] M. Janner, J. Fu, M. Zhang, and S. Levine When to trust your model: Model-based pol- icy optimization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. J'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL http://ttttttt/prreeeeeiggg http:/pp...cppppaa//////////1/1 StofGeeff0000711000011111777PP- pdf.
[60] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybullet: org, 2016-2021.
[61] Wang and E. Olson. AprilTag 2: Efficient and robust fiducial detection. In Proceedings of the IEFE/RSJ International Conference Intelligent Robots and Systems (IROS), October 2016.
12

---

A Summary of the Algorithm
Algorithm 1: Generative Skill Chaining (GSC) Algorithm
Hy Hyperparameters:
Number of reverse diffusion steps T
Forward-backwee backward dependency factor y
Gradient score function weight a
5 Inputs:
Pre-defined skill library IT {1, m RM}
7 Individual skill diffusion score functions 6
Task skeleton T ( ",,...,, Tk) be sequence of skills of length K
Initial state (0)
10 Goal condition ! Constraints h( (s. a)): suppose (s. a) =[((A), ,(2),s(K)
sk)] be the nodes affected by constraint
Initial skeleton solution Xy 1s ar ST a ..... [k sampled from (0,
(0)
(1) (1)
13
Initialize t T
while & 2 0 do
15
I/ Score of skeleton sequence
16
(s (0)
a
s =0
for 1: K do
18
// Update subvectors of cx
(s, (s a;
(1-1)
St t) = (s. (s. a; St:
(f-1) (1-1) (1) + E (5p (1111)
a a((-1) ,
19
(1) (s .t) = (s .) - (TE,, (s .t)
+1++ (1 -1--)0 7 (5) .t)
20
21
end
// Constraint handling
for v (("),((2) s(K] do
23
24
// Update subvectors 60
(v, () = eo (v., 1) - av, log (((()) 202) (2)) ((K)
25
26
end
// Obtain denoised samples
%o x orea (0)
(K)
28
29
// Get the updated noisy samples
30
Qo(e- 90(t-1>(Xx, (x, (x)) (x; 1)
31
end
33 Return Xg
Hyperparameters and Computation The number of reverse diffusion timesteps an important parameters which plays key role in deciding the time required to complete the sampling while keeping up with the quality of the generated samples. While a lower number of steps reduces the time taken for sampling, higher number of steps leads to finely denoised high-quality samples. We try with numerous values (256, 128, 64, and 50) and converge to using 128 diffusion steps for most of the tasks. The dependency factor 7 is set to be 0.5 following the explanations described in section 4 and section 5 Figure 6). A value of y makes GSC the same as a trivial policy rollout approach. Finally, in case of gradients, finetune the weights balance the effect of constraints in the reverse diffusion process. While it difficult drastically change the sampling trajectory due to the intricacies of the reverse process, use 1 for all our tasks with given planning constraints.
13

---

Individual Diffusion Model Score Function Hyperparameters
We follow the score-network architecture of DiT [44] and adopt to their open-source implementa- tion: github. wathuu.ccm/ffeeee cmm/facabookkresarcc//1111 We use the following hyperparameters for building score-network:
Table 3: Hyperparameters for Score-Network with Transformer Backbone
Hyper-parameter
Value
Hidden Dimension
128
Number of Blocks
Number of Heads
4
MLP Ratio
Dropout Probability
0.1
Number of Input Channels
17
Number of Output Channels
17
B Additional Discussion
Implementation details. The performance of the proposed skill sequencing framework depends on the diversity of the expert dataset, the maximum horizon dependency of action in the unseen skeleton, and the quality of the trained skill diffusion models. Furthermore, only the true distribu- tions are estimated and used to sample candidate solutions. To ensure high-success probability for all our tasks, we consider sampling multiple candidate sequence solutions (two of them are shown Figure 7) and consider the best probable solution based on the product individual skill success probability metric.
Box
Place Pushable
Placo Fushable
Figure 7: For a "pick-place" task of picking-and-plan the cyan box such that it can be pushed inside the rack: Left A correctly sampled state sequence while Right is incorrect. Hence, filtering candidate solutions is necessary.
Decision diffuser approach: state diffusion model with inverse dynamics actions. Diffusion models have been used for planning robotics. One such framework is that of the decision dif- fuser [40], which samples desired state trajectory and uses an inverse dynamics model find the best action sequence. Our framework However, this results in the distribution generated by diffusion models to be disjoint from the ac- tions given by the inverse dynamics models. This distribution shift is sensitive to the quality of the sampled states and hence results in cascading errors. Considering joint distribution of state-action- state transitions advantageous as it is less sensitive to such state perturbations.
achieve this by removing the action from the samples.
14

---

C Toy Domain: Explained
(A) Consider a toy domain with four different state (2D points) distributions and skills represented by unit vectors.
(B) We train diffusion models:
1. One with transitions from red square to two circles. This is analogous to training "pick hook" skill which is pre-condition for both "pull" and "push".
2. Other with the transition from bottom circle to green square. This is analogous to training only "push" (or "pull")
Now we merge both of these transitions to sample candidate solutions for the skeleton (pick hook, push block). When sampled parallely, the first diffusion model (pick skill) will sample post- conditions from both the circles (push and pull) whereas the second diffusion model (push) will sample pre-conditions only from the bottom circle. So, eventual outcome will from red square to bottom cirele green square (state sequence where hook is picked such that block can be pushed).
(C) We then impose conditions of "fixed initial state" and sample the chain.
(D) We then limit the goal states by imposing conditions that goal state variables must lie on the left half the green square distribution.
(E) We then fix both initial and goal states. This samples single solution. (as the states, 2D points, are connected via unit vectors with direction as the skill parameter)
domain
generated combination (fixed initial state)
(a)
(c)
learned skill models
w/ constraints for left segment (final state)
(d)
GSC
(b)
w/ fixed final state
(e)
Figure 8: Toy Domain: We model four distributions of states and segment one of them into left and right segments. The above figure illustrates diffusion model composition using GSC with fixed start state and unit actions followed by the addition of soft and hard constraint guidance.
15

---

D Skill Descriptions and Parametrization
We have a fixed library of skills consisting of: pick, place, push and pull. Each of the skills is parameterized with respect to the objects of interest according to the following setup:
pick: Parameterized by (z, 2, the pick location and the gripper's orientation around the axis. The parameters are calculated with respect the object of interest's (to be picked) origin. For example, pick block is w.r.t. the origin of the block, which is the centroid. Similarly, for pick hook, the origin is the center of the rectangle, of which the hook is one L-segment.
place: Parameterized by (z, y. 2, f) as the place location and the gripper's orientation around the z-axis. The parameters calculated with respect the object of interest's (on which the picked object will be placed) origin.
push: Parameterized by (æ, y, () as the location and orientation of the placement of the tool (hook) on the table (z = 0) and denotes the length by which the hook will be displaced away from the arm hase The parameters z; y, are w.r.t. the object of interest's (to be pushed) origin. The push distance is the position displacement of the tool in direction
pull: Parameterized by (z, V, 0, r) as the location and orientation of the placement of the tool (hook) on the table (: 0) and denotes the length by which the hook will be displaced towards the arm base The parameters z, y. 0 are w.r.t. the object of interest's s (to be pulled) origin. The pull distance r is the position displacement of the tool in direction A.
An example of the pre-condition and effect of the above skills shown in Figure 3 (left).
16

---

E Task Descriptions
As described in section 5, we evaluate our framework on three task domains (hook reach, constrained packing, and rearrangement push) with three tasks each. In addition, we val- idate the algorithm on more complex skill with longer-horizon action dependency and describe as the fourth task under the domain rearrangement push. Each of the considered suites focuses understanding long-horizon success of one particular skill. For example, hook reach is about the long-term effect of executing hook, while constrained packing focusses place and rearrangement push focuses push. Each task's challenge is directly proportional to the long-horizon action dependency required to complete it. For example, pull affects immediately the next skill is pick. But place affects the next skill after executing one intermediate skill (like pick). Similarly, action dependency after two skills for rearrangement push take We describe all of such considered tasks below.
Hook Reach Task sub-sequence of Figure
Scene: Box is out of workspace, Hook is inside workspace
Goal: Pick the Box
Skeleton: Pick Hook, Pull Box, Place Hook, Pick Box
Hook Reach Task easy version of Figure 9
Scene: Yellow Box is out of workspace, Blue Box inside the workspace, Hook inside workspace, Rack is inside workspace, Rack is empty
Goal: Yellow Box on Rack
Skeleton: Pick Hook, Pull Yellow Box, Place Hook, Pick Yellow Box, Place Yellow Box on Rack
Hook Reach Task 3 shown in Figure 9
Scene: Red Box is out of workspace, Hook inside workspace, Rack inside workspace, Rack already has two blocks (Yellow and Blue)
Goal: Red Box on Rack (without collision)
Skeleton: Pick Hook, Pull Red Box, Place Hook, Pick Red Box, Place Red Box Rack
Initial State
Pick Hook
Pull Red Box Using Hook
Place Hook
Pick Red Box
Place Red Box on Rack
Figure 9: Hook Reach Task
Constrained Packing Task shown in Figure 10
Scene: Three boxes in the workspace, Rack workspace, Blue block on Rack
Goal: All Boxes on Rack (without collision)
Skeleton: Pick Box, Place Box Rack, Pick Box, Place Box Rack, Pick Box, Place Box on Rack
Constrained Packing Task 2 sub-sequence of Figure 11
17

---

Initial State
Pick Yellow Box
Place Yellow Box
Place Red Box
Red Box
Cyan Box
Cyan Box
Figure 10: Constrained Packing Task
Scene: Three boxes in the workspace, Rack is in workspace, Rack is empty
Goal: Three Boxes on Rack (without collision)
Skeleton: Pick Box, Place Box on Rack, Pick Box, Place Box on Rack, Pick Box, Place Box on Rack
Constrained Packing Task 3 shown in Figure 11
Scene: Four boxes in the workspace, Rack is in workspace, Rack is empty
Goal: Four Boxes on Rack (without collision)
Skeleton: Pick Box, Place Box on Rack, Pick Box, Place Box on Rack, Pick Box, Place Box on Rack, Pick Box, Place Box Rack
Initial State
Pick Red Box
Place Red Box
Pick Cyan Box
Pick Blue Box
Place Blue Box
Yellow Box
Yellow Box
Cyan Box
Figure 11: Constrained Packing Task 3
Rearrangement Push Task shown in Figure 12
Scene: Box in workspace, Hook in workspace, Rack outside workspace
Goal: Box under Rack
Skeleton: Pick Box, Place Box, Pick Hook, Push Box using Hook
Initial State
Pick Cyan Box
Place Cyan Box
Pick Hook
Push
Cyan Box using Hook
Figure 12: Rearrangement Push Task
1
Rearrangement Push Task shown Figure
Scene: Three Boxes in workspace, Hook in workspace, Rack outside workspace
Goal: Yellow Box under Rack
Skeleton: Pick Hook, Place Hook, Pick Cyan Box, Place Cyan Box, Pick Hook, Push Yellow Box using Hook
18

---

Initial State
Pick Hook
Place Hook
Place Cyan
Pick Hook
Push Yellow using Hook
Cyan
Figure 13: Rearrangement Push Task 2
Rearrangement Push Task shown in Figure 14
Scene: Four Boxes in workspace, Hook in workspace, Rack outside workspace
Goal: Blue Box under Rack
Skeleton: Pick Red Box, Place Red Box, Pick Yellow Box, Place Yellow Box, Pick Cyan Box, Place Cyan Box, Pick Hook, Push Blue Box using Hook
Initial State
Pick Box
Place Box
Pick Yellow Box
Place Yellow Box
Pick Cyan Box
Place Cyan Box
Pick Hook
Push
Blue using
Figure 14: Rearrangement Push Task
Rearrangement Push Task shown in Figure 15
Scene: Box outside workspace, Hook in workspace, Rack outside workspace
Goal: Box under Rack
Skeleton: Pick Hook, Pull Box using Hook, Place Hook, Pick Box, Place Box, Pick Hook, Push Box using Hook
Inilial State
Pick Hook
Place Hook
Pick
Place Box
Pick Hook
Push Using Hook
Using Hook
Figure 15: Rearrangement Push Task
19

---

Additional Results
One of the attractive aspects of diffusion models is to visualize convergence to valid solution starting from Gaussian noise. We visualize such results and show one of them below.
50
Figure 16: Reverse Diffusion visualization of Rearrangement Push Task 3 for 50 timesteps.
20

---

G
Hardware Experiment Setup
Camera
Figure 17: Hardware Experiment Setup
The experimental setup, illustrated in Fig. 17, encompasses Franka Panda robot and an Intel RealSense camera, several blocks, a rack, and hock. The camera is positioned overhead, facing downward to fully observe the poses of all the objects. AprilTag [61] is employed for SE(3) pose detection of the objects. During planning, the Frankx controller is utilized to generate smooth linear motion toward the desired gripper pose.
H Data Collection and Real-World Experiment Details
We collect transition data in simulation from a random agent. For every selected skill ("pick", "place", "push" and "pull"), we start with a suitable state satisfying its pre-condition and collect successful skill parameters (from random samples) and effect (resulting next state). This is done for scenes containing varied number of objects. We collect around 5000 successful transitions from the simulator. The data is used to train the diffusion model. The success probability prediction module Q(s', s) is also trained on the same data, but we add the failure transitions as well.
For real-world experiments, we use pose detection using AprilTag [61] followed by real-to-sim scene reconstruction. All the experiments are performed with pre-trained diffusion models (trained on simulated data). The closed-loop planning in the real world is performed in the reconstructed scene in the simulator, and planned skill parameters executed directly in the real world. The scene is updated before each replanning phase.
21


Reviews:
[{"review_id": "l1-9lwbjzy", "reviewer": "Reviewer_VAfs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to find skill parameters for skill chaining using a diffusion model given a skill sequence (skeleton) of a target task and skill policies. Thanks to diffusion models, the proposed approach is flexible to additional constraints in any intermediate states or skill parameters, so that a combination of an initial state, intermediate state, final state, and skill behavior can be easily specified for planning. The experiments show that the proposed method works well under various planning scenarios and shows improved success rates.", "summary_of_recommendation": "The proposed approach of combining diffusion models and skill chaining is interesting and novel. The proposed method makes planning with new constraints easy, which makes skill chaining much more flexible for various scenarios. I would recommend weak acceptance for this paper.\n\n----\n\nThe analysis and presentation of the paper could be improved to support the claim of this paper. I would stay with weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- It is not easy to understand Figure 4. A bit more explanation in the caption (e.g. what do rectangles and circles mean? what does color represent? what is skill and what is state?) or labels in the figure would be helpful to understand it.\n\n- In experiments, it is not clear how to collect transition data. Is it granted? or collected from an expert? or with pre-defined skills? Then, what are the skills? How are they defined? It might be trivial from the prior work but for those who are not knowledgeable in the prior work, it would be great if the paper explains it even at a high level.\n\n- In the constrained packing results in Figure 5 and Table 2, it is not clear to me what are the additional constraints imposed on the task.\n\n- Similar to the generalization experiments, it would be great if more detailed information is provided about what task plans are used for training and what are used for testing, and what types of generalization are expected from these experiments. Also, some quantitative results for generalization would be helpful.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The idea of combining skill chaining with diffusion models is novel and makes sense. The use of diffusion models brings better generalization of skill chaining and more flexibility on additional constraints.\n\n- The paper is generally well written but some additional details in the experiments would be appreciated.\n\n- The experiments look solid at a high level but the details are not very clear to me. It would be helpful if the paper describes what each task suite tries to show, why it is challenging, and why things succeed or fail.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "58LeMJ_WZxs", "reviewer": "Reviewer_1DYZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Generative Skill Chaining (GSC) , a skill chaining method for long-horizon manipulation task. GSC is a probabilistic framework that learns skill-conditioned diffusion models and composes their learned distributions based on a predefined skill skeleton. More specifically, the proposed method captures skill's joint distribution of precondition, control parameters, and effect using diffusion model. A complete plan is recovered by the parallel sampling of multiple skill-centric diffusion models. \nGSC has the following advantages:\n1. It doesn't require knowing the true system state and dynamics model of the environment. \n2. It's able to generalize to unseen tasks.\n3. Robustness towards perturbation by replanning.\n\nThe effectiveness of the method is demonstrated on both simulated and physical long-horizon manipulation tasks.", "summary_of_recommendation": "The paper presents a framework to solve skill chaining with generative modeling. Although I have several questions regarding the details of the paper, I think it contains interesting ideas that the community can benefit from. Thus, I recommend accepting the paper.\n\n---\nAlthough it's interesting to see skill-chaining can be attained by sampling multiple diffusion model in parallel, it seems the that method struggles when horizon is long (8 steps). After reading the response from the authors, I would like to stay with my original rating.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- I'm not sure if I fully agree that the test-time optimization based methods are discriminative while GSC is generative, and that's a stark difference. On one hand, you can see optimizing the feasibility of a given plan as optimizing the likelihood of the trajectory, or generate a trajectory that has high likelihood. On the other hand, the underlying generative model of GSC, diffusion model, is based on an iterative sampling strategy that can be seen as optimization with learned gradient. \n\n- Why do you train a unconditional diffusion model for each skill instead of training a single skill-conditioned diffusion model?\n\n- As mentioend above, I have some concerns about sampling in parallel over a long sequence of skills. Does it guarantee to generate samples that follows all the factorized distributions? Does it require more steps than it normally is and how much does the length of the sequence affect this? It would be great if the authors can show theoretically or empirically.\n\n- During the skill sequencing phase, is it necessary to separate the forward and backward feasibility? According to the results, it seems that treating them equally achieves the best performance, which is expected.\n\n-  It'll be nice to show the length of plans for each tasks in table 1, so the readers can get a sense of the difficulties of the task.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The high-level idea of chaining skills by sampling  a sequence of generative models in parallel is very natural and reasonable. By modeling the joint distribution of precondition, controls and effect, planning is framed as sampling from a generative model. The compositionality of the method is impressive, i.e., it can generalize to new plan skeletons without any fine-tuning. It's also pretty flexible, i.e., constraints can be incoporated easily through classifier-based guidance. Additionally, it's amenable to replan.\n\n\nWeaknesses\n- The paper assumes that the plan skeleton is given, i.e., the sequence of skills that the robot should use, which is a very strong assumption. The authors claim that the proposed method is able to generate long-horizon plan, but in reality, it's generating the low-level controls of each skill. I believe knowing which skill to use is also very important and the authors should at least mention it in the limitation section.\n- The tasks shown in the paper are a bit simple. Diffusion model is known for its great capability to model complex distributions, such as images and videos. It's a waste to use it for modeling actions like pushing and pulling.\n- I also have some concerns over the scalability of the method. Since the diffusion models only operate locally, the time required for the goal to be propogated to the start of the trajectory grows linear wrt the length of the skeleton. Therefore, I believe parallel sampling from multiple diffusion models is much harder than sampling from a single diffusion model, especially when the sequence is long. I'm concerned that it may require more sampling steps and even fail to obtain realistic sample.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-tUQAWHn8m", "reviewer": "Reviewer_oxXV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a learning-based method for long horizon planning using sequences of pre-defined skills. The authors propose learning generative models of the state/skill parameter distributions under each skill, training a seperate diffusion model for each skill. The authors propose a method for composing the diffusion models at test time in order to sample feasible sequences of states & skill parameters to solve unseen tasks. The authors demonstrate their approach both in simulation and in real hardware showing improvement over baselines.", "summary_of_recommendation": "Overall I believe this is an interesting paper which is very relevant to the CoRL community and has impressive experimental results. However the paper has several issues that I have outlined above which if fixed, would result in a strong paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Fix mistakes in equations under line 151, line 156 and line 157\n- Add missing details as outlined above", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n\n- The range of experiments in the paper and the results are impressive. The demonstration on real hardware is particularly impressive. \n\n- While the potential composability with diffusion models has been quite well explored in prior work, notably in [40] as cited by the authors but also in [A, B] (which should be cited), the authors have applied these techniques in a way that is particularly relevant to robotic manipulation planning and I believe is novel.\n\nWeaknesses:\n- There is a mistake in both equations under line 151, line 156 and line 157, which makes things unclear. Either the distributions on the LHS should be $p_\\phi(s^{0:2}, a^{0:2}|s^0)$ or the RHS should be marginalizing out $a^0$ and $a^1$. It seems that it is probably the former, given the resulting score functions in equations (3) and (4). \n\n- There is quite a lot of detail missing from the paper which would make reproduction difficult. For instance, while the skills are described in Appendix C, what are their parameterizations and how are they executed? How is the expert data generated for learning the diffusion models? How much data is collected for each? How is the success probability prediction module Q(s', s) trained and does it use the same data as the diffusion models? Are the real-world demonstrations performed with data collected in simulation or in the real world?\n\n- In several places the authors claim to be solving TAMP problems. Given that the skill sequence is given as an input, I don't believe it is reasonable to claim the method solves TAMP problems. If the proposed method can be used to efficiently generate skill sequences for a given candidate skill skeleton then it would very interesting to explore it's use in a system which solves the full TAMP problem, but this is out of scope of the current paper, so it would be better to revise the claims. \n\nAdditional questions:\n- How long does it take to sample from the diffusion model? \n- What proportion of the generated skill sequences are feasible? How many skill sequences need to be generated to reliably ensure the sequence with the highest cumulative probability of success is feasible?\n\n[A] Du, Y., Durkan, C., Strudel, R., Tenenbaum, J.B., Dieleman, S., Fergus, R., Sohl-Dickstein, J., Doucet, A. &amp; Grathwohl, W.S..,  Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMC, ICML 2023\n\n[B] Liu, N., Li, S., Du, Y., Torralba, A., Tenenbaum, J.B., Compositional Visual Generation with Composable Diffusion Models, ECCV 2022\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for chaining skills based on diffusion. The method takes in a sequence of skills for a target task and skill policies. A plan is formed via sampling from composed diffusion models. The approach is flexible in that it can incorporate constraints on intermediate states or skill parameters. The results demonstrate that the method performs well in several scenarios.

Strengths:
- The use of diffusion models allows generalization of skill chaining to additional constraints
- Not requiring foreknowledge of the environment dynamics is an advantage
- The method can respond to perturbations via replanning

Weaknesses:
- The details of the experiments are not very clear (see review comments for specific questions)
- The paper assumes that the plan skeleton is given, but this is not often the case for real-world tasks
- The tasks shown in the paper are not very complex, more challenging tasks should be considered
- Any claims related to solving TAMP problems should be removed, as there is no task-level planning (the plan skeleton is given)

Overall, the rebuttal did not significantly change the reviewers' opinions.
Decision: Accept (Poster)