LLM-based Skill Diffusion for Zero-shot Policy Adaptation
Authors: Woo Kyung Kim
Youngseok Lee
Jooyoung Kim
Honguk Woo
Venue: NeurIPS 2024 poster (2024)
Keywords: Imitation Learning
Planning
Diffusion Model
Large Language Model
Primary area: reinforcement_learning
TL;DR: LLM-based Skill Diffusion for Zero-shot Policy Adaptation

Abstract: Recent advances in data-driven imitation learning and offline reinforcement learning have highlighted the use of expert data for skill acquisition and the development of hierarchical policies based on these skills. However, these approaches have not significantly advanced in adapting these skills to unseen contexts, which may involve changing environmental conditions or different user requirements. In this paper, we present a novel LLM-based policy adaptation framework LDuS which leverages an LLM to guide the generation process of a skill diffusion model upon contexts specified in language, facilitating zero-shot skill-based policy adaptation to different contexts. To implement the skill diffusion model, we adapt the loss-guided diffusion with a sequential in-painting technique, where target trajectories are conditioned by masking them with past state-action sequences, thereby enabling the robust and controlled generation of skill trajectories in test-time. To have a loss function for a given context, we employ the LLM-based code generation with iterative refinement, by which the code and controlled trajectory are validated to align with the context in a closed-loop manner. Through experiments, we demonstrate the zero-shot adaptability of LDuS to various context types including different specification levels, multi-modality, and varied temporal conditions for several robotic manipulation tasks, outperforming other language-conditioned imitation and planning methods.

LLM-based Skill Diffusion for Zero-shot Policy Adaptation
Woo Kyung Kim', Youngseok Lee", Jooyoung Kim', Honguk Woo
Department of Computer Science and Engineering. Sunkyunkwan University
2 Department of Electrical and Computer Engineering, Sungkyunkwan University
(kwk2696, ysley.llle... gs, onsaemiro, hwoo)@skku. edu
Abstract
Recent advances in data-driven imitation learning and offline reinforcement learn- ing have highlighted the use of expert data for skill acquisition and the development of hierarchical policies based on these skills. However, these approaches have not significantly advanced in adapting these skills unseen contexts, which may involve changing environmental conditions or different user requirements, In this paper, we present a novel LLM-based policy adaptation framework LDuS which leverages an LLM to guide the generation process of skill diffusion model upon contexts specified in language, facilitating zero-shot skill-based policy adaptation different contexts. To implement the skill diffusion model, adapt the loss- guided diffusion with a sequential in- in-painting technique, where target trajectories are conditioned by masking them with past state-action sequences, thereby enabling the robust and controlled generation of skill trajectories in test-time. To have a loss function for given context, employ the LLMM-basd code generation with itera- tive refinement, by which the code and controlled trajectory are validated to align with the context in closed-loop manner. Through experiments, we demonstrate the zero-shot adaptability of L.uSS to various context types including different specification levels, multi-modality, and varied temporal conditions for several robotic manipulation tasks, outperforming other language-conditioned imitation and planning methods.
Introduction
Skill-based learning has demonstrated its potentials in generalizing to novel downstream tasks by leveraging pre-trained skills learned from the offline dataset. Furthermore, the integration of skill- based learning and natural language realizes the remarkable ability to perform practical tasks via the provision of a oorinte interface where agents are controlled by instructions describing the goals of the task. Building upon this notion, several previous studies have investigated bridging the gap between the human instructions and physical world manipulation by learning semantically meaningful skills given the language-annotatee dataset [I] 2 3] However, due to the inherently open-ended nature of language, it is impractical to obtain a dataset annotated with a sufficiently wide range of contexts, encompassing various environmental conditions and user requirements, to develop a versatile language-conditiong Consequently, as shown in the left side of Figure [ these prior works limited processing the narrow scope of instructions that primarily convey only the goal of the task without any contextual information (presented as case 1).
policy capable of accommodating such diverse contexts.
To address the challenges in (LLM) policy adaptation approaches that enable zero-shot adaptation to contexts specified in
skill learning, we explore large language model
'Hnnguk Woo is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Test-time Instruction: [Goal] with [Context]
Case 1) Close Drawer
Case 2) Close Drawer A with target speed of 8m/s.
Case 3) Close Drawer with a low power usage.
User
Conventional
LLM-based Policy Adaptation
Langunge-comditooee Skill
Cuse
Slide Puck
Spoed
Druwer
Butten
Ban's
Skill l:mbeddings
Spoed
Low Power
Sma's
Case
Low Power
Skill Decoder
Default Trajoctory
Skill Diffusion Planner
power
Guided Trujcctory Generation
Case
Zero-shot policy adaptation contexts: In case 1, the instruction includes only the task goal. In cases 2 and 3, the instruction is supplemented by the task goal with the context. Conventional language-conditiones skill approaches struggle to generate trajectories well aligned with the contexts, and typically succeed only for instructions as in case 1. Conversely, our LLM- -baddddd policy adaptation approach effectively adapts the contexts in a zero-shot manner across all cases.
Figure I:
language at test-time, Leveraging the effectiveness of diffusion models in controlling their generation process via loss functions [4), and the code generation capability of LLMS [3). we adapt diffusion with guidance by the loss function generated through LLMs. As illustrated in the right side of Figure [ our LLM-based policy adaptation is capable of adapting to diverse contexts in zero-shot manner, such as "target speed of an agent should be 8m/s" or "minimize the power usage of an agent".
To this end, we present novel I.LM-based Skill Diffusion (LDuS) framework, designed facilitate zero-shot adaptation to unseen contexts by generating skill trajectories that are controllable through loss-guided diffusion. Specifically,
devise a hierarchical skill learning structure in which a diffusion model is employed as a skill planner with sequential in-painting. This in-painting method sequentially substitutes consecutive state-action pairs with their originals and learns on the remaining parts, thus allowing for robust trajectory generation by conditioning the trajectory on past experiences. For evaluation, LDuS provides an interface where context specified in language translated into a loss function to guide the generation process of the skill diffusion planner. The generation process is then continuously refined through an iterative process using an LLM as self-critic, ensuring that generating skill trajectory aligns with the given context. As such, our framework stands apart from existing innguage-condttonne skill imitation approaches, as it enables the zero-shot adaptation of skill-based policies to various contexts that extend beyond the training dataset.
The contributions of work are summarized follows.
We present the LDuS framework to address novel challenge of zero-shot policy adaptation to unseen contexts specified in language.
We develop hierarchical skill learning structure that adapts the skill diffusion planner with sequential in-painting, enabling robust skill trajectory generation.
We devise an interface that utilizes LLMs to translate context into loss function, which is then used to control the generation process of the skill diffusion planner. This further validated via iterative refinement to adequately align with the given context.
We experimentally show that LDuS achieves superior performance in zero-shot adaptation to wide range of contexts including different specification levels, multi-modality, and varied temporal conditions for robotic manipulation tasks.
2 Related work
2.1 Language-conditioned skill learning
In the domain of sequential decision- making, several researches have explored techniques for learning language-condittonee skills 6 7 LISA (2) employs hierarchical skill learning to achieve

---

a tanguage--onndition policy through discretized skill codes. Recently, LCD (3), PlayFusion [6]. and SkilIDU [8] commonly adopt diffusion models as a language-condtiioned policy to address high- dimensional vision inputs or to leverage play datasets collected by human. While these approaches primarily concentrate on learning by direct supervision from aims at adapting to unseen contexts that convey varying environment conditions or different user requirements beyond the datasets.
datasets, our L.DuS
Meanwhile, several studies have harnessed the code generation capabilities of LLMS to ground language instructions to actionable skills 99 10
13 [4] For example, in Kinematic-LLM, (14). an LLM is prompted with in-context samples to generate the waypoints for robotic manipulation with pre-defined code primitives. The performance of this in-context learning often depends on the quality and relevance of selected samples, leading to limited generalization to unseen contexts. Unlike this waypoint-basee high-level planning with pre-defined code primitives, our L.DUS adapts the code generation capabilities of L.Mss with iterative refinement to enable the fine-grained control of trajectories, particularly suited for adjusting the generation process of diffusion models.
2.2
Guided control for diffusion models
Diffusion models have shown promising results in various areas including computer vision 15
17
18], offline reinforcement learning (RL) [9 20, 21). and long-horizon planning [22
23
25
The strong generation ability of diffusion models leads to robust adaptability at deployment. In classifier guidance (26). diffusion models controlled at test-time, thereby supporting the generation of images belonging to specific classes. This controlled adaptation concept has been further investigated for text-driven image generation [27] 28 and noisy inverse problems [29 30). Recently, LGD [4] presents a loss-guided diffusion mechanism, by which diffusion models can be controlled via differentiable loss functions without additional training. To facilitate the query- compliant scene generation, CTG (5) leverages LLMs as a loss function generator for user queries.
In the RL domain, such guidance schemes for controlling diffusion-based policies have been investi- gated with pre-trained dynamics models [3T] and value functions [22 23). Yet, these schemes rarely accommodate tanguage-specfffee contexts. Our LDuS is the first to integrate the reasoning capabilities of LLMs and the controlled generation capabilities of diffusion models, thus enabling zero-shot policy adaptation to language-specified contexts in the domain of sequential decision decision-making.
3 Preliminaries
3.1 Problem formulation
Contextual Markov Decision Process (MDP). We consider task as contextual MDP [32, 3
(C, A, P r",r", Po) where e E C is a context space, 8 E S is a state space, a E is an action space, g E is goal space, pe C x S x x 5 - [0, 1] is transition probability conditioned on the context and the goal, r C x $ x A x G + R is a reward function conditioned on the context, y [0, 1] is discount factor, and Po : S > [o, 1] is an initial state distribution. Here, consider a goal to be specified in language. denoted as gr. such as "open the drawer" or "close the door". Moreover, we assume context is provided in language (3) denoted as a, which can affect either the reward function or the transition probability.
Policy adaptation to contexts. We assume access to a dataset D = (TA)iSN. where each trajectory T, is represented as a sequence of state and action pairs with a goal {(8,, ae g)hrrT for a T Teleggh episode without any contextual information.
We consider task evaluation scenarios with a context q; which conveys environmental conditions or user requirements, along with goal g:- Then, our objective is to develop a policy adaptation framework @(g, <t) that maps both goal gi and context e to a policy ne maximizing the return of context-conditioned rewards.
Σ e (st, at, (.)
argmax
E gi C (a: (-10
(1)
3

---

Learning Phase:
(ii Aduptation Phase:
Hierarchical Skill Learning Structure using Skill Dittusion Planner
Contexts
Zero-shot Adoptation to Contexts using LLM
Loss Function Translation
@ Loss-Guided Trajectory Generation
@ Closed-Loup
w/o context labels
Iterative Refinement
Figure Concept of LDuSS with skill learning and adaptation phases
3.2 Diffusion probabilistic models
Diffusion models have been explored for task planning [22 23 24 33), offline RL [19 20 21 and Lunguage-conditoons skill learning [3
6 8] In (22), diflusion model for planning was introduced, which generates h-llngth state and action sequences, denoted as r in a two-dimensional array.
50
Sh
x
= ao
(2)
01 ah
In particular, the diffusion model e( k) based the U-net architecture [34] predicts noise 7 (0, given noise-corrupted trajectory x² as input [15 16). and it optimized through the following loss.
min wwwp///wwwwwww//////.e (1l ,k) n/12
Here, k E [1, K] is denoising step, and x* = va + vI- an is generated by adding Gaussian noise to the original trajectory x with variance schedule parameter a² At sampling, the diffusion model generates trajectory x from random noisy input x² NN( I) by sequentially denoising
(a*
ak e(a kk)
+ +o o"n,
where o² is a parameter for variance schedule.
4
Our approach
4.1
Overall framework
To address the zero-shot adaptation to various contexts, develop the LDuS framework comprising two phases: (i) skill learning via the skill diffusion planner, and (ii) policy adaptation to unseen contexts via LLM-guided diffusion, illustrated in Figure (i) In the learning phase, establish hierarchical structure in which skills are learned, conditioned on a goal. This structure includes skill encoder, skill prior, and skill diffusion planner that generates skill trajectories. These components are learned on the dataset that contains only goals, without any contextual information. Additionally, we employ sequential in-painting technique when training the skill diffusion planner enhance robustness in skill trajectory generation. (ii) In the adaptation phase, we guide the generation process of the skill diffusion planner upon language-speciffee context, by harnessing the code generation capabilities of LLMs. The generated code serves as a loss function that guides the skill diffusion planner to generate skill trajectories at every denoising step. This facilitates alignment between the skill trajectories and the given context. Furthermore, LDuS employs an iterative refinement process, in which generated skill trajectories are repeatedly validated in a closed-loop manner to achieve robust alignment with the context.
4.2 Skill learning via diffusion planner
To facilitate skill learning using diffusion model, adopt variational autoencoder (VAE) 335 architecture with three components: a skill encoder q(2] Stath 41:+)), a skill prior (1(2]se g)), and a skill diffusion planner e (x* k; z). Given an h-llegtth skill trajectory x = (st, ag J.sh. the skill then the skill diffusion planner reconstructs the h-length skill trajectory æ based on z. To optimize both the skill encoder and skill diffusion planner, we employ conditional VAE objective that combines a diffusion reconstruction term in and prior regularization term such as
encoder predicts skill embedding
min Ex- sagg 20.. kx(kk] [llc(2*) Baliaa,,,,,,, k, ml + DKL. (9(2) a p(2)) g.c
(5)

---

Learning Phase: Skill Learning using Diffusion Planner
(ii) Adaptation Phase: M-guide Diffusion
agent's moving speed should be below Sua/s
Wihher
Trajectory
State Sy. Geal jl
Conteal E
Dataset D
Goal
Langraige Fecoder
L
Transition of to Lows Funcliou Queries = (o,
Chala of Thought
Language O1
Skill Encoder Seera
+ Skill Prior (lllbaa)
LLM
fundtionioos, acds retani
Skill Prier Arbellaaee
2 Los-Guuded Skill Trajectory Generation
Skill Diffusian Planner c0xt,..3
Youu S09
Ditfision d- c(s". A.2) ay2 -rr
Gridsnce
Stil Embbelding
3) Closed Chosed-Loop lterative Refnement sutisff user-defined contect function satisff contect
Skill
Skill Diffisson Planner
x
Guided x
Sequential In-painting
y
.(xf.A.e)
Hierarchical Skill Learning
LLM-guided Skill Diffusion Planner
Figure 3:
LDuS framework: In (i), given the dataset annotated with the goals, skills are learned through the hierarchical structure employing a skill diffusion planner with sequential in-painting techniques. In (ii), the context specified in language is translated into loss function, which is then used to guide the generation process of the skill diffusion planner. This process is further validated with closed-loop iterative refinement better align skill trajectories with the given context.
where p(2) is a unit Gaussian (0 I), DKL is the Kullback-Leible (KL) divergence, and 3 is weight for regularization (36.. establish versatile skill embedding space encapsulating common skills across multiple tasks, we use the skill encoder without goal conditions. In addition, to learn skills conditioned on the goal, the skill prior pt, which conditioned on state 8, and goal gt is tailored to align with the output of the skill encoder. To handle the goal provided in language, we also use a pre-trained language encoder & such as CLIP (37) that produces language embeddings of Ii for the skill prior. Then, the skill prior is jointly trained with the skill encoder and skill diffusion planner by minimizing the distance with the skill encoder, i.c.,
min DKL ([(=)/6444-
(g()))
pi
Using the skill prior, we predict an appropriate skill embedding at deployment based on the current state and specified goal.
Sequential in paintiinng. The in-painting technique is adopted in [22] to address goal-conditioned problems, in which diffusion model conditioned by replacing the last state of generating trajectory x with the goal state. We adapt this in-painting to train the skill diffusion planner with sequential replacement mechanism, where sequence of m ~ [1, h] states and actions of x is substituted with the corresponding original state-action pairs learn the remaining portion of x as illustrated in the bottom left side of Figure 3 Then, for evaluation, unlike conventional methods that constrain the diffusion model only with the current state [22 23 our approach constrains the skill diffusion planner using the previously encountered m states. This in-painting method enables the diffusion planner generate more robust and contextually aligned skill trajectories, described in Section 5
4.3 Policy adaptation via LLM-guided diffusion
In general, it is challenging for model to directly acquire the zero-shot adaptation ability for various contexts, particularly when the training dataset has limited coverage on those contexts. To tackle this challenge, we harness the controlled generation capabilities of diffusion models (26) along with the code-generation capabilities of LLMs [5].
In LDuS, for zero-shot adaptation to various contexts, we employ three procedures: translation of given context to its corresponding loss function, loss-guided trajectory generation, and closed-loop iterative refinement, as illustrated in the right side of Figure 3 Initially, the LLM is tasked with translating the context into loss function. This loss function then guides the skill diffusion planner in its trajectory generation process. To ensure the accuracy and relevance of generated skill trajectories, the process is validated by the L.LM a closed-loop manner, querying the LLM to verify whether the generated trajectories meet the specifications of the given context.
5

---

Algorithm Policy adaptation via LLM -ggidedd diffusion
1: Inputs: skill prior (2 9t), skill diffusion planner c(a", k 2), goal Il, context C₁2 LLM PLIM-
guidance weight 8
2: Obtain loss function {(*") using PLIM through 7
3: for every environment step /. do
4: I < pr (z St, .t)
5: while not validate do
6: Sample trajectory x² using e(x², 2) without guidance through
7: Sample guided trajectory & using ((=²) and c(x", k. through
8: Validate whether satisfy the context via LLM as (L _20000 .9,
9: Execute Om in x to the environment
Translation of context to loss function. For converting the context into loss function in code, we employ chain-of- -othuuhtttttttttt prompting (38] with a pre-defined list of queries u = [uI,-., Un]. The queries are designed to capture the specifications of the agent and the desired format of loss functions. Then, the queries are sequentially prompted to the LLM LLLM in conjunction with a goal gi and context Ct, i.c.,
{uz Mj PLLM (gt, Ct, {u,, y Wiee)
where { {us, Vihie represents a set of prompted queries uy and their respective responses ys from the LLM. The final response yn is then used as the loss function to guide the skill diffusion planner.
Loss-guided skill trajectory generation. Similar to prior work [29 4. we implement the loss- gus--g skill trajectory generation, where the guidance is computed as the gradient of the loss function with respect to x* at each denoising step k.
e(x"; x)
5 8 a..,, ((T))
(8)
vos (a* + (1 a²) (e(= 2)) is an approximation of x0 given z 29 and 8 is Here, 00
hyperparameter to modulate the strength of the guidance.
Closed-loop iterative refinement. In the closed-loop iterative refinement, we employ the LLM as self-critic to evaluate both the loss functions and generated trajectories. Specifically, we prompt the LLM with unguided trajectory nº, guided trajectory ε0, goal 9r, and context ct, i.e., (LLM(= 800 St. q). The [.LM then checks for errors. If errors are detected in the loss function, it is regenerated. If there is mismatch between the trajectory and the context, the frequency of the guidance application is increased. This ensures continuous improvement in the accuracy and relevance of trajectories generated by the skill diffusion planner. The process of zero-shot policy adaptation is summarized in Algorithm
Experiments
5.1 Experiment Settings
Datasets. We use the MetaWorld benchmark [39]. specifically with 10 different robot manipulation goals. We also utilize long-horizon goals from the multi-stage MetaWorld, where each goal comprises a sequence of short-horizon manipulation sub-goals. For data collection, we emulate rule-based expert policies. For each goal, we collect 60 trajectories, varying the speed of the agent well the position and weight of the objects being manipulated.
Contexts. We use two context groups: (i) language context where the context is solely specified in language, (ii) multi-modal context where additional information is provided through image input to assist in resolving the given context. These contexts used to direct the agent with instructions such as moving below or above a specific speed, adjusting its speed faster or slower along a specified axis, exerting more or less force heavy objects. The context details are in Appendix A.4.
Evaluation metrics. We use two metrics to assess the zero-shot performance of LDuS and the baselines. Success Rate (SR) quantífies the percentage of goals sub-goals that are successfully completed. Context Reward (CR) evaluates the average reward achieved based on how effectively the models satisfy the given context.

---

Table 1:
Zero-shot performance: The baselines and LDuS trained 10 different manipulation goals for Meta World and 3 different long aag-horizon goals for multi-stage MetaWorld For each manip- ulation goal, use ~ 5 different contexts. The success rate (SR) and context rewards (CR) are measured in 95% confidence interval. Each is evaluated with random seeds for language contexts and random seeds for multi-modal contexts. The highest performance is highlighted in bold.
(a) Performance in Meta World
Without context
Language context
Multi-modal context
Method
SR (%)
CR
SR (%)
CR
SR (%6
LangDT
38. = 7.42%
27.18 + 2.65
33.16 + 7.19%
0.00 + 0.00
0.00 + 0.00%
LISA
11.11 = 6.00%
17.58 + 6.95
10.32 + 4.13%
0.00 1.00::00 0.00
0.00 + 0.00%
LCD
52.98 = 8.90%
37.64 6.10
50.98 + 8.44%
4.38 + 3.00
11.11 + 4.00%
Diffuser
92.16 = 3.37%
57.66
6.20
86.90 3.71%
0.18
0.00 0.00%
LCD + Guidance
42.04 + 4.70
49.95 + 8.35%
22.13 0.87
0.00 + 0.00%
Diffuser Guidance
69.77 +: 4.23
76.42 + 5.26%
33.01 0.52
1.38 + 1.01%
LDuS (ours)
97.00 + 0.73%
87,36 + 3.40
94.60 + 1.70%
63.52 = 4.28
93.05 + 2.29%
(b) Performance in multi-stage MetaWorld
Without context
Language
modal context
Method
SR (%6
CR
SR (%)
CR
SR (%)
LangDT
5.83 +: 2.32%
3.82 + 1.71
4.86 + 1.91%
0.00 + 0.00
4.20 +: 9.30%
LISA
1.17 + 1.25%
30.56 0.00
0.93 + 0.68%
0.00 + 0.00
0.00 :: 0.00%
LCD
35.27 1: 3.23%
32.11 + 7.02
35.84 + 2.51%
34.65 + 13.69
20.85 : 7.94%
Diffuser
35.55 + 3.77%
45.49 4.23
40.66 3.29%
25.00 + 13.79
18.75 # 10.36%
LCD + Guidance
39.37 +: 4.81
35.28 + 2.30%
53.69 + 7.06
29.20 + 6.89%
Diffuser Guidance
45.89 + 3.20
36.67 + 2.81 1%
46.73 + 7.57
18.75 + 7.17%
LDuS (ours)
81.95
2.42%
81.99 +: 2.86
84.03 3.67%
82.03 + 4.20
60.45 8.46%
Baselines. For comparison, we use several lunguage-conditiong imitation and planning methods. 1) LangDT 44) is an imitation learning method that utilizes a language-conditione decision transformer, 2) LISA (2) is a hierarchical skill imitation framework that learns discredited skill codes conditioned on language instructions, 3) LCD (3) hierarchical planning framework that reconstructs state sequences using diffusion model conditioned on language input, 4) Diffuser (22 a task planning framework based on diffusion models.
For LLMs, we use GPT-3.5 [1 which is capable of generating loss functions in the form of
executable code. Moreover, for multi-modal contexts, use GPT-4 [42]. Since these baselines rarely account for zero-shot adaptation to contexts, we adopt the same diffusion guidance used in LDuS for those (i.e., Diffuser and LCD) employing diffusion models. In the cases where the diffusion model is guided by a hand- sand-designed loss function, which is considered optimal, we specify such basclines with the additional label of Guidance.
5.2 Main results
Zero-shot performance. Table shows the performance of LDuS and the baselines (LangD'T, Dif- fuser, LISA, LCD) across three different context input cases (without context, language context, multi- modal context) in Meta Morld and multi-stage MetaWorid. As in Table [(a] LDuS consistently yields the best SR and CR in MetaWorld, outperforming the most competitive bascline Diffueer&Guidan by 18. 88 higher in SR and 78.7% higher in CR for the cases of language contexts. For multi-stage MetaWorld. in Table I(b) LDuS demonstrates superior performance with 41.7% higher in SR and 75.5% higher in CR at average, compared to Diffuser+Guidance
In these experiments, LTngDT and LISA exhibit the lowest performance, even for the cases without contexts. This is attributed to the multi-modality in the dataset, which tends to hinder the learning of policies built with lulti-layer perceptrons or transformers. In contrast, the baselines employing diffusion models, such as Diffuser and LCD, show improved performance. However, none of these
7

---

Table 2: Performance w.r.t various context types
Precise context
Abstract context
Temporal context
Method
CR
SR (%)
CR
SR (%)
CR
SR (%)
LCD Guidance
30.35 +: 6.55
47.96 +: 8, 8.68%
50.18 + 4.17
48.71 = 9.01%
25.01 = 9.02
33.80 =: 381
Diffuser Guidance
65.05 +: 4.79
70.19 + 4.87%
57.33 + 2.05
50.58 = 4.68%
49.17 = 6.59
38.89 :: 0.68%
L.uSS (ours)
89.77 + 3.99
97.72 + 1.08%
75.52 + 3.86
86.98 + 3.00%
79.40 + 3.32
84.26 + 3.79%
baselines achieve robust comparable performance to LDuS for the cases involving language or multi-modal contexts. This limitation arises because the baselines are primarily designed to handle goal descriptions, which is the sole form of language annotation in the dataset. Consequently, they lack the capability to accommodate various contexts that convey environmental conditions or user requirements. Diffuser and LCD, when used with guidance, exhibit slightly improved CR. as the hand-designed optimal loss function can provide context-aligned guidance for trajectory generation. However, in some cases, SR slightly decreases for both the baselines and LDuS when guidance is applied. This decrease occurs because the gradient- basse loss-guidance could generate unexpected trajectories unless the tuning of the guidance weight ö was carefully managed. Overall, LDuS outperforms the baselines in CR by employing the iterative refinement that ensures context alignment, as well in SR by employing the sequential in-painting that allows for robust trajectory generation. While the contexts used in Table a mainly related to speed, we provide additional experiments on different types of contexts, such as energy constraints and spatial limitations, in Appendix C.2.
Various context types. In Table 2 we evaluate the performance across several context types such as precise, abstract, and temporal contexts, while all given contexts are specified in language. Specifically. precise contexts include detailed user requirements, such as a specific target speed, e.g., "the agent speed should move between 5m/s and 6m/s." In contrast, abstract contexts lack specific details. For instance, if the user wants the agent to increase its speed, the abstract context could be phrased as 'I am very busy; the agent needs hurry." Temporal contexts are dynamic and vary over time, which are particularly relevant in long-horizon goals. As shown, the results indicate that LDuS significantly enhances CR, with an increase of 43.7% average compared to Diffuser+Guidance.
Comparing with waypoint generation. In addition to the lcarning- based baselines compared previously, we compare approach with Kinematic-LLM, [14 by which waypoints for pre-defined skill primitives are generated through an LLM with in-context samples. To implement Kinematic- LLM, we define basic skill primitives such as move, push, and pull, and use the same samples for prompting, which are used for LDuS. As shown in Table 3 Kinematic-LLLM shows comparable performance in SR for MetaWorld, but lags in multi-stage MetaWorld. This is attributed to the increased complexity of planning with LLMs for long- horizon goals. Regarding CR, Kinematic-LLM consistently demonstrates lower performance compared LDuS. This result stems from a lack of versatile samples and skill primitives that are necessary to effectively adapt to various contexts.
Table 3: Comparison with waypoint generation method
MetaWorld
Multi-stage MetaWorld
Method
CR
SR (%)
CR
SR (%)
Kinematic-LLM
48.83 + 3.80
95.24 + 1.53%
51.19 + 0.93
61.82 + 3.05%
LDuS (ours)
90.38 + 3.14
95.36 1.57%
75.99 3.81
84.03 + 3.67%
Inference Time. In Table 4 we present the average inference time (in milliseconds) required per timestep for LDuS and the baselines. The measurements are conducted on system equipped with an Intel(R) Core(TM) 19-10980XE CPU and an NVIDIA RTX A6000 GPU, and we use GPT-3.5 for the LLM. As LDuS requires both diffusion sampling time and LLM inference, we measure these component separately, denoted as "Diffusion" and "LLM" in the parenthesis. Diffuser and LCD exhibit the shortest inference time, as these baselines do not require loss guided sampling or LLM inference. When considering only the diffusion sampling time excluding LLM inference, LDuS demonstrates an inference time comparable to the baselines that use the loss guidance. However, the full inference time of LDuS is longer due to its LLM-based code generation and iterative refinement
8

---

process. This overhead can be mitigated by using smaller language model, which can be obtained by distilling the essential knowledge required for LDuS.
Table Inference time required per timestep
Method
Diffuser
LCD
Diffuser+Guidance
LCD+Guidance
LDuS
Inference Time
55ms
56ms
102ms
100ms
108ms(Diffusion) 55ms(LLM)
Skill trajectory coverage. iiluteesl the I-SNE embeddings of h-length trajectories pre- sented in the dataset (yellow-colored dots) and skill trajectories generated by LDuS with guid- ance colue-colored dots) for MeraW. We col- lect successful skill trajectories from LDuS, using different contexts and varying the guidance weight ó ranging from 0.05 to 0.4. This weight setting regulates the strength of gradient application, as described in (8) As observed, the embeddings are expanded via LDuS, specifying that LDuS is ca- pable of rendering novel skill trajectories, which
Trajectory Dataset
Trijectory generated LDuS
dataset
0.05
0.15
0.40
Figure 4: Skill trajectory coverage
are not presented in the dataset but necessary to adapt to different contexts. This demonstrates the versatility of LDuS that arises from the generation capabilities of the diffusion model.
5.3 Ablation study
LLM-guided skill diffusion planner. Table 5 shows the impact of our LLM-guided skill diffusion planner. In this ablation study with MetaWorld we evaluate two variants of LDuS, one without loss guidance and the other without iterative refinement. LDuS achieves improved CR of 37.8% over the variant without the loss guidance for language contexts. also achieves improved CR of 46.8% on average over the variant without iterative refinement. These results underscore the effectiveness of LDuS with loss guidance and iterative refinement for zero-shot adaptation.
Table 5: Ablation on LLM-guided skill diffusion planner
Language context
modtil context
Method
(%)
CR
SR (%)
LDuS
90.38 3.14
95.36 + 1.57%
63.52 + 4.28
93.05 2.29%
Loss Guidance
65.58 + 5.02
95.50 + 0.96%
0.58 + 0.10
0.00 = 0.00%
Iterative Refinement
85.74 + 4.28
94.34 + 1.17%
33.75 + 0.48
4.17 + 1.36%
Sequential in-painting. Figure 5 shows the ef-
fect of our sequential in-painting technique. With multi-stage Meta Morld, the hatched bars denote the performance of LDuS and Diffuser without quential in-painting, while the solid-colored bars indicate that of these models with sequential in- painting. SR and CR are significantly improved for both LDuS and Diffuser, when the in-painting technique is applied. This is attributed to condi- tioning on the past m experiences, which function as long-term memory. This memory feature en- hances performance, particularly for long-bo bong-horizon goals like multi-stage Meta MetaWorld. Furthermore, the performance enhancement is more pronounced for LDuS, as it benefits from learning common skills from the dataset containing trajectories of multiple goals.
without in-paintinn
with in-painting
100
100
45,8%
45,8%
44.2
34.3%
80
80
Success Rate (%)
Context Reward
60
60
20
20
Ditfuser
LDuS
Diffuser
LDuS
Figure 5: Ablation sequential in in-painting

---

Conclusion and limitations
In this work, we presented the LDuS framework for zero-shot skill-based policy adaptation to contexts specified in language. The framework employs a hierarchical structure for skill learning, in which the skill encoder learns task aggnosic skill abstractions and the skill diffusion planner generates various skill trajectories. The skill diffusion planner is enhanced with sequential in in-painting, thus enabling context-aligned trajectory generation for the skills. At test-time, given specific context describing environmental conditions user requirements, L.uSS directly influences the generation process of the skill diffusion planner, allowing for skill-based policies to adapt to the context. This zero- shot adaptation is achieved by combination of refinement, along with the controllable structure of the skill diffusion planner. LDuS stands apart from other iinguage-condttioned approaches, which are limited to certain variations of the instructions present in the dataset and generalize insufficiently to a range of unseen contexts.
loss function generation and iterative
Limitations. LDuS has several limitations which direct us to future work. One limitation is related to the inference time, as LDuS relies on iterative LLM inferences for refinement. This issue could be mitigated by distilling only essential knowledge, such code generation and verification capabilities of an LLM, into smaller language model. Furthermore, since L.DuS relies on the LLM for multiple components, including code generation and iterative refinement, another limitation is its robustness, which can be affected by the variability in the LLLM's performance and the design of the prompts.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITPP) grant funded by the Korea government (MSIT) (No. R8S-2022-11220043 (2022-0- 00043) Adaptive Personality for Intelligent Agents, 0S-2022-11221045 (2022-0-01045) Self-directed multi-modal Intelligence for solving unknown, open domain problems, RSS-0019-11104211, Artificial Intelligence Graduate School (Sungkyunkwan University)), by ICT Creative Consilience Program through the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. Rs-2020-11201822 by the National Research Foundation of Korea (NRF) grant funded by MSIT (No. RS- RS-2023-002131188. by BK21 FOUR Project (No. 0-2024-0580-000), and by Samsung electronics.
References
[1] Suraj Nair et al. "Learning Crowd-Sourced Annotation". . In: Proceedings of the 5th Annual Conference on Robot Learning. 2021.
Conditioned Robot Behavior from Offline Data and
[2] Divyansh Garg et al. "LISA: Learning Interpretable Skill Abstractions from Language". In: Proceedings of the 36th Conference Neural Information Processing System. 2022.
[3] Edwin Zhang et al. "Language Control Diffusson: Efficiently Scaling through Space, Time, and Tasks". In: Proceedings of the 12th International Conference on Learning Representations. 2024.
[4] Jiaming Song et al. "Loss-Guider Diffusion Models for Plug-and- and-Play Controllable Genera- tion". In: Proceedings of the 40th International Conference on Machine Learning. 2023.
[5] Ziyuan Zhong et al. "Language-Guidee" Traffic Simulation via Scene-Level Diffusion". In: Proceedings of the 7th Annual Conference on Robot Learning. 2023.
[6] Lili Chen, Shikhar Bahl, and Deepak Pathak. "PlayFusion: Skill Acquisition via Diffusson from Language-Annotated Play". In: Proceedings of the 7th Annual Conference on Robot Learning. 2023.
[7] Zhaoxun Ju et al. "Rethinking Mutual Information for Language Conditioned Skill Discov- ery on Imitation Learning". In: Proceedings of the 38th Annual Conference on Artificial Intelligence. 2024.
[8] Zhixuan Liang et al. SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Baar -Base Task Execution. 2023. arXiv: 2312. 11598
[9] Wenlong Huang et al. "VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models". In: Proceedings of the 7th Conference on Robot Learning. 2023,
10

---

[10] Wenke Xia et al. "Language to Rewards for Robotic Skill Synthesis". . In: Proceedings of the IEEE International Conference on Roboties and Automation. 2024.
[11] Jacky Liang et al. "Code as Policies: Language Model Programs for Embodied Control". In: Proceedings of the IEEE International Conference Robotics and Automation. 2023.
[12] Kevin Lin et al. Tex2MMtiion: From Natural Language Instructions Feasible Plans. 2023.
arXiv: 2303. 12153
[13] Junting Chen et al. "Language to Rewards for Robotic Skill Synthesis". In: Proceedings of the
62nd Annual Meeting of the Association for Computational Linguistics. 2024.
[14] Wenke Xia et al. Kinematic-aware Prompting for Generalizable Articulated Object Manipula-
tion with LLMs. 2024, arXiv:
[15] Jonathan HO, Ajay Jain, and Pieter Abbeel. "Denoising Diffusion Probabilistic Models". In:
Proceedings of the 34th Conference on Neural Information Processing System. 2020.
[16] Jiaming Song, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models". In: Proceedings of the 9th International Conference on Learning Representations. 2021.
[17] Jonathan Ho and Tim Salimans. Classifier-Fre Diffusion Guidance. 2022. arXiv: 2207
12598
[18]
Manuel Brack et al. "SEGA: Instructing Diffusion using Semantic Dimensions". In: Pro- ceedings of the 34th IEEE/CVF Conference on Computer Vision and Patlern Recognition. 2023.
[19]
Anurag Ajay et al. "Is Conditional Generative Modeling all you need for Decision-Making" In: Proceedings of the 11th International Conference Learning Representations. 2023.
[20] Cheng Lu al. "Contrastive Energy Prediction for Exact Energy-Guine Guided Diffusion Sampling in Offline Reinforcement Learning". In: Proceedings of the 40th International Conference on Machine Learning. 2023.
[21] Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou, "Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning.. In: Proceedings of the 11th International Conference on Learning Representations. 2023.
[22] ZMichael Janner et al. "Planning with Diffusion for Flexible Behavior Synthesis", In: Pro- ceedings of the 39th International Conference on Machine Learning. 2022.
[23] Zhixuan Liang et "AfaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners". In: Proceedings of the 40th International Conference on Machine Learning. 2023.
[24] Chang Chen et al. "Simple Hierarchical Planning with Diffusion". In: Proceedings of the 12th International Conference on Learning Representations. 2024.
[25] Woo Kyung Kim, Minjong Yoo, and Honguk Woo. "Robust Policy Learning via Offline Skill Diffusion". In: Proceedings of the 38th Annual Conference on Artificial Intelligence. 2024. [26] Prafulla Dhariwal and Alexander Quinn Nichol. "Diffusion Models Beat GANs on Image Synthesis". In: Proceedings of the 35th Conference Neural Information Processing System. 2021.
[27] Omri Avrahami, Dani Lischinski, and Ohad Fried. "Blended Diffusion for Text-driven Editing of Natural Images". In: Proceedings of the 33th IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[28) Alexander Quinn Nichol et al. "GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models". In: Proceedings of the 33th IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[29] Hyungjin Chung et al. "Diffusion Posterior Sampling for General Noisy Inverse Problems". In: Proceedings of the 11th International Conference on Learning Representations. 2023.
[30] Hyungjin Chung et al. "Improving Diffusion Models for Inverse Problems using Manifold Constraints". In: Proceedings of the 36th Conference on Neural Information Processing System. 2022.
[31] Fei Ni et al. "MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RL", In: Proceedings of the 40th International Conference on Machine Learning. 2023.
[32] Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual Markov Decision Processes.
2015. arXiv: 1502. 02259
[33] Siyuan Zhou et al. "Adaptive Online Replanning with Diffusson Models". In: Proceedings of the 37th Conference Neural Information Processing System. 2023.

---

[34] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. 2015. arXiv: 04597
[35]
Yoshua Bengio and Yann LeCun. "Auto-Encoding Variational Bayes". In: Proceedings of the 2nd International Conference Learning Representations. 2014.
[36] Irina Higgins al. "beta-VAE: Learning Basic Visual Concepts with Constrained Variational Framework". In: Proceedings of the 4th International Conference on Learning Representations. 2016.
[37] Alec Radford et al. "Learning Transferable Visual Models From Natural Language Supervi- sion". In: Proceedings of the 38th International Conference on Machine Learning. 2021.
[38] Jason Wei et al. "Chain-of-Though Prompting Elicits Reasoning in Large Language Models".
In: Proceedings of the 36th Conference on Neural Information Processing System. 2022.
[39] Tianhe Yu et al. AMeta-World: A Benchmark and Evaluation for Multi-Task and Meta Rein- forcement Learning". In: Proceedings of the 3rd Conference on Robot Learning. 2019.
[40] Lili Chen et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling". In: Proceedings of the 35th Conference Neural Information Processing System. 2021. [41] Tom B. Brown et al. "Language Models are Few-Shot Learners.. In: Proceedings of the 34th
Conference on Neural Information Processing System. 2020.
[42]
OpenAl et al. GPT-4 Technical Report. 2024. arxiv: 2303 08774
[43] Sangwoo Shin et al. "One-shot Imitation in Non-Stationar, Environment via Multi-Modd Skill". In: Proceedings of the 40th International Conference on Machine Learning. 2023.
[44] Alec Radford et al. "Language models are unsupervised multitask learners". In: OpenAl blog
1.8 (2019), p. 9.
[45] Mohsen Soori, Behrooz Arezoo, and Roza Dastres. "Optimization of energy consumption in industrial robots, review". In: Cognitive Robotics 3 (2023), pp. 142-157.
[46] Yecheng Jason Ma al. Eureka: Human-Level Reward Design via Coding Large Language Models. 2023. arxiv: 2310. 12931.
[47] Wenhao Yu et al. "Language to Rewards for Robotic Skill Synthesis", In: Proceedings of the 7th Annual Conference on Robot Learning. 2023.
12

---

A Benchmark environments
In this section, we provide detailed information about our environment settings, dataset collection strategy, and context configurations used for zero-shot evaluation.
A.1 MetaWorld
For evaluation, we use MetaWorld 39 where an agent tasked with manipulating an object to achieve given goal. We use 10 different manipulating goals in Meta World including press button, open window, close window, open door, open drawer, close drawer, pick place cube, insert peg, push cube, open faucet. The left side of Figure iillustrates several manipulating goals in Meta MetaWorld. For zero-shot evaluation, we utilize all manipulating goals for language contexts and select two manipulating goals for multi-modal contexts.
A.2 Multi-stage Meta World
To evaluate long-horizon secenarios, we modify MetaWorld configure the long-horizon goals Each goal in the multi-stage Metaworld consists of four existing MetaWorld sub sob-goals including slide puck, close drawer, push button and insert peg. The agent is then tasked to complete these sub-goals in specified order. We use 3 different long-horizon goals in this multi-stage MetaWorld, each with unique sub-task completion sequence. The goal description for multi-stage Meta World formed by concatenating the descriptions of the sub-goals in the specified order, such as "close drawer and insert peg and push button and slide puck". The right side of Figure @illustrate an example of long-horizon goal used in the multi- stage MetaWorld. For zero-shot evaluation, utilize all three ang-horizon goals for language contexts and one long-horizon goal for modal context.
Press Hucton
Close Window
Open Faucst
Open Dzawer
Slide Puck - Close Drawer - Push Bution - Insert Pog
Meta World
Multi-stage Meta World
Figure 6: Visualization of Benchmark Environments
A.3 Dataset collection
To generate training datasets, we implement rule-based policies for each goal in Metaworld and multi-stage MetaWorld. These rule-based policies are configured using 6 different skill primitives: move along the x-axis, move along the y-axis, move along the z-axis, push, pull, and grab. Fach skill primitive operates based on Proportional-Interall -neiegga----ervvativv (PID) control. To emulate diverse expert behaviors, we vary the speed of the agent, as well as the position and weight of the manipulation objects. For each goal, we collect 60 successful trajectories.
A.4 Context configuration
We configure various contexts specified in natural language to evaluate the zero-shot adaptability of LDuS and the baselines. These contexts are defined based different specification levels (abstract
13

---

and precise), multi-modality and varied temporal conditions. To assess whether LDuS and the baselines align with the given context, we manually design -pnecit-specific reward functions for each scenario.
For precise contexts, concrete requirement is given with numerical value. We define four different precise contexts as follows:
The agent should move at speed slower than æ m/s
The agent should move at speed greater than æ m/s
The agent should move at a speed faster than x m/s but slower than x m/s
The agent should move at speed along the y-axis a minimum of æ m/s.
Here, x is filled with different numerical values, depending on the specific goal and the context. To configure the zero-shot adaptation settings, test the model with speeds different from those used in the training dataset.
For abstract contexts, user requirement is conveyed with some degree of ambiguity. We define two different abstract contexts as follows:
want to be relaxed, but the agent is too distracting.
I'm very busy, the agent should hurry up.
For multimodal context, the agent is required to exert greater force only when in context with the manipulating object, aiming to minimize energy consumption of the end executor. this scenario, the agent utilizes image input to determine whether it is in context with the manipulating object.
The agent should exert a greater speed m/s only when in contact with the object.
For temporally varying context, the context changes over time to one of the precise contexts.
B Implementation details
In this section, we provide the implementation details LDuS and other baselines, along with the hyperparameter settings used for training. All experiments are conduced system equipped with an Intel(R) Core(TM) i9- 10980XE CPU and an NVIDIA RTX A6000 GPU.
B.1 LangDT
implement LTngDT based on Decision Transformer (40) LangDT consists of single decision transformer, built on GPT-2 [44), conditioned by the goal specified in language. Since LangDT is trained solely by conditioning on goals, which are the only language annotations in the dataset, struggles with unseen contexts. Therefore, for zero-shot evaluation, condition the model solely the given goal. The hyperparameter settings for LT are summarized Table Note that short refers to World, while long refers to multi-stage Meta World.
Table 6: Hyperparameter settings for LangDT
Hyperparameter
Value
Total timesteps
le6
Batch size
8 (per goal)
Learning rate
3e-5
Embedding size
128
Hidden size
512
Number of heads
4
Number of layers
2
Planning horizon
8 (short), 16 (long)
14

---

B.2 LISA
We implement LISA using the open source project LISA consists of skill predictor and policy, both implemented with causal transformers. The skill predictor generate discrete skill codes given a goal and current state. Then, the policy generates a h-length skill based on the predicted skill code. Similar to LangDT, LCD is evaluated by conditioning the model solely on the goal. The hyperparameter settings for LISA are summarized in Table
Table 7: Hyperparameter settings for LISA
Hyperparameter
Value
Epoch
2500
Batch size
8 (per goal) le-4
Policy learning rate
Skill predictor learning rate
le-4
Language model learning rate
le-6
Skill horizon
10
Number skills
20
B.3 LCD
We implement LCD using the open source projects Jax i and Haiku LCD consists of diffusion model, which generates a state trajectory given a current state and a goal, and an inverse dynamics mode, which reconstructs actions given pair of states. For zero-shot adaptation, controlled with a loss function, denoted as LCD+Guidance in the main manuscript. The loss functions hand-designed for each context, and loss guidance is applied each denoising timestep of the diffusion model, following the same approach as in the loss-guided trajectory generation process of LDuS. The hyperparameter settings for LCD are summarized in Table 8
implement LCD
Table 8: Hyperparameter settings for LCD
Hyperparameter
Value
Total timesteps
le6
Batch size
(per goal)
Learning rate
3e-5
Denoising timesteps
20
Variance scheduler
cosine
Planning horizon
8 (short), 16 (goal)
B.4 Diffuser
We implement Diffuser [22 using the open source projects Jax and Haiku. Diffuser is butile with a diffusion model that generates a trajectory given a current state and a goal. Similar to LCD, we implement Diffuser controlled with manually designed loss function, which is denoted as Diffuser+Guudance in the main manuscript. The hyperparameter settings for Diffuser are summarized Table 9
B.5 Kinematic-LLM
We implement Kinematic-LLLM (14) using GPT-3.5 41 Kinematic-LLLM consists of a kinematic
knowledge parser and kinematic-aware planner. The kinematic knowledge parser generates an XML file that describes the current state of the environment, including details about the kinematic properties
"htrps//githuu.com/ggggggg https///gghubbboooooogg/ddeeeeep haiku
15

---

Table 9: Hyperparameter settings for Diffuser
Hyperparameter
Value
Total timesteps
le6
Batch size
8 (per goal)
Learning rate
3e-5
Skill embedding size
128 20
Denoising timesteps
Variance scheduler
cosine
Planning horizon
8 (short), 16 (long)
of both the manipulating object and the agent. Following this, the kinematic-aware planner generates an abstract manipulation sequence along with 3D manipulation waypoints. The abstract manipulation sequence provides the description of the procedure, while the 3D manipulation waypoints give the numerical values of the skill primitives in structured format. The LLM prompted with the generated XML file containing current kinematic information, a goal description, the available skill primitive, and sufficient in-context samples. For skill primitives, we use the 6 different skills defined for rule-based expert policy emulation, as described in Section A.3 To evaluate zero-shot performance goal description. Below, we provide
on unseen contexts, the language context is concatenated the prompt used for Kinematic-LLLM.
N Number context Samples
*# In In-context Sample 1
### Goal Description: Faucet Open
=#* Object Kinematic Knowledge:
<hand pos-"0.000 006 0.400 0.195*/>
<joint pos="0.007 0.839 0 0.124 type="hinge" range=" -1.57 1.57" axis="O 1"/> <faucet poa="0.007 6.689 o. 0.124"/>
### Guidance:
First generate abstract sequence, further convert into waypoints
##
Actions for execution:
[MOVE_X] target_x (grab) [MOVE_Y] target_y (grab) [MOVE_2] target_z (grab)
[GRIP]
[PUSH] target_x target_y target_z (grab)
[PULL] target_x target_y target_z (grab)
#* Abstract Manipulation Sequence:
Align with faucet's position (0.689)
2. Go to faucet's position (0.007)
Go smaller value than faucet's position (-0.093)
4. Align with faucet's position (0.
5. Now push the faucet toward front left direction (0.20 207, 0.889,
0. 024)
## Manipulation Waypoints:
[MOVE_Y] Y] 683
[MOVE_X] x] 0.7
[MOVE_X] -9.093
[MOVE_Z] 0.124
[PUSH] 0.207 0. 889 0.124
## In-context Sample 2
16

---

## In-context Sample
It is your turn to generate the waypoints for the specified task. You should fill in Abstract Manipulation Sequence and 3D Manipulation Waypoints sections.
*## Goal Description:
Faucet Open
Object Kinematic Knowledge:
<hand pos="0.006 0.400 0.195*/>
<joint 00.="-0.013 0.840 124 type="hinge" range="-1.65 1.57 axis="O 1"/> <faucet pos=" -0.03 6.690 0.124"/>
##* Guidance:
First generate abstract sequence, further convert into waypoints
Actions for execution:
[MOVE_X] target_x (grab) [MOVE_Y] target_y (grab)
[MOVE_Z] target_z (grab)
[GRIP]
[PUSH] target_x target_y target_z (grab)
[PULL] target_x target_y target_2 (grab)
## Abstract Manipulation Sequence:
### 3D Manipulation Waypoints:
Listing 1: The prompt used for Kinematic-LL.M
B.6 LDuS
We implement LDuS using the open source projects Jax and Haiku. LDuS consists of skill encoder based on an LSTM, skill prior based on MLPs, and a skill diffusion planner. At test-time, the skill prior is used to predict the skill embedding given goal and current state. Then, the skill diffusion planner generates skill trajectory based on the skill embedding. To embed goals specified language, utilize CLIP 37). The hyperparameter settings for LDuS are summarized in Table I0
Table 10: Hyperparameter settings for LDuS
Hyperparameter
Value
Total timesteps
le6
Batch size
(per goal)
Learning rate
3e-5
Input embedding size
128
Skill embedding size
64
Hidden size
128
Denoising timesteps
20
Variance scheduler
cosine
Planing horizon
(short), 16 (long)
Translation of context to loss function. We manually design queries that describe specifications of the agent and the desired format of the loss function. These queries are used in chain-of- (CoT) prompting to guide the LIM in generating a loss function form a given context. We utilize GPT-3.5 [4]. Below, present the prompt used to translate contexts to loss functions for LDuS.
17

---

Query 1: "The shape of the action sequence of the agent is (B, H, 4) where is the batch size, H the number of sequences, and 4 represents the (x, Y, 2, grab on/off) in corresponding agent coordinate of the agent. Explain the configuration the agent's action"
Query 2: "The agent should satisfy the given user requirement The user requirement is given as follows: [context]. Note that the speed of the agent is determined by the L2 norm of the actions. In order to satisfy the user requirement, what should be considered?
Query 3: "Now, generate a loss function that guides the generating trajectory to satisfy the given user requirement. Generated Python based loss function should follow the following format: 'def _loss_fn obs_dim): act oxs,ioobs_dim return loss', where a numpy array representing the action sequences.
Listing 2: The prompt used for generating loss function
Then, the generated loss function is used guide the generation process of our skill diffusion planner. Below, we present an example of code generated by the LLM.
Goal Description
Faucet Open
Context:
The agent should 0.40.
speed faster than 0.38 but slower than Generated Loss Function:
def _loos_fn (x, obs_dim):
x(:,: :bb.;,os__die obs_dim]
speed up. www.iinlgg..o Conaal..nommace.. (act, axis=-1)
min_speed = 0.38
max_speed = 0.40
loss np maximum (speed max_speed, 0) jnp maximum (min_s
speed, 0)
return up.mean (loss)
Listing 3: An example of code generated by the LLM
Loss-guided skill trajectory generation. We implement the loss-guided skill trajectory generation based on previous studies [4] 29]. In practice, instead of predicting noise from a noise-corrupted trajectory = our skill diffusion planner reconstructs the original trajectory x ensure robust generation. Thus, we do not need to explicitly calculate z from x², and the loss-guidance is directly applied to the output of the skill diffusion planner. The frequency guidance applications, denoted as n, is determined by an iterative refinement procedure, where n is gradually increasedfrom pre-defined maximum value until the generated trajectory aligns with the given context. The procedure of loss- poss-guided skill trajectory generation is summarized in Algorithm
Closed-loop iterative refinement. We use LLM to detect errors in the loss function and identify any mismatch between the generated trajectory and the given context. Below, we present the prompt used regenerate the loss function when LDuS detects such errorsn.
Query "The previously generated code has error. Regenerate the [contert]."
Python code. Note that the
requirement
Listing 4: The prompt used to detect error in the loss function
Below, we present the prompt use to detect a mismatch. If mismatch is found between the generated trajectory with guidance and the context, the frequency the guidance application is increased. This validation process is conducted every h steps using the LLM.
18

---

Algorithm 2 Loss-guided skill trajectory generation
1: Inputs: skill diffusion planner c(x", k, 2), loss function e, total denoising timestep K, guidance weight 8, frequency for guidance application n
2: xKK (0, (0,
3: for k = K 1,... 00.0 do
4: 2 40:8m + 80:m
5:
+ c(r 2)
6:
for 1, n.. do
7:
Et 8 8V
2r (( =
8:
1 7) ~~,10,
9:
ak-le Va 11-11- (1-
zb
Ig
é
o
1
1-at
10:
return x"
Query 1:
"The originally generated trajectory is [unguided trajectory] and modified trajectory [trajectory with guidance]. Did the modified action satisfy the given user requirement? The user's requirement is [contert]."
Query 2:
"According to the answer above, did the modified action meet user requirement? Answer with 'Yes' or "N
Listing 5: The prompt used to detect mismatch between generated trajectory and the context
Multi-modal context. For a multi-modal context, LDuS needs to determine whether the agent in contact with the manipulating object from an image input. We utilize GPT-4 [42). as it can concurrentyy process both image and text inputs. If the LLM detects that the agent is in contact with the object, the loss loss- guss-guidnnce is applied. Initially, we generate caption describing the image using the LLM. Then, we prompt the LLM to on the caption. To further enhance the reasoning capability of the LLM, we prompt the LLM with the previous image and the corresponding answer better understand the context of the current image. Below, we present the prompt used for a multi-modal context.
whether the agent is in contact with the object based
Caption "In the image, the red robot
is attempting to close the green drawer with white drawer handle. The robot's has cylindrical blue and white tool attached to its end, which called as an end effector. The background is an indoor setting with a wooden floor and gray wall."
Query "[captionl. [current imagel Given the image, is the robot contacting on the handle of the drawer? [previous image] The asnver for the previous image was "No?."
Listing 6: The prompt used for multi-modal context
C Additional experiments
C.1 Detailed experiment results
Table II and 12 shows the detailed experiment results in MetaWorld and multi-stage MetaWorld respectively, under the conditions where contexts are provided. As shown, LDuS consistently achieves the best performance in SR for all goals. The low performance of LangDT and LISA is attributed to the multi-modal nature of the dataset, which hinders the learning of policies built with MLPs or transformers. LCD and Diffuser mitigate this issue to some extent by employing the diffusion models, but their performance is still lower compared to LDuS. This performance gap is more pronounced in multi-stage Meta World. This because LDuS leverages as sequential in in-painting method, which contributes more robust trajectory generation.
Table 13 and show the detailed experiment results in MetaWorld under precise language contexts, described A.4 In these table, we report the results of LCD and Diffusion with guidance only,
19

---

Table 11: Performance without context in Meta World
Method
Press Button
Open Window
Close Window
Open Door
Open Drawer
LangDT
36.7 + 5.5%
68.3 + 9.7%
83.3 = 8.5%
6.7 + 3.1%
23.3 + 8.6%
LISA
27.8 9.2%
19.4 12.4%
30.6 = 14.8%
8.3 = 4.7%
5.5 6.0%
LCD
31.6 + 12.0%
60.0 11.0%
93.3 = 3.1%
20.0 = 8.8%
36.7 2.4%
Diffuser
96.7 2.4%
100.0 + 0.0%
100.0 = 0.0%
100.0 = 0.0%
93.3 - 4.9%
LDuS (ours)
100.0 0.0%
100.0 + 0.0%
100.0 + 0.0%
80.0 + 0.0%
100.0 0.0%
Method
Close Drawer
Pick Place Cube
Insert Peg
Push Cube
Open Faucet
LangDT
40.0 5.7%
45.0 15.1
3.3 + 2.4%
41.6 = 12.5%
33.3 + 3.1%
LISA
8.3 + 4.7%
0.0 0.0%
0.0 0.0%
0.0 + 0.0%
11.1 + 8.1%
LCD
90.0 +: 3.3%
41.6 14.2%
33.3 14.9%
53.3 + 14.9%
70.0 4.3%
Diffuser
80.0 + 0.0%
83.3 + 8.1%
86.6 = 9.7%
91.7 + 5.3%
90.0 3.3%
LDuS (ours)
100.0 0.0%
96.7 2.4%
93.3 4.9%
100.0 0.0%
100.0 0.0%
Table 12: Performance without context in mtlti-stage Meta WorlW: We abbreviate each goal by using initials of its words (c.g. "Close Drawer" is CD, and "Push Button" is PB)
Method
CD-SP-IP-PB
SP-CD-PB-IP
PB-CD-SP-IP
LangDT
12.5 3.9%
2.5%
0.8 0.6%
LISA
0.7.0.8%%
1.4 1.5%
1.4 1.5%
LCD
40.0 -.7% 3.7%
4 3.4%
22.5 - 2.6%
Diffuser
46.7 1.4%
36.7 1: 5.1%
23.3 1 4.7%
LDuS (ours)
78.3 = 1.4%
75.0 + 3.5%
92.5 + 2.4%
they the most comparable baselines. As shown, LDuS consistently delivers robust zero-shot performance in both CR and SR across all configurations. In multi-stage MetaWorld, the performance fluctuates in the baselines, due their limited ability generate plans for long-horizon goals.
Table 13: Zero-shot performance with language context in Meta World
Context1
Context2
Method
CR
SR (%)
CR
SR (%)
LCD + Guidance
30.6 8.5
55.7 8.8%
34.7 4.5
53.7 1 8.4%
Diffuser + Guidance
64.0 3.0
57.7 + 8.1%
63.5 + 6.6
91.5 + 2.6%
LDuS (ours)
82.1 5.9
94.3 + 1.7%
91.0 = 1.7
96.3 + 1.0%
Context3
Context4
Method
CR
SR (%)
CR
SR (%)
LCD + Guidance
49.1 + 3.5
53.8 + 8.1%
53.8 + 2.3
45.7 + 9.3%
Diffuser + Guidance
73.8 + 3.7
85.1 + 6.4%
77.8 + 3.6
86.7 + 4.7%
LDuS (ours)
86.7 + 3.2
96.7 + 0.8%
89.6 + 2.0
90.0 + 3.6%
Table 14: Zero-shot performance with language context in multi-stage Mcta World
Context1
Context2
Method
CR
SR (%)
CR
SR (%)
LCD Guidance
58.2 + 1.4
35.3 + 2.7%
20.5 + 8.2
35.3 + 1.9%
Diffuser + Guidance
48.1 3.4
10.6 + 1.8%
43.2 = 3.4
62.8 + 3.8%
LDuS (ours)
75.1 11: 3.2
87.0 + 3.2%
88.9 + 2.5
81.0 1 3.6%
20

---

C.2 Performance on different context types
In Table 15 we present additional experiments involving two different context types: energy con- straints and spatial limitations. We conduct these experiment on a single task in MetaWorld. For the energy context, the agent aims minimize its energy consumption by reducing acceleration or deacceleration (45). For the spatial context, the agent is tasked with stying within a specified spatial boundary without crossing it. As shown, LDuS outperforms the baselines CR and SR, demonstrating its scalability cross diverse context types.
However, for complex tasks such as dexterous control, interactions with the environment [46] or AP functions [47] are necessary. This is because LLMs are not inherently grounded in these complex environments, making challenging for them directly generate directly loss functions, We believe that incorporating such approaches will enable LDuS to accommodate more complex tasks, which we plan explore as part of our future research directions.
Table 15: Zero-shot performance with energy and spatial context in MetaWorld
Energy Context
Spatial Context
Method
CR
SR (%)
CR
SR (%)
LCD + Guidance
56.3 0.3
66.7 = 30.8%
29.4 38.8
33.3 0.0%
Diffuser Guidance
62.3 + 0.3
66.7 = 30.8%
75.4 + 37.5
100.0 + 0.0%
LDuS (ours)
88.8 + 0.1
100.0 0.0%
87.0 16.6
100.0 1: 0.0%
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our abstract and introduction include claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include the limitations of our work in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: We include details of our environment settings and implementation with hyperparameter settings in Appendix A and B
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
We submit the code and shows the details of our implementations in Ap- pendix
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
SttpiiiiggiddeeeCCCoooe /ooeeee/eooodubmmiiiaoe ssiocPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We include how we emulate the dataset and how we evaluate the models in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We include the confidence interval in our tables of the main manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
We include the details of our computer resources in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https///aaarrp...pppoa.cccccctea
Answer:
[Yes]
Justification:
We reviewed the NeurlPS Code of Ethies.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed in our work
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include the url of open source projects in Appendix B
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "yktk0LAEcT", "reviewer": "Reviewer_UfZc", "summary": "This paper presents an LLM-based policy adaptation framework for a language specified context. Here, context is a slight variation in how the task is performed. It has two stages, in skill-learning phase, a skill-based diffusion policy is learned with in-painting technique. Here, skill is a VAE encoding of a sequence of states-actions. In adaptation phase, the generation process of diffusion policy is guided with loss-guided diffusion (LGD) mechanism, where loss is generated by LLM conditioned on the context. Experimental results generally show that their skill-based policy along with LLM-generated-LGD is better at context following that other baselines.", "questions": "1. Line 197-198: How's error in the loss function identified if true loss function is not known? Line 638 mentions \"if LDuS detects errors in the loss function\", how is this detected? If this step is not robust then how will it be sure if mismatch is due to loss function or due to guidance weight?\n2. Line 207: If speed variation is already captured in the training data then how is the evaluation ensured too be zero-shot? Details about the range of speed values during data collection and evaluation should be mentioned. t-SNE figure does show OOD generation but does not imply if all evaluation instances were OOD.\n3. LDuS's no-iterative-refinement version outperforms diffuser suggesting skill-based diffusion is contributing a lot to the performance gain. In this regard, [Skill-diffuser] is a more relevant baseline to compare as it also conditions the action diffuser on latent skill computed using lang and image obs. It would have been a more fairer comparison in skill-based category as compared to LCD, as LCD is diffusion in latent skill space with RNN action head. Do you agree with this? If no, why? If yes, why wasn't this baseline compared?\n\n[Skill-diffuser]: https://arxiv.org/abs/2312.11598", "limitations": "Image-based policy experiments are important to ensure real-world applicability of robotic methods and their abscence should have been addressed.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper does a good job of combining LGD's usefulness with controlled generation and capabilities of LLMs to generate context following loss functions for robotic manipulation problem. The experiments/ablations are exhaustive and comparison with diffuser validates most key design choices like skill-conditioning, iterative-refining, in-painting etc). The paper is clear and coherent. Overall, it's a good proof of concept of how knowledge in LLMs can be distilled at test-time in zero-shot manner for controlled generation.", "weaknesses": "1. The scope of \"context\" is only limited to the agent's speed in the experiments. Meaning, the only type of adaptation/user-requirement studied in the paper is speed variation. Generating loss function for this is an easy task for an LLM given we have already seen objective generation on much harder tasks in works like [Eureka] and [Lang-to-Rew]. Perhaps experimenting with other types of context would have offered better insight into scalability of such approaches. \n2. The methods has only been shown to work with low-dimensional state and not with images which raises question on it's real-world applicability. Although for given contexts, proposed framework might work with image-action data (as actions are sufficient to define trajectory for loss generation by LLM) but proof of that is missing. Perhaps an ablation where no low-dimensional state is used elsewhere except in skill-encoder and skill-prior, with only actions in trajectory, could have been performed to study the reliance of performance on state-info in both stages.\n\n[Eureka]: https://arxiv.org/abs/2310.12931 \n[Lang-to-Rew]: https://arxiv.org/abs/2306.08647", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Weakness 1] Scope of Context\nWe thank the reviewer for the thoughtful and constructive feedback. As the scope of context is limited in our experiments, we conduct additional experiments on two different types of user requirements including energy and spatial. In the energy context, the agent aims to minimize its energy consumption by reducing its acceleration or deceleration [1]. In the spatial context, the agent is tasked to not cross a specified spatial boundary. In the table below, we conduct tests on a single task in MetaWorld. As shown, LDuS outperforms the baselines in context reward (CR), demonstrating its scalability on diverse context types.\n\n|Method|Energy Context (CR)|Energy Context (SR)|Spatial Context (CR)|Spatial Context (SR)|\n|---|---|---|---|---|\n|LCD+Guidance|56.3|66.7\\%|29.4|33.3\\%|\n|Diffuser + Guidance|62.3|66.7\\%|75.4|100.0\\%|\nLDuS|88.8|100\\%|87.0|100.0\\%|\n\nMoreover, for complex tasks such as dexterous control, interactions with the environment [2] or API functions [3] are necessary. This is because LLMs are not inherently grounded on these complex environments, making them challenging to  generate loss functions directly. Therefore, we believe that incorporating such approaches (Eureka and Lang-to-Rew) will enable LDuS to accommodate more complex tasks.\n\n[1] Soori, Mohsen, et al. \"Optimization of energy consumption in industrial robots, a review.\" Cognitive Robotics 2023\n\n[2] Ma, Yecheng Jason, et al. \"Eureka: Human-level reward design via coding large language models.\" ICLR 2024\n\n[3] Yu, Wenhao, et al. \"Language to rewards for robotic skill synthesis.\" CoRL 2023\n\n----\n\n## [Weakness 2 & Limitation] Image-based Experiment\nOur LDuS framework can be seamlessly extended to accommodate image inputs, and we conduct an additional experiment in a single task in MetaWorld. To handle image inputs, we first train a ResNet-based autoencoder to obtain the image encoder. Subsequently, we use the image embeddings produced by the encoder as the state for training LDuS,  without using any low-dimensional states.\n\nIn the below table, LDuS+Img denotes the image-based version of LDuS. As shown, LDuS+Img exhibits only a slight performance drop compared to LDuS, demonstrating the applicability for image-based experiment setups. Additionally, there is certainly room for improvement of LDuS+Img, as we do not have enough time to optimize the hyperparameters or test with larger network sizes. We will include these experiments in the final version of our paper.\n\n|Method|Without Context (SR)|Language Context (CR)|Language Context (SR)\n|---|---|---|---|\nLDuS |100.0\\%|92.7|100.0\\%|\nLDuS+Img|83.3\\%|87.5|83.3\\%|\n\n----\n\n## [Question 1] Error Detection in Loss Function\nFor errors due to the loss function, we employ self-critic mechanism of LLMs. This involves prompting the generated code again into the LLM to verify its alignment with the given context, and this verification occurs only at the initial. For error due to the guidance weight, the LLM evaluates whether the guided trajectory requires further modification, and if so, increases the frequency of loss guidance application. This verification is executed for every skill horizon. As mentioned in the response to the first question, for complex tasks, grounding the LLM to the environment is necessary to ensure robust error detection in the loss function.\n\n----\n\n## [Question 2] Experiment Settings for Zero-shot Adaptation\nWe set the target contexts for evaluation to those not present in the training dataset. For example, in the button press task, the training dataset comprises speeds of $[0.22, 0.27, 0.31, 0.34, 0.37]$,  and we test speeds of $[0.32, 0.33, 0.38, 0.40]$ for zero-shot adaptation. As we set different speed configurations for each task, we will include the details of our experiment settings in our final version. \n\n----\n\n## [Question 3] Skill-diffuser Baseline\nGiven the similarities in the overall skill learning structure between Skill-diffuser and LCD, we deem it sufficient to compare only one as the baseline. Instead, we choose to compare with Diffuser and LangDT, which employ different structural approaches, such as decision diffusion-based planner and decision transformer, to highlight the differences between such design choices. Moreover, as the Skill-diffuser also focuses on learning skills by direct supervision from the datasets, we conjecture that it may not be capable of zero-shot adaptation to contexts. We will include the Skill-diffuser as a baseline for our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tFuNgr3OPK", "reviewer": "Reviewer_261B", "summary": "LDuS is a diffusion-based approach for offline skill learning that adopts several advances to improve performance in goal-driven settings. The main contribution is the adoption of LLM-based guidance, allowing to comply with contextual information/conditions, while achieving the given goal.", "questions": "* what is the current inference time and how does it compare with the other approaches?\n* line 461 multimodal, should be multi-modal to align with the rest of the text\n* typo \"Planing\", in Appendix", "limitations": "Limitations are described in the paper, though the authors may consider adding some of the Weaknesses described above (e.g. prompting).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work presents advances that improve skill diffusion and allow to comply with additional contextual information. The strengths of the work are:\n* **Novelty**: the work presents two mechanisms, sequential in-painting and LLM-guided skill diffusion, which are somehow incremental but contribute in a novel way to the field. Given that these topics are of particular interest, I consider the novelty introduced significant.\n* **Presentation**: the work is clearly presented, well-structured and with high-quality figures and tables\n* **Evaluation**: the evaluation of the work is extensive, in terms of tasks, including various context types, in terms of baselines, where I found the \"+guidance\" baselines particularly useful, and in terms of ablation studies performed.", "weaknesses": "The work presents some weaknesses, other than the limitations presented by the authors:\n* **Problem**: the main problem addressed by this work (how to \"stylize behaviours\" given context information) is interesting but it's somehow narrow. Nonetheless, the method seems to perform better also in settings where context is not provided \n* **Prompting**: as many works employing LLMs, the way the LLM is prompted seems to be crucial for correctly guiding the diffusion process. This means that changing the LLM, a new specific prompt structure would be required", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Weakness 1] Problem Definition\nWe believe our work addresses a novel practical problem where the agent adapts to unseen language-specified contexts in a zero-shot manner, even when trained without a context-labeled dataset. Previous works on language-conditioned skill learning [1,2,3] primarily focus on learning through direct supervision from language-annotated datasets. However, obtaining datasets annotated with a broad range of contexts is impractical in real-world scenarios due to the open-ended nature of language. Furthermore, in the realm of RL, while diffusion-based policies have been explored primarily for imitating given datasets [4,5], language-guided test-time control for zero-shot adaptation has not been thoroughly investigated. To the best of our knowledge, our LDuS is the first to integrate LLMs with diffusion models for zero-shot adaptation to language-specified contexts in the domain of sequential decision-making.\n\n[1] Garg, Divyansh, et al. \"LISA: Learning interpretable skill abstractions from language.\" NeurIPS 2022\n\n[2] Zhang, Edwin, et al. \"Language control diffusion: Efficiently scaling through space, time, and tasks.\" ICLR 2024\n\n[3] Chen, Lili, et al. \"Playfusion: Skill acquisition via diffusion from language-annotated play.\" CoRL 2023\n\n[4] Wang, Zhendong, et al. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" ICLR 2024\n\n[5] Chen, Chang, et al. \"Simple hierarchical planning with diffusion.\" ICLR 2024\n\n----\n\n## [Weakness 2] LLM Prompting\nIn order to obtain better responses from LLMs, it is widely known that the prompt should be optimized for each LLMs. However, our primary focus is on addressing the challenge of zero-shot skill adaptation in language contexts, not on developing prompting methods for LLMs. Therefore, there remains room for improving our prompting approach.\n\n----\n\n## [Question 1] Inference Time\nIn the table below, we present the average inference time (in milliseconds) required per timestep for LDuS and the baselines in MetaWorld. For comparison, we also report LDuS-Guidance, which is LDuS without the loss guidance. As shown, LDuS exhibits inference times similar to the baselines when only considering the diffusion sampling time. However, LDuS requires additional time for LLM inference. The LLM inference time can be further reduced by distilling essential knowledge, such as code generation and verification capabilities, into a smaller language model.\n\n|Model|Diffuser|LCD|LDuS-Guidance|Diffuser+Guidance|LCD+Guidance|LDuS|\n|---|---|---|---|---|---|---|\nInference Time|55ms|56ms|56ms|102ms|100ms|108ms (Diffusion Sampling) + 55ms (LLM)|\n\n----\n\n## [Question 2 & 3] Typo \nWe thank the reviewer for identifying the typos. We will correct these typos in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8G4NdO36Se", "reviewer": "Reviewer_MnsC", "summary": "This paper presents LDuS, a framework that adapts skill diffusion models to unseen contexts in a zero-shot manner. The proposed method first couples VAE with a diffusion planner for hierarchical skill learning. To perform zero-shot policy adaptation to the language-specified context, LDuS translates contexts to loss functions using LLM and leverages loss-guided diffusion for controllable trajectory generation. Furthermore, LDuS adopts LLM to iteratively evaluate and improve the alignment between the generated trajectory and the given context. The method is extensively evaluated in two zero-shot settings of MetaWorld, and achieves superior performance in both success rate and context reward compared to the baselines.", "questions": "- Would sequential in-painting increase the training cost (e.g. training time) of the learning phase?\n- How is the context reward defined specifically for each context?\n- In line 228, how is the loss function manually designed for the \"Guidance\" setting?\n- How many times of refinement are usually needed before a generated trajectory is considered contextually aligned? Would that be too time-consuming for long-horizon tasks?\n- How sensitive is LDuS to loss guidance strength?", "limitations": "The authors adequately discussed the limitations of the proposed method.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- LDuS innovatively integrates the reasoning ability of LLM and loss-guided diffusion for policy adaptation and effectively improves the zero-shot performance given unseen contexts.\n- The authors further demonstrate the robustness of LDuS in different context types, and provide more insights on LDuS capabilities through the results of waypoint generation and trajectory coverage.\n- The ablation study highlights the importance of several design choices.\n- The illustration of motivation and methodology is sound and clear, and the paper is well-written and organized.", "weaknesses": "- Tested contexts seem to be mainly limited in speed specifications.\n- More seeds should be tested in experiments to draw more convincing and robust conclusions.\n- The application of loss guidance may harm the model's capability of completing the goal/task (i.e. success rate).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Weakness 1] Scope of Context\nAs the scope of context is limited in our experiments, we conduct additional experiments on two different types of user requirements including energy and spatial. In the energy context, the agent aims to minimize its energy consumption by reducing its acceleration or deceleration [1]. In the spatial context, the agent is tasked to not cross a specified spatial boundary. In the table below, we conduct tests on a single task in MetaWorld. As shown, LDuS outperforms the baselines in context reward (CR), demonstrating its scalability on diverse context types. \n\n|Method|Energy Context (CR)|Energy Context (SR)|Spatial Context (CR)|Spatial Context (SR)|\n|---|---|---|---|---|\n|LCD+Guidance|56.3|66.7\\%|29.4|33.3\\%|\n|Diffuser + Guidance|62.3|66.7\\%|75.4|100.0\\%|\nLDuS|88.8|100\\%|87.0|100.0\\%|\n\n[1] Soori, Mohsen, et al. \"Optimization of energy consumption in industrial robots, a review.\" Cognitive Robotics 2023\n\n----\n\n## [Weakness 2] Random Seed\nIn the below table, we conduct the additional experiments using 5 different seeds for LDuS and the baselines (Diffuser, LCD).  As shown, LDuS consistently outperforms the baselines, demonstrating the robustness of our experiments. Moreover, we conduct 120 different combinations of task and context to draw the result for language context, in our tables. In the final version, we will conduct additional experiments using 5 or more random seeds for all cases. Moreover, we conduct 120 different combinations of task and context to draw the result for language context, in our tables.\n\n|Method|Without Context (SR)|Language Context (CR)|Language Context (SR)|\n|---|---|---|---|\n|LCD+Guidance|52.9\\%|42.0|49.9\\%|\n|Diffuser+Guidance|92.2\\%|69.8|76.4\\%|\n|LDuS|97.0\\%|87.4|94.6\\%|\n\n----\n\n## [Weakness 3] Loss Guidance\nWhile there is potential for loss guidance to impair the model's ability to complete tasks, a similar gradient-based diffusion control approach has been extensively studied across various domains such as vision [1,2,3,4], motion generation [5,6], traffic scene generation [7], and reinforcement learning [8,9,10]. These studies consistently demonstrate its robustness in diverse applications. Moreover, we have shown that our LDuS framework achieves robust performance using loss guidance, further validating its effectiveness.\n\n[1] Ho, Jonathan, et al. \"Classifier-free diffusion guidance.\" NeurIPS 2022\n\n[2] Kwon, Gihyun, et al. \"Improving Diffusion-based Image Translation using Asymmetric Gradient Guidance.\" ICML 2023\n\n[3] Dinh, Anh-Dung, et al. \"PixelAsParam: A gradient view on diffusion sampling with guidance.\" ICML 2023\n\n[4] Liu, Xihui, et al. \"More control for free! image synthesis with semantic diffusion guidance.\" WACV 2023\n\n[5] Song, Jiaming, et al. \"Loss-guided diffusion models for plug-and-play controllable generation.\" ICML 2023\n\n[6] Karunratanakul, Korrawe, et al. \"Guided motion diffusion for controllable human motion synthesis.\" CVPR 2023\n\n[7] Zhong, Ziyuan, et al. \"Language-guided traffic simulation via scene-level diffusion.\" CoRL 2023\n\n[8] Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" ICML 2022\n\n[9] Ni, Fei, et al. \"Metadiffuser: Diffusion model as conditional planner for offline meta-rl.\" ICML 2023\n\n[10] Liang, Zhixuan, et al. \"Adaptdiffuser: Diffusion models as adaptive self-evolving planners.\" ICML 2023\n\n----\n\n## [Question 1] Training Cost\nAs the sequential in-painting requires optimization with respect to $m \\sim [1,h]$, the training time is increased for LDuS at the learning phase. However, since our objective is to enable zero-shot policy adaptation, efficiency at the training time is not our primary concern. Nonetheless, we ensured that all baselines received a sufficient amount of training time to achieve convergence.\n\n----\n\n## [Question 2] Definition of Context Reward\nFor **precise contexts**, where specific user requirements such as speed are quantified numerically, we calculate the difference between the specified and actual values. In cases of **abstract contexts**, where user requirements are expressed ambiguously (without explicit numerical values), we assess context reward by comparing the actual speed with the average speed in the dataset. For **multi-modal context**, where the agent should follow the user requirement only when it makes contact with the manipulating object, we compute context reward by calculating the difference between the specified and actual value when contact is made. Additionally, since the agent should not apply loss guidance without making contact, we impose penalties when the agent attempts to provide loss guidance while not in contact with the object.\n\n----\n\n## [Question 3] Loss Function Design for \"Guidance\"\nThe loss functions are manually designed by the domain experts for each context in the \"Guidance\" setting. To ensure the optimality of the loss functions, we conducted multiple tests and revisions to effectively reflect the given contexts.\n\n----\n\n## [Question 4] Number of Iterative Refinement\nThe refinement process is required between 0 to 4 depending on the context and the tasks. Moreover, our iterative refinements are executed for every skill horizon. For long-horizon tasks, we set the skill horizon at 16 steps, compared to 8 steps for short-horizon tasks. Consequently, the time required for iterative refinement is not significantly increased for long-horizon tasks.\n\n---- \n\n## [Question 5] Sensitivity to Guidance Strength\nLDuS is less sensitive to the loss guidance strength, as it employs iterative refinement to control the frequency of guidance application. A small guidance weight is preferable, because it allows for more detailed control. However, a small guidance weight may necessitate a more iterative refinement process to meet the specified context, presenting a tradeoff between detailed control and increased inference time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSNT1LynhZ", "reviewer": "Reviewer_UqKP", "summary": "This paper presents a novel framework called LLM-based Skill Diffusion (LDuS), designed to enable zero-shot skill-based policy adaptation to various contexts specified in natural language. It leverages LLMs to guide a skill diffusion model, allowing for the generation of adaptable skill trajectories. The framework uses a hierarchical skill learning structure and a loss-guided diffusion process with sequential in-painting techniques. The authors demonstrate the effectiveness of LDuS in adapting to diverse contexts through experiments on robotic manipulation tasks, showing improved performance compared to other methods.", "questions": "1. What is the justification for Equation (5)? \n2. Please explain the seeding procedure.", "limitations": "The authors briefly discuss one limitation of their work in line 320. However, more discussion is needed given the multiple components and heavy reliance on LLMs for tasks such as encoding the goal with CLIP or encoding the context with GPT. Additionally, potentially negative societal impacts are not discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The approach is interesting and tackles an important problem. The method appears to be novel to my knowledge on the topic.\n\n* The method is tested on various challenging tasks and achieves superior performance compared to several baselines, demonstrating its effectiveness.", "weaknesses": "### Major comments: \n* The combination of a diffusion model and a VAE to learn the skills from a dataset of trajectories is not well justified. In particular, Equation (5), which is the loss function for learning the skills, informally replaces the reconstruction loss in ELBO of VAE with the noise prediction loss of a diffusion model. It is unclear if this is mathematically and conceptually correct. If this equation is adapted from a paper, it should be cited. Otherwise, more justification for this combination is needed. The authors are encouraged to look into works that combine these two models, such as:\n\nKingma, Diederik, Tim Salimans, Ben Poole, and Jonathan Ho. \"Variational diffusion models.\" Advances in neural information processing systems 34 (2021): 21696-21707.\n\n\n* The seeding procedure is unclear and possibly incorrect. In the caption of Table 1, the authors state that the methods are \"evaluated\" on 3 seeds. This suggests that they trained on a single seed and evaluated on 3 seeds, which is not the correct use of seeding. The proper approach is to train and evaluate for each seed. Additionally, more seeds should be used to ensure robustness. While computation costs are a concern, using at least 5 seeds, and preferably 10, is recommended.\n* The contribution of the paper does not appear to be significant. It combines multiple components into a framework. In some cases these combinations are not well-justified either (e.g., VAE+diffusion mentioned above). While I understand the results are promising, I have concerns about the significance of the contributions.\n\n\n### Minor comments:\n* Notation fix: In Equation 1, the action sampling needs to be added to the subscript of the expectation: $a_t \\sim \\pi_c(\\cdot | s_t, g_t)$. \n\n* Given that VAE is one of the main components of the method, its paper should be cited: \n\nKingma, Diederik P., and Max Welling. \"Auto-encoding variational bayes.\" arXiv preprint arXiv:1312.6114 (2013).\n\n* The cross-reference to Table 3 in the related works section is out of place. Table 3 presents empirical results, and referring to it during the literature review breaks the coherence of the paper and the order of cross-references.\n\n* A bit nit picky, but the abbreviation \"LDuS\" for \"LLM-based skill diffusion\" is unclear and should be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## [Weakness 1 & Question 1] Mathematical Justification for VAE+Diffusion\nWe appreciate the reviewers for addressing concerns about the mathematical justification LDuS. We provide the mathematical justification for Equation (5), where the reconstruction loss in the ELBO is replaced with the diffusion loss. Due to length constraints in the rebuttal text, **the complete proof is included in the PDF of the global rebuttal**.\nWe start from the ELBO formulation for VAEs [1],\n\n$$\\mathop{\\mathbb{E}}\\_{x_T \\sim q(x_T|x,z), z \\sim q(z|x)} [\\log p(x)] \\geq \\underbrace{\\mathop{\\mathbb{E}}\\_{x_T \\sim q(x_T|x,z), z \\sim q(z|x)} \\left[\\log p(x|z)\\right]}\\_{\\text{reconstruction loss}} - \\underbrace{D_{\\text{KL}}(q(z|x)||p(z))}\\_{\\text{regularization loss}}$$\nThen, the reconstruction error term satisfies the following,\n\n$$\\mathop{\\mathbb{E}}\\_{x_T \\sim q(x_T|x,z), z \\sim q(z|x)} [\\log p(x|z)]  \\geq \\mathop{\\mathbb{E}}\\_{x_T \\sim q(x_T|x,z), z \\sim q(z|x)} \\left[\\log p(x_T)\\right] + \\underbrace{\\mathop{\\mathbb{E}}\\_{x_T \\sim q(x_T|x,z), z \\sim q(z|x)} \\left[ \\log \\frac{p(x|x_T,z)}{q(x_T|x,z)}\\right]}\\_{\\text{diffusion loss}}$$\nwhere $T$ is the total denoising timestep. Thus, the reconstruction term can be seamlessly replaced with the diffusion loss. Moreover, on how the diffusion loss term is optimized with the noise reconstruction term, please refer to the original DDPM paper [2]. We will include this theoretical analysis in the final version.\n\nMoreover, our LDuS framework is structurally different from the approach presented in the \"Variation Diffusion Models (VDM)\" [3] paper. In VDM, a diffusion model itself functions as a VAE. In contrast, LDuS utilizes an encoder-decoder structure, where an encoder generates skill embeddings from trajectories, and a diffusion-based skill planner produces skill trajectories conditioned on these embeddings. Moreover, using VAE structure to learn skill embeddings and decode skills is a common practice in skill-based RL [4,5].\n\n[1] Kingma, Diederik P., and Max Welling. \"Auto-encoding variational bayes.\" arXiv 2013\n\n[2] Ho, Jonathan, et al. \"Denoising diffusion probabilistic models.\" NeurIPS 2020\n\n[3] Kingma, Diederik, et al. \"Variational diffusion models.\" NeurIPS 2021\n\n[4] Pertsch, Karl, et al. \"Accelerating reinforcement learning with learned skill priors.\" CoRL 2020\n\n[5] Hakhamaneshi, Kourosh, et al. \"Hierarchical few-shot imitation with skill transition models.\" ICLR 2022\n\n----\n\n## [Weakness 2 & Question 2] Random Seed\nWe used 3 different seeds to **train** and evaluate our model. To avoid misunderstanding, we will revise the caption in Table 1.\n\nIn the below table, we report the performance of LDuS and the baselines on 5 random seeds. As shown, LDuS consistently outperforms the baselines, demonstrating the robustness of our experiments. In the final version, we will conduct experiments using 5 or more random seeds for all cases.\n\n|Method|Without Context (SR)|Language Context (CR)|Language Context (SR)|\n|-|-|-|-|\n|LCD+Guidance|52.9\\%|42.0|49.9\\%|\n|Diffuser+Guidance|92.2\\%|69.8|76.4\\%|\n|LDuS|97.0\\%|87.4|94.6\\%|\n\n----\n## [Weakness 3] Contribution of LDuS\nWe believe our work addresses a novel practical problem where the agent adapts to unseen language-specified contexts in a zero-shot manner, even when trained without a context-labeled dataset. Previous works on language-conditioned skill-learning [1,2] primarily focus on learning through direct supervision from language-annotated datasets. However, obtaining datasets annotated with a broad range of contexts is impractical in real-world scenarios due to the open-ended nature of language. Furthermore, in the realm of RL, while diffusion-based policies have been explored primarily for imitating given datasets [3,4], language-guided test-time control for zero-shot adaptation has not been thoroughly investigated. To the best of our knowledge, our LDuS is the first to integrate LLMs with diffusion models for zero-shot adaptation to language-specified contexts in the domain of sequential decision-making.\n\n[1] Garg, Divyansh, et al. \"LISA: Learning interpretable skill abstractions from language.\" NeurIPS 2022\n\n[2] Zhang, Edwin, et al. \"Language control diffusion: Efficiently scaling through space, time, and tasks.\" ICLR 2024\n\n[3] Wang, Zhendong, et al. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" ICLR 2024\n\n[4] Chen, Chang, et al. \"Simple hierarchical planning with diffusion.\" ICLR 2024\n\n----\n## [Weakness 4,5,6] Minor Comments\n* Subscript in Equation 1: We will add the subscript for the Equation 1. \n* Citation: We will cite the paper \"Auto-encoding variational bayes\" in Section 3.2.\n* Cross-Reference: We will remove this reference to Table3 in the related works, to enhance the coherence of our paper.\n----\n## [Weakness 7] Abbreviation of LDuS\nAs the reviewer mentioned, the abbreviation \"LDuS\" is unclear. Initially, we intended to name our model 'LSDu', based on the order in \"LLM-based Skill Diffusion\".  However, to avoid the association with the acronym 'LSD', which is commonly known for the drug, we rearranged the letters, resulting in the final model name 'LDuS'. \n\n----\n## [Limitations] Reliance on LLMs\nThe LDuS framework relies on LLMs for its performance; however, it maintains robustness through an iterative refinement process, with its effectiveness detailed in Table 4. For goal description, using RoboCLIP [1], a vision language model trained on robot manipulation tasks, would improve the representation of the goal descriptions. Additionally, the use of LLMs to control the generation process of diffusion models raises concerns about potential negative societal impacts, such as the possibility of malicious prompts being injected and used for harmful purposes. We will discuss these concerns in the limitation section and provide potential negative social impacts.\n\n[1] Sontakke, Sumedh, et al. \"Roboclip: One demonstration is enough to learn robot policies.\" NeurIPS 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We deeply appreciate the valuable and constructive feedback from the reviewers. We addressed all identified weaknesses and questions to resolve the concerns raised. For those that require additional experiments, we have conducted further studies to provide a more comprehensive understanding. Lastly, we provide mathematical justification for Equation (5) in the attached PDF.

Meta-review: This paper presents an LLM-based policy adaptation framework for language-specified context in robotic tasks. 
It combines a skill-based diffusion policy learned with in-painting, All reviewers agree that this is an innovative integration of LLM reasoning and loss-guided diffusion, and the paper demonstrates robustness across context types, with a clear presentation. The approach shows good performance  in skill-based learning task with context-specific instructions, particularly for speed variations in robotic manipulation tasks. Although, limitations exist in prompt dependency. 

However the rebuttal did a good job of providing clarifications with more than one reviewers updating the scores leaning more positively.  All reviewers lean accept after rebuttal phase, and the Metra-reviewer concurs on the overall agreement. . 
The authors are advised to review the final comments, and update the manuscript accordingly.. In addition to including clarifications in the main paper, the authors are also advised to describe the possible limitations of this work in more detail.
Decision: Accept (poster)