Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks
Authors: Edwin Zhang
Yujie Lu
Shinda Huang
William Yang Wang
Amy Zhang
Venue: ICLR 2024 poster (2024)
Keywords: planning
diffusion
language
RL
reinforcement
Primary area: reinforcement learning
TL;DR: We scale diffusion models for language conditioned planning.

Abstract: Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have allowed for improved scaling along one or two of these axes, but are still computationally prohibitive to use. In this paper, we propose to address all three axes by leveraging Language to Control Diffusion models as a hierarchical planner conditioned on language (LCD). We effectively and efficiently scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long horizon control problems conditioned on natural language instructions, as a step towards generalist agents. Comparing LCD with other state-of-the-art models on the CALVIN language benchmark finds that LCD outperforms other SOTA methods in multi-task success rates, whilst improving inference speed over other comparable diffusion models by 3.3x~15x. We show that LCD can successfully leverage the unique strength of diffusion models to produce coherent long range plans while addressing their weakness in generating low-level details and control.

Published as a conference paper at ICLR 2024
LANGUAGE CONTROL DIFFUSION: EFFICIENTLY SCAL- ING THROUGH SPACE, TIME, AND TASKS
Edwin Zhang Harvard, Founding
Yujie Lu,, Shinda Huang, William Wang University of California, Santa Barbara
Amy Zhang UT Austin
ABSTRACT
Training generalist agents is difficult across several axes, requiring to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have allowed for improved scaling along one or two these axes, but are still computationally prohibitive to use. In this paper, we propose to address all three axes by leveraging Language Control Diffusion models as a hierarchical planner conditioned on language (LCD). We effectively and efficiently scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long horizon control problems conditioned on natural language instructions, step towards generalist agents. Comparing LCD with other bate-of-the-an models on the CALVIN language robotics benchmark finds that LCD outperforms other SOTA methods in multi-task success rates, whilst improving inference speed over other comparable diffusion models by 3.3x- 15x. We show that LCD can successfully leverage the unique strength of diffusion models to produce coherent long range plans while addressing their weakness in generating low-level details and control.
INTRODUCTION
It has been longstanding dream of the Al community be able create household robot that can follow natural language instructions and execute behaviors such as cleaning dishes or organizing the living room (Turng Haugeland] 1950 Bischoff & Graefe 1999). Generalist agents such as these are characterized by the ability to plan over long horizons, understand and respond to human feedback, and generalize to new tasks based on that feedback. Language conditioning is an intuitive way to specify tasks, with a built-in structure enabling generalization to new tasks. There have been many recent developments to leverage language for robotics and downstream decision- makinng and control, capitalizing on the recent rise of powerful reasoning capabilities of large language models (LLMs) (Ahn et al. 2022 Liang et al. 2022 Diang et 2022 Li al., 2022a)..
However, current methods have few pitfalls. Many existing language-conditioned control methods utilizing LLMs assume access to a high-level discrete action space (e.g. switch on the stove, walk to the kitchen) provided by lower-level skill oracle (Ahn et al. 2022 Huang et al. 2022 Jiang et al. 2022 ei al. 2022a). The LLM will typically decompose some high-level language instruction into of predefined skills, which then executed by control policy oracle. However, fixed set of predefined skills may preclude the ability to generalize to novel environments and tasks. Creating wagguugg-conditionnee. policy that performs effective direct long-horizon planning in the original action space remains an open problem.
One promising candidate that has emerged for direct long-horizon planning is denoising diffusion models. Diffusion models have recently been proposed and successfully applied for low-dimensional long-horizon planning and offline reinforcement learning (RL) (Janner et al., 2022; Wang al.
20226 Chi et al. 2023), beating out several prior Transformer or MLP- models Chen et al. (2021): Kumar et al, (2020). particular, Diffuser (Janner et al.) (202) has emerged as an especially
promising planner for low-dimensional proprioceptive state spaces with several properties suited for
Code and visualizations available httphhtp///// http:///.d.edddi.... available this URL.
Our task is best exemplified by videos

---

Published as a conference paper at ICLR 2024
Latent Plan
Low Level Policy Encoder
$0
8c
$2c
1.
||x x1|
Sampling algorithm (forward process)
2.
Latent Plan (with noise)
80
80
Sc $20
Latent Plan
(denoisedd
Time (H1
3.
Temporal U-Net
State Trajectory (RGB Video)
Prompt em embedding from text encocer
Denoising (backward process)
An overview of our high-level policy training pipeline. The frozen low-level policy encoder used to encode a latent plan, or a subsampled sequence of RGB observations encoded into a lower dimensional latent space (1), which will be used later on as goals for the goal-conditioned low-level policy (LLP). We then noise this latent plan according to uniformly sampled timestep from the diffusion process' variance schedule (2), and train Temporal U-Net conditioned on natural language embeddings from frozen upstream large language model reverse the noising process (3), effectively learning how to conditionally denoise the latent plan, To train U-Net, one can simply use the p-norm between the predicted latent plan and the ground truth latent plan as the loss (4). We use 1 in practice following Danner et al. (2022).
Figure I:
language conditioned control: flexibility in task specification, temporal compositionality, and ability scale long-horizon settings.
Nevertheless, Diffuser does not work directly out of the box when scaling to pixels. This is perhaps unsurprising, as Diffuser in high-dimensional image space is effectively text-to-video diffusion model, and even internet-scale video diffusion models have demonstrated only a mediocre understand- ing of physics and temporal coherence over fine details (Ho et al. 2022a Singer et al., 20022). This is because the training objective for generative models has no notion of the underlying task, meaning they must model the entire scene with equal weighting. This includes potentially task-irrelevant details that may hinder or even prevent solving the actual control problem (Zhang et al.) [2200] Wang jet al.. (022a)), which can lead to catastrophic failure in control where fine details and precision are essential. Additionally, training video diffusion model is computationally expensive and may take several days or weeks to train, which leaves such an approach out of reach to most researchers (Rombach et al. (022). In summary, diffusion models are computationally prohibitive run high-dimensional input spaces and also tend to be inaccurate at control in high nigh-dimensional image space.
In this paper, we aim to take a step towards training practical generalist agents. We address all three issues of direct long-horizon planning, non task-specific representation, and computational inefficiency by proposing the usage of a hierarchical diffusion policy[] By utilizing a diffusion policy, can avoid manual specification of fixed skills while keeping flexible long-horizon generation capabilities that other architectures may not have. In addition, we solve both the representation and efficiency issue of a naive application of diffusion by applying the frozen representation of a goal-conditioned policy as low-level policy (LLP) (see Step in Figure and more details in Appendix C).
In consequence, this hierarchical approach allows us to scale the diffusion model along three orthogo- nal axes: the Spatial dimension through a dineeseniional representation that has been purposely optimized for control, the Time dimension through temporal abstraction enabled by utilizing goal-conditioned low-level policy (LLP), as the LLP can use goal states several timesteps away from the current state, and the Task dimension through language, the diffusion model acts as powerful interpreter for plugging any large language model into control. In addition, the entire pipeline is fast and simple train, as well as a low-dimensional representation for generation. We can achieve an average of 88.7 7% success rate across all tasks on the challenging CALVIN language-conditioned robotics benchmark.
utilize DDIM (Song al. 2020a), temporal abstraction on the horizon,
'To he clear, referring to hierarchical policy where the high-level policy is diffusion model, not hicrarchical diffusion model that utilizes multiple diffusion models scale generation (Ho 2022a
2

---

Published as a conference paper at ICLR 2024
Additionally, we elucidate where diffusion models for text-to-control work well and highlight their limitations.
In summary, core contributions are: 1) We propose effective method for improving difffusion policies' scaling to high-dimensional state spaces, longer time horizons, and more tasks by incorpo- rating language and by scaling to pixel-based control. 2) We significantly improve both the training and inference time of diffusion policies through DDIM, temporal abstraction, and careful analysis and choice of image encoder. A successful instantiation of our language control diffusion model that substantially outperforms the state of the art on the challenging CALVIN benchmark.
2 BACKGROUND
Reinforcement Learning. We formulate the environment as a Markov decision process (MDP) M = (S, AA, R, 7,P), with state space S, action space A, reward function R. discount factor 7, and transition dynamics p. At each time step t, agents observe a state s e S C R", take an action E A C R"" and transition of RI is then to learn either a deterministic policy क. S + A or a stochastic policy a ~ I (+|ss) that maximises the policy objective, [TT Bawwsrss' Σ 2t=07 7'
state s' with reward following s', r ~ p(-,-ls, a). The goal
Language Conditioned RL. We consider a language conditie-on RL setting where we assume that the true reward function R. is unknown, and must be inferred from natural language instruction L E L. Formally, let F be the function space of R. Then the goal becomes learning an operator from the language instruction to reward function v : L + F. and maximizing the policy objective conditioned on the reward function v²(); (- s. R) = Eawzrrs' Et=g 7'r,, This formulation can be seen as a contextual MDP (Hallak al., 2015), where language is seen as a context variable that affects reward but not dynamics. We assume access to prior collected dataset D N annotated trajectories т; = ((50, ao ...ST), L.). The language conditioned policy Ty, or the behavior policy, is defined to be the policy that generates the aforementioned dataset. This general setting can be further restricted to prohibit environment interaction, which recovers offline RL or imitation learning (IL). In this paper, we assume access to a dataset of expert trajectories, such that Ty = optimal policy πτ. Although this may seem like a strong assumption, the setting still poses a challenging learning problem as many unseen states will be encountered during evaluation and must be generalized to. Several prior methods have failed in this setting (De Haan et al. 2019 Ross et al.) 2011
Goal Conditioned Imitation Learning and Hierarchical RL. Goal-conditioned reinforcement learning (RL) is subfield of RL that focuses on learning policies that can achieve specific goals or objectives, rather than simply maximizing the cumulative reward. This approach has been extensively
studied in various forms in the literature (Li et al. 2022a| Arora & Doshi 2018, [odhani al..
2021 Harrison et al. 2017: Schaul et al.] 2015 Rosete-Beas et al.) 2023). although we focus on the hierarchical approach (Sutton et al. 1999). Following (Nachum et al. 2018), we formulate the framework in a two-level manner where high level policy Thi(āls) samples a goal state g ā every e time steps. However, whilst most prior work only considers generating a single goal forward, we can leverage the unique capabilities of diffusion models to do horizoo planning several goal steps forward (Janner et al.) (022). We refer to c as the temporal stride. deterministic low level policy (L.PP) Ti (8t, Zt) then attempts reach this state, which can be trained through hindsight relabelling (Andrychowicz: et al.
2017) the offline dataset D. In general this LLP will minimize -oo (Sc. g = [ For our specific
action reconstruction error, or LBC EL ae.ete [l|a
training objective, please see Appendpii C
3 THE LANGUAGE CONTROL DIFFUSION (LCD) FRAMEWORK
In this section, we develop the Language Control Diffusion framework to improve generalization of current language conditioned policies by avoiding the usage of predefined low level skill oracle. Furthermore, we enable scaling to longer horizons and sidestep the computational prohibitiveness of training diffusion models for control directly from high-dimensional pixels. We start by deriving the high-level diffusion policy training objective, and go on to give a theoretical analysis on the subopti- mality bounds of our approach by making the mild assumption of Lipschitz transition dynamics. We
3

---

Published as a conference paper at ICLR 2024
then solidify this theoretical framework into practical algorithm by describing the implementation details of both the high-level and low-level policy. Here we also detail the key components of our method that most heavily affect training and inference efficiency, and the specific model architectures used in our implementation for the CALVIN benchmark (Mees et al. 2022b).
3.1 HIERARCHICAL DIFFUSION POLICIES
High-level Diffusion Policy Objective. We first describe our problem formulation and framework in detail. Since we assume that our dataset is optimal, the policy objective reduces to imitation learning:
min RARDD [Dx. (Fa(- s, R) R))]
(1)
As we are able to uniquely tackle the problem from planning perspective due to the empirical generative capabilities of diffusion models, we define a state trajectory generator P and switch the atomic object from actions to state trajectories T = (sp. ,.,,.., 8T) ~ P(-(R). Thus we aim to minimize the following KL:
min DEL (Pp(T) R), P(T R)) = min R-D P
R-D [log Pp(T (T RR) log P(T [R]]
(2)
P
This can be reformulated into the following diffusion training objective:
min Ero.elle, calVo,To + v1 - a,c,,t)11))
Here refers uniformly sampled diffusion process timestep, e, refers to noise sampled from (0, I), To refers to the original denoised trajectory т, a denotes a diffusion variance schedule, and o = II 1 0z.. We refer to Appendix for a more detailed derivation of our objective.
Near Optimality Guarantees. To theoretically justify our approach and show that can safely do diffusion in the LLP encoder latent space without loss of optimality up to some error we prove that a near optimal policy is always recoverable with the LLP policy encoder under some mild conditions: that our low-level policy has closely imitated our expert dataset and that the transition dynamics are Lipschitz smooth. Lipschitz transition dynamics is fairly mild assumption to make (Asadi et al.. 2018, Khalil 2008). Several continuous control environments will exhibit a Lipschitz transition function, as well as many robotics domains. If an environment not Lipschitz, it can argued be in some sense unsafe (Berkenkamp al. 2017). Moreover, one could then impose additional action constraints on such an environment to make it Lipschitz again.
In this paper, we consider high dimensional control from pixels, and thus factorize our low level policy formulation into (St (s,, 9t) == of (st). 9t)) where 2 := {st) defines an encoder function that maps into latent representation and ø translates the representation 2 into a distribution over actions a given a goal g. Note that á is an arbitrary function, and can be made as simple as matrix or complex as another hierarchical model. Let (s) == á E(s) be deterministie low-level policy. Here, the o symbol refers to function composition. Similar to Nachum et al. (2019), we can then define the sub- subooptimality of the action and state space induced by A = S = (s) to be the difference between the best possible high-level policy Th argmax J/ (m)) within this new abstract latent space and the best policy in general:
SubOpt(E, x) sup V" (s) - (s).
sES
Intuitively, this captures how much potential performance stand to lose by abstracting state and action space.
Proposition 3.1. If the transition function p(s'|s, Lipschitz continuous with constant Ky and sup,eS,aEA (s)(s)) - a²1 S then
27 SubOpi(E, 0) (1 --
Here, dom (Ps'|ss a)) is the cardinality of the transition function. Crucially, this shows that the suboptimality of our policy is bounded by and that our framework can recover near-optimal
4

---

Published as a conference paper at ICLR 2024
Algorithm 1 Hierarchical Diffusion Policy Training
Input:
baseline goal-conditiondd policy To == ((E(s)), gt). diffusion variance schedule aze, temporal stride language model
Output: trained hierarchical policy π (ar)ss)) == Too (aslasl (s, The Thi (s)), where Ir is sampled every c time steps from Fhi the first state in latent plan T².
Collect dataset D Poppolicy by rolling out trajectories
TToo
2: Instantiate Thi as diffusion model Eo(Tnoisy t, p(L)
3: repeat
4: Sample mini- batch (r, L) = B from D Poupolicy.
5: Subsample latent plan T = (80), (80). E(82)),... E(ST))
6: Sample diffusion step / ~ Uniform( (1, ...., T},, noise ~ (0.1)
7: Update high-level policy Thi with gradient - -Velee - Ea ca(VārT + vI 11-1:p dee, t, (,,)),
8: until converged
policy with fairly mild assumptions on the environment. We give a detailed proof in Appendix G The beauty of this result lies in the fact that due to the nature of the LLPP's training objective of minimizing reconstruction error, we are directly optimizing for Proposition 3.1 second assump- tion a (s) a*I S c. Moreover, if this training objective converges, achieves an approximate π -riieeelevance abstraction (Li al. (006). Therefore, utilizing the LLP encoder space for diffusion is guaranteed to be near-optimal as long as the LLP validation loss is successfully minimized. However, there still exists gaps between the theory and practice, which detail in Appendix G
3.2 PRACTICAL INSTANTIATION
We now describe our practical implementation and algorithm instantiation in this section. We first give an overview of our high-level policy instantiation before specifying the implementation of our low-level policy. Finally, we describe our model architecture in detail.
High-Level Policy. In Algorithm I we clarify the technical details of our high-level diffusion policy training. Before this point, we assume that a low-level policy (LLP) has been trained. Whilst framework is flexible enough to use any LLP trained arbitrarily long it belongs to the factorized encoder family described in Section 3.1 in our practical implementation we first train L.PP through hindsight relabelling. As our choice of diffusion policy allows tractably modeling long-horizon plans as opposed to other architectures, we proceed to prepare a dataset of long-horizon latent plans. We take the pretrained encoder representation and use it to compress the entire offline dataset of states. This is useful from a practical perspective as it caches the computation of encoding the state, often allowing for 100-200x less memory usage at train time. This enables significantly more gradient updates per second and batch sizes, and is largely why our method is drastically more efficient than prior work utilizing diffusion for control. After caching the dataset with the pretrained encoder, we then induce temporal abstraction by subsampling trajectories from the dataset, taking every ed state in line of Algorrihhh 1, where refers to the temporal stride. This further increases efficiency and enables flexible restructuring of computational load between the high-level and low-level policy, simply by changing c. Our preparation of the dataset of latent plans now complete. These plans can now be well modeled with the typical diffusion training algorithm, detailed lines 6-7. During inference, the LLP will take as input the first goal state the latent plan and the current environment state, and output actions. After e timesteps, we resample new latent plan from the HLP and repeat this process until either the task has been finished or a fixed timeout (360 timesteps).
Low-Level Policy. We adopt the HULC architecture (Mees et al.) 2022a) for our low-level policy, and the state encoder as the visual encoder within the HULC policy. In HULC, the authors propose an improved version of the hierarchical Multi-Context Imitation Learning (MCIL). HULC utilizes hierarchy by generating global discrete latent plans and learning local policies that are conditioned this global plan. Their method focuses several small components for increasing the effectiveness of cext-to-control policies, including the architectures used to encode sequences in relabeled imitation learning, the alignment of language and visual representations, and data
5

---

Published as a conference paper at ICLR 2024
augmentation and optimization techniques. We did not choose simpler flat MLP as aimed to implement the strongest possible model. For a more detailed explanation, please refer to Appendix C
Model Architecture. We adopt the T5-XXL model (Raffel al. 2020) as our textual encoder, which contains 1 IB parameters and outputs 4096 dimensional embeddings. T5-XXL has similarly found success in the text to image diffusion model Imagen (Ho al. 2022a). We utilize temporal U-Net (Janner et al.] 2022), which performs ID convolutions across the time dimension of the latent plan rather than the 2D convolution typical in text-to-image generation. This is motivated by our desire to preserve equivariance along the time dimension but not the state-action dimension. In addition, we modify the architecture in Janner al. (2022) by adding conditioning via cross attention in fashion that resembles the latent diffusion model (Rombach et al., (2022). Finally, we use DDIM (Song et al., 2020a) during inference for increased computational efficiency and faster planning. DDIM uses strided sampling and is able to capture nearly the same level of fidelity typical DDPM sampling (Ho et al. 2020) with an order of magnitude speedup. For rolling out the latent plans resample
generated by the denoiser, until either the task is completed successfully or the maximum horizon length is reached. Our low-level policy takes over control between samples, with goals generated by the high-level policy as input.
sequence of goals y with temporal stride frequeney
EXPERIMENTS
In our experiments we aim to answer the following questions. In In/subsection 4.2 and (ubsection 4.3 we answer 1) Does diffusion-based approach perform well for language-conditiones
RI. In subsection 4.4 we answer 2) How much efficiency is gained by planning in a latent space? Finally, in subsection 4.5 (ubsecttin 4.7 we answer 3) is a hierarchical instantiation of the diffusion model really necessary, and what trade-offs are there to consider?
4.1 EXPERIMENTAL SETUP
Dataset and Metric. We evaluate on the CALVIN, and CLEVR- Robot benchmark (Mees et al. 20226] Diang et al., (219)), both challenging multi-task, long-horizon benchmarks. After pretraining our low-level policy on all data, we freeze the policy encoder and train the diffusion temporal U-Net using the frozen encoder. For more details on our training dataset and benchmarks, please refer to Appendix H We roll out all of our evaluated policies for 1000 trajectories on all 34 tasks, and all comparisons are evaluated in their official repository
Baselines.
We compare against the prior state of the art model Skill Prior Imitation Learning (SPIL)
(Zhou al.
2023). which proposes using skill priors enhance generalization. As introduced in
Section 3.2 we also compare against HULC and MCIL (Mees et al.) 2022a] Lynch & Sermanet
2020b). HULC provides a strong baseline is also a hierarchical method. MCIL (Multicontext
Imitation Learning) uses a sequential CVAE to predict next actions from image or language-basee goals, by modeling reusable latent sequences of states. GCBC (Goal-conditioned Behavior Cloning) simply performs behavior cloning without explicitly modeling any latent variables like MCIL, and represents the performance of using the simplest low-level policy. Finally, Diffuser generates a trajectory by reversing the diffusion process. Diffuser is most comparable to our method. However, it utilizes no hierarchy or task-specift representation. In our benchmark, Diffuser- 1D diffuses in the latent space of a VAE trained from scratch the CALVIN dataset, whilst Diffuser-20 22D diffuses in the latent space of Stable Diffusion, a large internet-scale pretrained VAE (Rombach et al. (022). We also attempted to train Diffuser directly on the image space of CALVIN, but this model Tailed to converge on a 8x 2080 Ti server (we trained for two weeks).
For the CLEVR robot environment. We consider the following representative baselines: Goal- Conditioned Behavior Cloning Transformer, which is trained end to end by predicting the next action given the current observation and language instruction. We also consider using Hierarchical Trans- former, which is trained using the same two-stage training method as our method. The Hierarchical
https: tww.. www..oooo/oooo...oooo www.hhheeennn//e ree/ 4454 we54 1:55599999999990 50e 301 6388 Relo

---

Published as a conference paper at ICLR 2024
Table 1: HULC Benchmark Results. We compare success rates between diffusion model and prior benchmarks on multitask long-horizon control (MT-LHC) for 34 disparate tasks. report the mean and standard deviation across 3 seeds for our method with each seed evaluating for 1000 episodes. We bold the highest performing model in each benchmark category. For hyperparameters please see Appendix D
Horizon
GCBC
MCIL
HUTCC
SPII.
Diffuser- ID
Diffuser-2D
Ours
One
64.7 1
76.4 1,5
82.6
84.6 1 0.6
47.3 1 2.5
37.4 1 3.2
88.7 1. 1.5
Two
2.2
4.1
64.6+27
1.3
18.8 1.8
1.3
69.9
+.9:22. 2.8
Three
1.12+4.1
4.5
47.9 3.2
+0.8+00.4
5.9 0.4
1.3+ 0.2
54.5 + 5.0
Four Five
2.0
1..1
2.4
0.6
2.0
0.2 0.0
42.7 + 5.2
1.3 0.9
9.3 3.5
26.5
0.3
0.5
0.07 = 0.09
32.
+ 5.2
Avg horizon Icn
1.11 111::03
1.82 0.2
2.57 + 0.12
2.67 + 0.01
0.74 0.03
0,48 = 0.09
2.88 + 0.19
Transformer is equivalent LCD (our method), other than the usage of transformer rather than diffusion model for the high-level policy.
We focus our comparisons for CALVIN on the current strongest models on the CALVIN benchmark in our setting of general policy learning (SPIL and HULC). To ensure a fair comparison, we rigorously follow the CALVIN benchmark's 's evaluation procedure and build directly from their codebase. SPIL, MCIL, and GCBC results are taken from Zhou et al. (2023) and MMess et al. (2022a), whilst HULC results are reproduced from the original repository. Diffuser was retrained by following the original author's s implementation.
4.2 LCD PERFORMS WELL FOR DONDITTONN RL.
As shown in Table I we significantly outperform prior methods on the multi-task long-horizon control (MT- LHC) benchmark on every metric, and improve on the strongest prior model's 's average performance on horizon length one tasks by 4.1%, and average horizon length by 0.21. In addition, we outperform the next best diffusion model by 41.4% on length one tasks and by 2.14 on average horizon length. Here, average horizon length refers to the average amount tasks that a given model was able to finish. For example, model's 's average horizon length of 2.88 means was on average able to finish 2.88 tasks.
Table 2: CLEVR-Robot Results. Success rates between diffusion model and GCBC trans- former and Hierarchical Transformer (HT) on 80 disparate tasks. We report the mean stan- dard deviation across 3 seeds for our method with each seed evaluating for 100 episodes.
GCBC
HT
Ours
Seen
45.7 2.5
38.0 2.2
53.7 1.7
Unseen
46.0
36.7
54. 5.3
Noisy
42.7
33.3 1.2
48.0
Results in Table 2]show that LCD surpasses the GCBC Transformer and the Hierarchical Transformer in Seen, Unseen, and Noisy settings. In "Seen," LCD achieves 53.7 1.7 SR, confirming its capacity to perform tasks similar to its training. "Unseen" tests generalization to new NL instructions, with LCD leading at 54.0 5.3 SR, which evidences its ability to parse and act on unfamiliar language commands. In "Noisy" conditions with perturbed language input, LCD attains 48.0 + 4.5 SR, displaying resilience to language input errors. This answers our first research question on whether a diffusion-basee approach can perform well for condiiie-d RL.
4.3 LCD CAN GENERALIZE TO NEW TASKS
We test the task generalization of LCD on a collection of five held out tasks in Table This benchmark is difficult it requires zero-shot generalization to a new task, which requires generalization across language and states. We find that LCD can successfully generalize, and is able to compose several disparate con- cepts from the training dataset together such as the color of the blocks, verbs such as lift and push, and object posi- tions and state.
Table 3: Task generalization LCD on a collection five held out tasks. We test with 3 seeds and report the mean and std, evaluating on 20 rollouts per task for a total of 100 evaluations.
Task
Diffuser- ID
Ours
Lift Pink Block Table
31.67 + 10.27
55.00 16.33
Lift Red Block Slider
13.35 + 10.25
88.33 + 8.50
Push Red Block Lefi
1.64 + 2.35
35.00 + 7.07
Push Into Drawer
3.34 + 4.71
90 00.00-0 = 10.80
Rotate Blue Block Right
5.00 + 4.10
36.67 14.34
Avg SR
12.67 - 3.56
61.00 +: 7.79
7

---

Published as a conference paper at ICLR 2024
4.4 INFERENCE TIME IS 3.3x TO 15X FASTER THROUGH HIERARCHY
Table 4: Wall clock times for training. Latent dims denotes the size of the latent space that we perform the diffusion generation in. We compare against the same variants of Diffuser as presented in Table
HULC
SPIL
Diffuser- ID
Diffuscr-2D
Ours (HLP only)
Ours (full)
Training (hrs) (1 Lower is Better)
82
86
20.8
49.2
13.3
95.3
Inference time (scc) (J) Avg V updates/sec (f)
0.005
0.005
1.11
5.02
0.333
0.336
.5
.5
4
2.1
6.25
6.25
Model size (J)
47.1M
54.8M
74.7M
125.5M
20.1M
67.8M
Latent dims (4)
N/A
N/A
256
1024
32
Through the usage of DDIM, temporal abstraction, and low-dimensional generation, we find in Table 4 that LCD is significantly faster during rollout and training than Diffuser. In addition, method is significantly faster to train than HULC, albeit slower during rollout. This is to be expected, HULC is not a diffusion model and only requires a single forward pass for generation. However, when comparing to other diffusion models we find that our method is 3.3x-15x faster during inference and 1.5x-3.7x faster during training, answering our second research question regarding how much efficiency is gained by planning in latent space. All numbers are taken from our own experiments and server for reproducibility and fairness, including baselines.
4.5 Is DIFFUSION REALLY NECESSARY?
In order to analyze whether diffusion planning actually improves HULC or if the gain comes just from the usage of a high-level policy and to answer our third research question, we perform an ablation study in Table by using a simple MLP and Transformer which receives the ground truth validation lan- guage embeddings
Table 5: Ablation of our methad comparing against a simple residual eight layer MLP with 2048 hidden dimensions (25,5M 5M total parameters) and a four layer Transformer with 4096 hidden dimensions (1.5M to- tal parameters) high level policy. We use the same methodology as in Table I and report the dard deviation across sceds for our method with each seed evaluated for 1000 episodes. For hyperparameters and more details please see Appendix E
and stan-
high-level policy,
well the current state, and attempts to predict the next goal state. An example of the ground truth language is "Go push the red block right", which is never ex- plicitly encountered during training. Instead, similar prompts of the same task are given, such as "slide right the red block". We find that even given the ground truth validation language em- beddings (the MLP and Transformer do not need to generalize across language), these ablations significantly underperforms LCD, and slightly underperforms HULCC This suggests that our gains in performance over HULC truly do come from better modeling of the goal space by diffusion.
Task
MI.P
Transformer
Ours
One
86.6 + 2.3
85.4 + 0.5
88.7 7 + 1.5
Two
64.1 + 0.1
60.9 +.9:000 0.5
69.9 + 2.8
Three
48.1 + 8.3
42.6 1.9
54.5 + 5.0
Four
35.7 + 8.8
29.8 + 2.5
42.7 + 5.2
Five
25.5 + 7.6
20.0 + 1.8
32.2 + 5.2
Avg SR
2.60 = 0.33
2.36 = 0.08
2.88
0.19
4.6 ROBUSTNESS TO HYPERPARAMETERS
In ?? we show that our diffusion method is robust to several hyperparameters including a frame offset and the hidden dimensions of the model. Frame offset controls the spacing between goal states in the sampled latent plan, which affects the temporal resolution of the representations. We consider augmenting
dataset by sampling d ~ Unif{-o of for goal state so4d rather than just se, potentially improving generalization. However, we find that this effect is more or less negligible. Finally, find our method is also robust with respect to the number of parameters, and instantiating a larger model by increasing the model dimensions does not induce overfitting.
4.7 SUMMARY OF CRITICAL FINDINGS
Diffuser fails to plan in original state space. After investigating the performance of diffuser, findings in Figure 2 inddcaae that Diffuser fails to successfully replan in high dimensional state spaces when using Variational Autoencoder (VAE). Based on our observations, we hypothesize that the failure of Diffuser to replan successfully is due to the diffusion objective enabling interpolation
8

---

Published as a conference paper at ICLR 2024
between low density regions in the VAE's latent space by the nature of the training objective smoothing the original probability distribution of trajectories. In practice, this interpolation between low density regions correspond to physically infeasible trajectories. This suggests that interpolation between low density regions in the VAF's latent space is a significant factor in the failure of diffuser to plan and replan successfully. Our results highlight the need for better representation, and for further research of scaling diffusion models to high dimensional control.
Diffuser 1D
DDff
The encoder representation of deep goal- conditioned RL models can be effectively used for hierarchical RL. We find that hav- ing good representation is essential for effec- tive diffusion in general control settings, as ev- idenced in Figure 2 Our results imply that the encoder representation of deep reinforce- ment learning models has the potential to be effectively utilized in hierarchical reinforcement learning. In modern learning from pixels, propose that an effective task-aware representa- tion can be extracted by using the latent space of an intermediate layer of a low-level policy as an encoder. Our results suggest that using shared policy encoder between the high and low level policies can improve the effectiveness and efficiency of generative modeling in hierarchical RL.
Time
(1))
Figure 2: Denoised Latent Representations. Directly using latent diffusion models fails. Hallucination occurs on a 3-TC VAE trained from scratch the CALVIN dataset (Diffuser-ID) and loss of fine details occurs with SD v1.4's ' (Rombach et al.] (2022) internet-scale pretrained autoencoder (Diffuser-DD) For more and en- larged samples please refer to Appppdiii K
RELATED WORK
Text-to-Control Models. text-to-control
models or language-conditioned policies (Lynch
& Sermanet 2020a Tiang et al. 2019 Colas et al. 2020
Mees et al.) 2022a) have been explored in
tasks and environments (Hermann et al. 2020 Wang et al.. 2020a). Although they are not the only way to incopportte external knowledge in the form of text to decision making tasks
the RL community for improving generalization to novel
2017 Bahdanau et al. 2018 Hill et al.
2019 Colas et al.
(Luketina et al.) [2019] Zhong et al. 2019)), they remain one of the most popular (Wang et al.) 20206)
Zheng et al., 20222 Duan et al. 2022). However, language grounding in RL remains notoriously
difficult, as language vastly underspecifies all possible configurations of corresponding state (Quine)
1960).
Diffusson models such as DALL-E (Ramesh et al. 2022) and GLIDE NNchol et al. 2022) have recently shown promise as generative models, with state-of-the-art text-to-image generation results demonstrating a surprisingly deep understanding of semantic relationships between objects and the high fidelity generation of novel scenes. Video generation models especially relevant to this work, as they are a direct analog of diffusion planning model Diffuser (Janner et al.) 2022) in pixel space without actions. Diffuser first proposed transform decision making into inpainting and utilize diffusion models to solve this problem, which much work has followed up on (Dai et al. 2023 Chi al. 2023 Ajay al. 2022).
Diffusion Models.
For additional related work, please refer to Appendix B
6 CONCLUSION AND LIMITATIONS
Using language to learn atomic sub-skills is critical for complex environments. We use learned state and temporal abstractions to address this, leveraging the unique strengths of diffusion models for long-term planning. Their shortcomings in detailed generation are managed by teaching low-level, goal-driven policies through imitation. Experiments show our model's simplicity and effectiveness, with LCD reaching top performance in a language-bai based control benchmark. For future work, one could extend LCD to incorporate additional levels of hierarchy and scale to even longer horizons.

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huel Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as say: Grounding language in robotic affordances, 2022. URL https://eraa http///wwwvvvrr///////////////// /tt.://///////..2.52/22
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15653 2022.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress, 2018. http:/tttt///attt www.awwww.....oo..aaaaa.oao..eea abs/ abw.///0.10600077 06877
Kavosh Asadi, Dipendra Misra, and Michael Littman. Lipschitz continuity in model-based rein- forcement learning. In International Conference Machine Learning, pp. 264-273. PMLR, 2018.
Dzmitry Bahdanau. Felix Hill, Jan Leike, Edward Hughes, Arian Hosscini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv: 0/946, 2018.
Felix Berkenkamp, Mattco Turchetta, Angela Schoellig, and Andreas Krause. Safe model-based reinforcement learning with stability guarantees. Advances in neural information processing systems, 30, 2017.
Rainer Bischoff and Volker Graefe. Integrating vision, touch and natural language in the control of a situation-oriented behavior-base humanoid robot. IEEE SMC'99 Conference Proceedings. 1999 IEEE International Conference on Systems, Man, and Cybernetics (Cat. No. 99CH37028), volume 2, pp. 999-1004. IEEE, 1999.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:2212.06811 2022.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in newral information processing systems, 34:1508: 15084-15097, 2021.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arXiv:2303.04137, 2023.
Cédric Colas, Ahmed Akakzia, Pierre-Yvel Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Language-conditioned goal generation: new approach to language grounding for rl. LAREL Workshop 2020, abs/2006.07043, 2020.
Yilun Dai, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.001 11, 2023.
Pim De Haan. Dinesh Jayaraman, and Sergey Levine. Causal confusion in imitation learning. Advances in Neural Information Processing Systems, 32, 2019.
10

---

Published as a conference paper at ICLR 2024
Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6(2):230-2444 2022.
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. arXiv preprint arXiv: 00..1502.222222 2015.
Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference Machine Learning, pp. 4051-4062. PMLR, 2021.
Brent Harrison, Upol Ehsan, and Mark O Riedl. Guiding reinforcement learning exploration using natural language. arXiv preprint arXiv: 1XXi:1707.086166 08616, 2017.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer. David Szepesvari, Wojeiech Marian Czarnecki, Max Jaderberg, Denis Teplyashin, et al. Grounded language learning simulated 3D world. arXiv preprint arXiv: nee.10006055555 2017.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James I.. McClelland, and Adam Santoro. Emergent systematic generalization situated agent, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. arXiv preprint arxiv:2006. 1239, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a.
Jonathan Ho, Chitwan Saharia, William Chan, David J. Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022c. http::/hhhh..cooapppppss////////
html
wtt.a/////2...2.2 03458
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. Advances in neural information processing systems, 32, 2019.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. ICML, 2022.
Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David
Ding. Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14793 2021.
Eric Jang. Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pp. 991-1002. PMLR, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem, 2021.
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for 09991
flexible behavior synthesis, 2022. URL
Yiding Jiang. Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language Abstraction for Hierarchical Deep Reinforcement Learning. Curran Associates Inc., Red Hook, NY, USA, 2019.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.00 10.00944 2022.

---

Published as a conference paper at ICLR 2024
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXIv:2001.08361, 2020.
Hassan K. Khalil. Nonlinear systems third edition. 2008.
Diederik Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1179-1191, 2020.
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint
2018.
Lihong Li, Thomas Walsh, and Michael Littman. Towards unified theory state abstraction for mdps. 01 2006.
Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba, Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making arXiv preprint arXiv:2202.01777, 2022a.
Xiang Lisa Li, John Thickstun, Ishaan Guirajani, Percy Liang, and Tatsunori B. Hashimoto. Diffusion- Im improves controllable text generation, 2022b. URL htt.hhttps/eee http///axvvv...///a/ http///www...//a/////////00/
14217
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. arXiv preprint arXiv: 1900:09966,9,, 2019.
Corey Lynch and Pieme Sermanet. Grounding language in play. CoRR, abs/20005.076443 2020a. 2020a. URL
httts:://rxxiii http////wwi...rr//////.0..0..0 07648
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint 0rXiv:2005.07648, 2020b.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data, 2022a. URL http:///wwxxw..oo/a///////////// a22/
06252
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for iinuuage-oondttioned policy learning for long-horizon robot manipulation tasks, IEEE Robotics and Automation Letters, 2022b.
Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. arXiv preprint arXiv:2202.08938, 2022.
Ofrr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. Advances neural information processing systems, 31, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA. May 6-9, 2019. OpenReview.net, 2019. URL https
//openreview net//orumm d= emus0qF7
12

---

Published as a conference paper at ICLR 2024
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards photorealistic image generation and editing with text-guided diffusion models. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16784-16804. PMLR, 17-23 Jul 2022. URL
nichol22a. html
Jing-Cheng Pang, Xin-Yo Yu Yang, Si-Hang Yang, and Yang Yu. Natural language-conditioned rein- forcement learning with inside-out task language development and translation. arXiv preprint arXiv:2302.09368, 2023.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Willard Van Orman Quine. Word and object mit press. Cambridge MA, 1960.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-5511, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents, 2022. U http:///xxvvv..g///ee 2204
06125
Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf. Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting. 2021. doi: 10.48550/
ARXIV.2101. 12072. /tR//tttps::/axxi..rrrr///ee
12072
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo. Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint wxxv:2205..06155 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Erick Rosete-Beas, Oier Mees, Gabriel Kalweit, Joschka Boedecker, and Wolfram Burgard. Latent plans for task-agnostic offline reinforcement learning. In Conference on Robot Learning, pp. 1838-1849. PMLR, 2023.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635. JMLR JML. Workshop and Conference Proceedings, 2011.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In Francis Bach and David Blei (eds.). Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07-09 Jul 2015. PMLR. URL htttps://procestt
m www.mmmr..........mmmmmmmm.m v
schaul15.btml
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.1443 14792, 2022.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcemnntn learning with context- based representations, 2021.
06677
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2256-2265, Lille, France, 07-09 Jul 2015a. PMLR. URL https www.ohhiiiiikkkkkiiiii ckstein15..... htm]
7/proceed ngs. ml press arress/v37/s0n /3t///////////// -ii
13

---

Published as a conference paper at ICLR 2024
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pp.
PMLR, 2015b.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv: 13346,,, 2020b.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211, 1999.
Alan M Turing and J Haugeland. Computing machinery and intelligence. The Turing Test: Verbal Behavior as the Hallmark of Intelligence, pp. 29-56, 1950.
Hu Wang. Qi Wu, and Chunhua Shen. Soft expert reward learning for vision-and-language navigation. In European Conference Computer Vision, pp. 126-141. Springer, 2020a.
Tongzhou Wang. Simon S Du, Antonio Torralba, Phillip Isola, Amy Zhang, and Yuandong Tian. De- noised mdps: Learning world models better than the world itself. arXiv preprint arXiv:2206. 15477, 2022a.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Vision-language navigation policy learning and adaptation. IEEE transactions on pattern analysis and machine intelligence, 43(12):4205 4216, 2020b.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffussion poliuuss as am expressive policy class for offline reinforcement learning, 2022b. URL http:::/aaraiv.ccrr/ab/////
06193
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics, 2011.
Lilian Weng. What are diffusion models? lilianweng..gthhubbio, Jul 2021. URL https:// modells/
lilianweng github..hh.... httpaaaoo...op..//00000111111111
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
Kaizhi Zheng, Xiaotong Chen, Odest Chadwicke Jenkins, and Xin Eric Wang. VImbench: A
compositional benchmark for lision-and-languaee manipulation. arXiv preprint arXiv:2206.08522 2022.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to novel environment dynamics via reading. arXiv preprint arXiv: 1910.082100 2019.
Victor Zhong, Jesse Mu, Luke Zettlemoyer, Edward Grefenstette, and Tim Rocktäschel. Improving policy learning via language dynamics distillation. arXiv preprint arXiv:2210.00066 2022.
Hongkuan Zhou, Zhenshan Bing, Xiangtong Yao, Xiaojie Su, Chenguang Yang. Kai Huang, and Alois Knoll. Laagguage-condiiinn imitation learning with base skill priors under unstructured data. arXiv preprint 2rXiv:2305.19077 2023.
14

---

Published as a conference paper at ICLR 2024
Appendix
TABLE OF CONTENTS
Appendix
details the broader impact of work.
Appendix
gives detailed treatment to additional related work.
Appendix clarifies the usage of the LLP, and the specific choice of HULC.
Appendix presents experiment details, HP settings and corresponding experiment results.
Appendix
presents the derivation for the trajectory based denoising training objective.
Appendix presents the proof for Theorem 3.1. and the gaps that still exist between theory
and practice.
Appendix
gives a more detailed account of our training dataset.
Appendix
presents an example figure of the denoising process for the Diffuser-2D model.
Appendix presents task distributions of MT-LHC benchmark results given in
Table
Appendix K offers more samples with higher resolution of the representation failures of Diftuser- ID and Diffuser-2D, first referenced IFigure 2
Appendix L
presents comparison between the TSNE visualizations of a ground-truth encoded trajectory and one gneerated from Diffuser-ID.
Appendix presents a TSNE visualization of the discrete latent plan space within HULC. We clarify that this is not the latent goal space that our model does generation in.
Appendix N contains our LCD's model card.
Please refer to our website htttps://lcd.ede win/ for more qualitative results in video format. We release our code and models at https: //gtttbttt/h///////eee//tt com/ 7423/
15

---

Published as a conference paper at ICLR 2024
A BROADER IMPACT
In this paper, we present research on diffusion-basee models for reinforcement learning. While our work has the potential to advance the field, recognize the importance of being transparent about the potential societal impact and harmful consequences of our research.
Safery: Our research focuses the development of diffusion models for reinforcement learning, and we do not anticipate any direct application of our technology that could harm, injure, or kill people.
Discrimination:
We recognize the potential for LCD to be used in ways that could discriminate against or negatively impact people. As such, we encourage future work to take steps to mitigate potential negative impact, especially in the areas of healthcare, education, or credit, and legal cases.
Surveillance: LCD did not involve the collection or analysis of bulk surveillance data.
Deception and Harassment: We recognize the potential for our technology to used for deceptive interactions that could cause harm, such as theft, fraud, or harassment. We encourage researchers to communicate potential risks and take steps to prevent such use of LCD.
Our research required the usage of GPUs, which may be potentially energy intensive and environmentally costly. However, we seek to minimize the usage of GPUs through LCD's efficiency and the impact of our work could be beneficial to the environment.
Environment:
Human Rights: We prohibit the usage of LCD that facilitates illegal activity, and we strongly discourage LCD to be used to deny people their rights to privacy, speech, health, liberty, security, legal personhood, or freedom of conscience or religion.
Bias and Fairness: We recognize the potential for biases in the performance of LCD. We encourage researchers building off this work to examine for any biases and take steps to address them, including considering the impact of gender, race, sexuality, other protected characteristics.
In summary, we recognize the potential societal impact and harmful consequences of our research and encourage researchers to consider these factors when developing and applying new technologies on top LCD. By doing so, we can help ensure that our work has a positive impact on society while minimizing any potential negative consequences.
16

---

Published as a conference paper at ICLR 2024
B ADDITIONAL RELATED WORK
Here we gives a more detailed treatment to additional related work.
B.1 TEXT-TO-CONTROL MODELS
Modern text control models often still struggle with long-horizon language commands and misinterpret the language instruction. Hu et al. (2019) attempt to solve a long-horizon Real-Time Strategy (RTS) game with a hierarchical method utilizing language as the communication interface between the high level and low level policy, whilst Harrison et al. 2017) consider training language encoder-decode for policy shaping, Hanjie et al. (2021) utilize an attention module conditioned on textual entities for strong generalization and better language grounding. Zhong et al. (2022) propose a model-based objective to deal with sparse reward settings with language descriptions and Mu al. (2222) also tackle the sparse reward settings through the usage of intrinsic motivation with bonuses added on novel language descriptions. However, only Huu et al. (2019) consider the long- horizon setting, and they do not consider a high-dimensional state space. Jang et al. (2022) carefully examine the importance of diverse and massive data collection in enabling task generalization through language and propose FiLM conditioned CNN-MLP model (Perez et al. 2018). Much work has attempted the usage of more data and compute for control (Brohan et al.) [ et al. [001]] Reed et al. 2022). However, none these methods consider the usage of diffusion models as the medium between language and RL.
B.2 DIFFUSION MODELS
One driver of this recent success in generative modeling is the usage of classifier-free guidance, which amenable the RL framework through the usage of language reward function. The language command is a condition for optimal action generation, which draws direct connection to control as inference (Levine 2018). Given the success of denoising diffusion probabilistic models (Ho et al.. 2020) in text-to-image synthesis DSohl-Dickstein et al. (215a),, the diffusion model has been further explored in both discrete and continuous data domains, including image and video synthesis (Ho) et al. 2022b c). text generation (Li et al. 2022b), and time series (Rasul al.) (021). Stable diffusion, an instantiation of latent diffusion (Rombach et al. 2022), has also achieved great success and is somewhat related to our method as they also consider performing the denoising generation in smaller latent space, albeit with a variational autoencoder (Kingma Welling 2013) rather than low level policy encoder. To expound further on Janner al. (2022)'s Diffuser, they diffuse the state and actions jointly for imitation learning and goal-conditioned reinforcement learning through constraints specified through classifier guidance, and utilize Diffuser for solving long-horizon and task compositional problems in planning. Instead of predicting the whole trajectory for each state, (Wang al., 2022b) apply the diffusion model sample a single action at a time conditioned by state. However, neither of these works considers control from pixels, or utilizing language for generalization. One successful instantatiation of a direct text video generation model is (Dai et al. 2023). However, they again avoid the issue of directly generating low level actions by using low-level controller or an inverse dynamics model which we are able to directly solve for.
17

---

Published as a conference paper at ICLR 2024
C Low LEVEL POLICY (LLP) CLARIFICATIONS
We would like to clarify how the state encoder and L.P trained, how the state encoder is chosen, the specific state encoder that is chosen HULC, and the reasoning for how we chose HULC the LLP in our practical instantiation. The state encoder is never trained explicitly, but rather is trained end-to-end part of the LLP. Thus, after the LLP is trained the state encoder has also been trained. Both the LLP and the state encoder are then frozen during the training of the diffusion model as high-level policy (HLP), as seen in Figure
C.I LLP TRAINING OBJECTIVE
As detail in section 2] in general the loss for the LLP will be some form of behavior-cloning:
LBc Bsr,ar [Ila, [oo g s1+,
The specific loss used in HULC for our L.P is described here (Mees et al. 2022a):
Lact (Dplayy V) - In( on P(as, (iVV),
Here, P(a,, (V.),,, (V.)) = F( ( - F(" F and F(-) is the logistic distri-
bution's cumulative distribution function.
We choose the state encoder be the visual encoder within HULC, which has total of 32 latent dimensions. The state encoding of the current state and some future goal state is then fed to the downstream multimodal transformer and action decoder to generate new action.
C.2 REASONING FOR AGA OF HULC AS Low LEVEL. POLICY (LLP)
We clarify the usage of using hierarchical model use a hierarchical approach for our low-level policy for two main reasons. First, we aim to leverage the strongest existing baseline maximize our chances of creating SOTA model. Second, encountered issues replicating the results for the GCBC flat policy from its original paper (Mees) et al. 2022b), making it less favorable for approach. It is also important note that there in principle, no reason why LCD cannot work with flat policy, and this would be straightforward addition for future work.
LLP here. We would like to clarify that
D HYPER- HYPER-PARAMETER SETTINGS AND TRAINING DETAILS
For all methods proposed in Table deviation of each method across 3 seeds. Each seed contains the individual training process and evaluates the policy for 1000 episodes.
Table and Table 6
obtaín the mean and standard
Baselines are run with either 8 Titan RTX or 8 A10 GPUs following the original author guidelines, whilst our experiments are run with single RTX A10. In total this project used around 9000 hours of compute.
D. HP AND TRAINING DETAILS FOR METHODS IN TABLE
18

---

Published as a conference paper at ICLR 2024
Model
Module
Hyperparameter
Value
Max Epochs
30
3 for KL Loss
0.01
Trainer
for Contrastive Loss
3
HULC
Optimizer
Adam
Learning Rate
2e-4
Transformer Hidden Size
2048
Model
Language Embedding Size
384
Action Dimension
32
Action Weight
10
Loss Type
I.2
Observation Dimension
32
Gaussian Diffusion
Diffusion Steps
20
Model Dimension
64
EMA Decay
0.995
Label Frequency
200000
Sample Frequency
1000
LCD
Batch Size
512
Learning Rate
2c-4
Trainer
Train Steps
250k
Number of Steps Per Epoch
10000
Normalizer
Gaussian Normalizer
Frame Offset
Table 6: Hyperparameters for our methods Table
19

---

Published as a conference paper at ICLR 2024
E HYPER-PARAMETERS FOR ABLATIONS IN TABLE
E. DETAILED INFERENCE PROCEDURE
During inference. give the MLP a concatenation of the current state s; and ground truth language. and train the model to output the goal state 8g 8ite, which is then given to the LLP. This is repeated every e time steps. For the transformer, we give the current state 8; and cross-attend to the ground truth language. In order to advantage the ablations, we do not ask the MLP to predict several steps the future, which been shown to generally fail practice Chen et al. (2021); Janner et al. (2021) 2022). This simplifies the learning objective and eases the burden on the high-level policy. We keep the same inference procedure for the Transformer in order to ensure fairness. In addition, we give both the MLP and Transformer ablation the ground truth language, that ablations need only to generalize across goals and not across language.
E.2 TRANSFORMER HYPERPARAMETERS
We control all other evaluation parameters not listed in Table E.2 to be the same as in our prior experiments for Diffusion high-level policy. These hyperparameters can be found in Appendix D),
Hyperparameter
Value
Number of gradient steps
100 K
Mini-batch size Transformer hidden dim
512
4096
Transformer layers
4
Final Hidden Activation
ReLU
Final Hidden Dim
4096
Number heads
8
Dropout
0.1
Learning Rate
2e 4
Layer Norm
32
Total Params
1.8M
Table 7: Hyperparameters for the Transformer in
Table
We gridsearch over the following parameters and pick the best performing combination from the following:
Num of transformer layers: [4, 8, 24]
Width of transformer layers: [2048, 4096, 8192]
Learning rate: [2e-5, 2c-4, 4e-4]
20

---

Published as a conference paper at ICLR 2024
F TRAINING OBJECTIVE DERIVATION
To model this, we turn diffusion models (Weng 2021). ) whom we borrow much of the following derivation from. Inspired by non-equilibrium thermodynamics. the common forms of diffusion models Sohl-Dickstein al. 2015b; Ho et al. 2020 Song et al. 2020a) propose modeling the data distribution p(T) random process that steadily adds increasingly varied amounts of Gaussian noise to samples from p(T) until the distribution converges to the standard normal. We denote the forward process as ((TaTTe-1). with sequence of variances (So, fr...rr). We define oz == 3
and or II 0'..
T [(TITTTO)) = II ((TATTe)), where [(FI(T=-1) = N(TH VT 1,T1.1
(6)
One can tractably reverse this process when conditioned on To, which allows for the construction of of the typical variational lower bounds for learning the backward process' density function (Sohl-Dickstein et al. (0155).. Since the backwards density also follows a Gaussian, it suffices to predict (4 and Eg which parameterize the backwards distribution:
Pe (Te-1 Tτ) = =N (Ti-1i (Te,t) t). Eg (T),
In practice, Eg is often fixed to constants, but can also be learned through reparameterization. Following (Ho et al. 2020) we consider learning only (4g, which can be computed just as a function of T: and (T,,)), One can derive that
Vo vI- cG, for (0, I), through successive
reparameterization of (6) until arriving at ((T)/T)). Therefore to sample from p(T), we need only to learn Eg. which is done by regressing to the ground truth given by the tractable backwards density. Assuming we have cg. we can then follow a Markov chain of updates that eventually converges to the original data distribution, procedure reminiscent of Stochastic Gradient Langevin Dynamics (Welling Teh 2011
1
Be VI
co (Tx. + o,2, where z ~ (O.
T--1
Ti
(8)
VI-
To learn Eg, we can minimize the following variational lower bound on the negative log-likelihood:
LCE = Ey(ro) log (TT))
Eg(( log (/ Po(ToTTddTIo)
-Eg(T log (ʃ (f((TIZ(TO)
po (o(To:)) dr
((TIIT(TO)
(o(TO.T)
-Ey(ro) log (EgITIIITT)
(T))
Ex((T.)) log Po(To:r)
((TI:T(TO)
log ((T::ITTO)
By(Torr)
LVLB-
po Po(To:))
LVLB = Ly + T-- ......1 To
where LT (KI.(((TTH)) (To) (o(T)))
L, DKL(g(Te) T TO) pe(Ti(Te41)
for 1 1 st and
Lg log log
Which enables us to find tractable parameterization for training, as the KL between two Gaussians analytically computable.
21

---

Published as a conference paper at ICLR 2024
L Ero.e
[ Tu) - ()(
2 Eg T1, t)
To.
2 12
1
Vaa
T
Vā-
1 1-a _
VI- āi k1)
Vas
1 (TI- 1- a2
1
v1- at
<o 1) 112
(10)
(1 - 04)2
Ero.e
| Ea Ea(Tr::)) +)1?]
20xe(1 (+,)
Σg l1
(1 -1- )²
To.c
||+ ca Va + v1 - 0,6,, 1)]
20x (x))
Eo
After removing the coefficient at the beginning of this objective following (Ho el al. 2020). we arrive the objective used in the practical algorithm Algorithm I
E STo.ellle co(vā,To + VI a,E,, +)
(11)
Furthermore, thanks to the connection between noise conditioned score networks and diffusion models (Song et al. 2020b Ho al. 2020), we are able to state that eg x V log p(T):
So ST( t) V,, log p(T:)
Ey(ro) V (T.P(T
E (T(
Eu((o)
(12)
VI
(T(T)) V1
Therefore, by using variant eg conditioned on language to denoise our latent plans, we can effectively model P3p7 R) with our diffusion model, iteratively guiding generated trajectory towards the optimal trajectories conditioned on language.
22

---

Published as a conference paper at ICLR 2024
G PROOF OF PROPOSITION 3.1
Proof. The proof fairly straightforward, and can be shown by translating our definition of subopti- mality into the framework utilized by (Nachum al. 2019). We then able to leverage their first theorem bounding suboptimality by the Total Variation (TV) between transition distributions to show our result,
TV is bounded by the Lipschitz constant multiplied by the domain of the function.
(Nachum et al. 2019) first define low level policy generator V which maps from S À IT. Using the high level policy to sample goal 9r (hi(g they use y to translate this to a policy TT (5r, 9t), which samples actions dr+* Te (a]sst++ k) from & E [0, -- 1]. The process is repeated from St+c- Furthermore, they define an inverse goal generator (p(s, a), which infers the goal g that would cause to yield an action (s g). The following can then be shown:
Theorem G.1. there exists yp : x A - Å such that,
sup sES,
Drv(P(s/s,
((s se
(13)
then SubOpt' (VV) & Ce, where C [I-7)z Rmaxx
2y
Note that their SubOpt' is different from ours; whilst we defined in terms of the encoder and action generator o, they define it in terms of V. Note, however, that the the temporal stride = 1, as y becomes mh = 6 o E It is essential to note that when using a goal conditioned imitation learning objective, as we do in this paper, Too becomes equivalent an inverse dynamics model IDM {s)) and that cpss, a) becomes equivalent to & (s'). This is the key to our proof, as the second term in the total variation
equivalent when
P(s'ss, (' ((s,)( = P(sls, ((s,((())
(14)
P(s'ls,a
Since we have that the transition dynamics are Lipschitz:
SAL 1P(8's,,)) P(6'1, + du
A.S
{ Kyla Kyla- e)
A.S
(15)
( du
KKee
A Ke e (om(P(s'((s.) a))
Which we can then plug into 13 to obtain the desired C' (1 (1-73a Rmaz Rmar ydomm))d
G.1 CURRENT GAPS BETWEEN THEORY AND PRACTICE
Although have given principled motivation for choosing the low level policy state encoder by bounding the suboptimality of our method based on the reconstructed action error, we acknowledge that there still exists several gaps between our theory and practical implementation. Firstly, assume optimality in our training objective derivation, which may not hold in practice. Specifically in the CALVIN benchmark, the data is suboptimal. Secondly, do not incorporate the architectural details of the diffusion model into our suboptimality proof, which may yield tighter bounds that what we currently have. Finally, it would be interesting to explicitly analyze the support of the data distribution, in order to give insights into the language generalization capabilities of LCD.
23

---

Published as a conference paper at ICLR 2024
H EXPERIMENTAL SETUP DETAILS
H.1 CALVIN TRAINING DATASET
In this section we give more detailed account of our training dataset for interested readers.
In order to first train our LLP, we use the original training dataset provided with the CALVN bench- mark to perform hindsight relabelling with goal states further down in the trajectory (Andrychowicz et al. 2017). At this point there is no need to incorporate the language labels, as only the high level policy needs to condition its generation on language.
Next, to train our HLP we first collect new on-policy dataset (as mentioned in Line I of Afgorihi 1 The training dataset for the high-level policy is generated using trajectories from the low-level policy, Since the original dataset contains language labels for each trajectory, policy on a goal state from the original dataset and label this rollout with the corresponding language annotation using this original annotation. We label each rollout with the language embeddings from the 1IB T5-X XXL model (Raffel et al., 2020), as mentioned in subsection 3.2
can condition the low-level
This allows to effectively tackle longer horizon control problem than the low-level policy alone, since these generated trajectories are used to conditionally train the high-level policy to generate coherent plans over longer horizon than the low-level policy. This is achieved by feeding subsampled trajectory and language embeddings to the high-level policy, and crucially, it generates a sequence of latent goals that leads to the final state. This training objective forces the high level-policy to plan several steps (through predicting a sequence) into the long-term future (through subsampling). In contrast, the low-level policy only concerned with predicting the very next action given the current state and goal state.
H.2 FURTHER DETAILS REGARDING CLEVR- ENVIRONMENT
Jiang et al. 2019) first introduced the CLEVR- Roboot Environment, a benchmark for evaluating task compositionality and long-horizen tasks through object manipulation, with language serving the mechanism for goal specification. CLEVR TLEVR-Robot is built on the MuJoCo physics engine, features an agent and five manipulable spheres. Tasked with repositioning a designated sphere relative to another, the agent follows natural language (NI.) instructions aligning with one of 80 potential goals. We use 1440 unique NL directives using 18 sentence templates, following Pang et al. (2023). For experimental rigor, we divide these instructions into training and test sets. Specifically, the training set contains 720 distinct NL instructions, corresponding each goal configuration. Consequently, only these instructions are seen during training. We compare Success Rate (SR) of different methods, with success defined as reaching some L2 threshold of the goal state for a given language instruction. To establish statistical significance, we evaluate 100 trajectories for 3 seeds each. We follow the same training methodology for our method as in the original paper, by first training Low Level Policy (LLP) on action reconstruction error, and then training diffusion model to generate goals given language for this LLP.
24

---

Published as a conference paper at ICLR 2024
I DIFFUSER-21 2D
Here we give details for our strongest Diffusion-basel ablation, which uses Stable Diffusion's VAE for generating latent plans, which outputs latent 2D feature map, with height and width 1/8 of the original image. Plans are sampled with a temporal stride of 7, such that each trajectory covers total 63 timesteps with 0, 7, 14...63. Overall, generation quality tends be higher and more temporally coherent than that of the 1D model, but low level details still not precise enough for control from pixels. For examples of model outputs, please refer subsecction K.2
T-100
T-04 T-92 T-40
T-43
T-36
T-24
T-12
T-9
Denoising
Figure 3: An overview of our Denoising process, In Figure and Figure give example of the denoising process of one of our ablations, the Diffuser-2D model. This model utilizes the 2D autoencoder of (Rombach et al. 2022 with (Janner al. 2022)
Diffusion
mean Model Diienstoon:: 128
Training Loss
Model Cimenstans: 128
Model Dimenstons: 128
Model Dimensions:
Model Dninenssion:: 64
Model Otmensions:
0.21
0.2
2.004
0.19
1.003
1.002
3002
Slap
100K 280K
3009
4006
500H
BOCK
103K 2038 300K
400K
530K
0002
100%
2008
2004
4004
500%
600K
Figure 4: Diffusson Loss Comparison. Here give study how varying the Diffusion model's size changes the performance of model. As can be seen, scaling the model from hidden dimensions to strictly increases generation quality, and would likely follow scaling laws observed (Kaplan al. 2020).
25

---

Published as a conference paper at ICLR 2024
J TASK DISTRIBUTION
Task Distribution
drawer
move slider right
move
turn off led
turn on lightbulb
bauee
prush rec block left
turn off lighbbll
pink bloek lift
Task
lift blte block sldee
push pink blook risa
rotate red block right
rotate blue block right
blook slidee
lift pink block table
rotate hlue block left
lift blue block table
push biue block right
rotate pinis block left
rotate into drawer
Percentage (%)
Figure 5: The Evaluation Task Distribution. We visualize the distribution all the tasks considered our experiments in Figure Note the long-tailedness of this distribution, and how it skews evaluation scores upwards if one can solve the relatively easier tasks that occur most frequently, such as Open Drawer, Move Slider Right, and Move Slider Left. These tasks only deal with static objects, meaning there very little generalization that needed order to solve these tasks when compared to other block tasks involving randomized block positions.
K REPRESENTATION FAILURES
K.1
DIFFUSER-ID ID (BETA-TC VAE LATENT REPRESENTATION/ FAILURES
We give few failure cases of decoded latent plans, where the latent space is given by trained from scratch 3-TC VAE on the CALVIN D-D dataset. The top row of each plan comes from the static camera view, whilst the bottom tool center point of the robot arm). The VAE trained by concatenating the images in the channel dimension, and compressing to 128 latent dimensions. Plans
comes from the gripper camera view (a camera located at the
sampled with temporal stride of such that each trajectory covers a total of 63 timesteps with t = 0, 9, 18...63. Interestingly, we found that replanning during rollout did not work, precluding the possibility of success CALVIN with our implementation of this method.
example the Close Notiee the block right of the table. Also not the entangled red and blue blocks the top left the
(b) An example the Lif Dlue Block Slider Task. The gripper view temporally
incoberent, red blue blocks slider entanledd.
blocks begin appear the table the end of generation. The red black is also not clearly generased
example
(d) example the Right becomes red by the end of the static view, whereus the opposite happens in the gripper view.
26

---

Published as a conference paper at ICLR 2024
K.2 DIFFUSER-21 2D (STABLE DIFFUSION LATENT REPRESENTATION FAILURES
We additionally give some failure cases for Diffuser-2D. For more information on the training of this model, please refer Appendix I this model.
We also found that replanning during rollout did not work with
An esample of the Lift Red Block Drawer Tisk. Note the pink block that disapgears.
(b) An example of the Lift Blue Block Drawer Task. The gripper am is entangled block.
An example of the Lift Pink Block Slider Task. Note the entangled red/biue
example Close Dravr Note eatangled pink/hlae blocks.
27

---

Published as a conference paper at ICLR 2024
L TSNE COMPARISON BETWEEN GROUD TRUTH (GT) TRAJECTORY AND DIFFUSER 1D (DM) TRAJECTORY
In order better understand whether the representation failures found in Appendix K are result of the underlying encoder or the diffusion model, we visualize the TSNE embeddings an encoded successful trajectory from the dataset, which we refer as a Ground Truth trajectory, and the TSNE embeddings of generated trajectories from Diffuser- ID (DM) in Figure 8 If we observe that the DM's embedddings are fairly close the GT- GAE'ss,,, then we can reasonably presume that the VAE is the failure mode, whereas if the trajectories are wildly different this would imply that the DM failing to model the VAE's latent distribution properly. Here, all samples other than appear to fairly close, so we suspect that the failure case lies in the underlying latent distribution and not the DM's modeling capabilities. This is further backed by LCD, as we show that by using the proper underlying latent space with LLP leads to success.
Sample
Sample
Sample
Sample
Stapple
Sample
(p) Sample 7
(b) Sample 8
6 All
Figure 8: TSNE visualization of GT- VTE trajectory vs. Diffuser- 1D trajectory, where the purple and light blue color range is the ground truth VAE, and the copper color range is Diffuser- ID. All states are normalized, and all trajectories taken from the task "lift pink block table".
28

---

Published as a conference paper at ICLR 2024
M HULC LATENT PLAN TSNE
We give TSNE embeddings of several Latent Plans generated during inference by HULC below.
de Jel, 10M
lamt an Liditiull 2X0
plact puller
Herr.ruliewe 200
Dame alt wettbck
mouog der/tftt 130
DC miiaefrree 11.3
block drower 240)
RloweeMocuueeee 10.0
Di Fww.e 20.0
modee tiak left
block
tut.ouugghicccc U.4
1a (KM.JRA_AaK.MM.l 11.0
Movt rale,
push Meoouggece
push track
pust Muk stoucee, 30 more
Figure 9: TSNE of Latent Plan. give TSNE emhedding The latent plan space is the communication layer between the high level policy and low level policy of the HULC model, which corresponds to the intermediate layer between the lower level and lowest level policy in method. We clarify that this the latent goal space that model does generation in. Our method performs latent generation in the carlier layer from the output of the goal encoder, which corresponds 32 latent dimensions.
latent plan space of HULCC in Figure 9
29

---

Published as a conference paper at ICLR 2024
N MODEL CARD FOR LANGUAGE CONTROL DIFFUSION
A hierarchical diffusion model for long horizon language conditioned planning.
N.1
MODEL DETAILS
N.1.1
MODEL DESCRIPTION
A hierarchical diffusion model for long horizon language conditioned planning.
N.2
USES
N.2.1
DIRECT USE
Creating real world robots, controlling agents in video games, solving extended reasoning problems from camera input
N.2.2
DOWNSTREAM USE
Could be deconstructed so as to extract the high level policy for usage, or built upon further by instantiating multi-level hierarchical policy
N.2.3 OUT-OF-SCOPE USE
Discrimination in real-world decision making, military usage
N.3 BIAS, RISKS, AND LIMITATIONS
Significant research has explored bias and fairness issues with language models (see, e.g., Shhng al. (2021) and [ender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups
N.4 TRAINING DETAILS
N.4.1
TRAINING DATA
N.5
ENVIRONMENTAL IMPACT
Carbon emissions be estimated using the Machine Learning Impact calculator presented in
Lacoste et al. (2019)
Hardware Type: NVIDIA Titan RTX, NVIDIA A10
Hours used: 9000
Cloud Provider: AWS
Compute Region: us-west-2
Carbon Emitted: 1088.64 kg
N.6
TECHNICAL SPECIFICATIONS
N.6.1
MODEL ARCHITECTURE AND ORIECTIVE
Temporal U-Net, Diffusion objective
30

---

Published as a conference paper at ICLR 2024
N.6.2
COMPUTE INFRASTRUCTURE
Hardware
Nvidia Titan RTX, Nvidia A10
Software Pytorch
31


Reviews:
[{"review_id": "pImdNaXavL", "reviewer": "Reviewer_fAWH", "summary": "The authors propose a novel hierarchical framework called LCD, which leverages a language-conditioned diffusion model as a high-level goal planner on top of HULC. The proposed model achieves scalability in the spatial, time, and task dimensions. It outperforms other baselines on the CALVIN benchmark.", "questions": "1. The experimental results seem to indicate that HULC plays an important role in the success of LCD. Do the authors believe that the performance of LCD necessarily relies on a well-trained HULC baseline? Can LCD still achieve excellent performance with a simple goal-conditioned policy and a well-designed RL encoder representation? Alternatively, if a HULC baseline is used, can a Transformer with a parameter size comparable to the Diffusion HLP achieve similar performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is simple and effective, achieving scalability in the spatial, time, and task dimensions.\n2. By employing a hierarchical decision-making approach, the algorithm reduces the model size and inference time.\n3. The authors experimentally demonstrate that vanilla Diffuser fails to successfully replan in high-dimensional state spaces when using a Variational Autoencoder (VAE). This provides insights for researchers interested in using Diffuser for planning tasks with image state spaces.", "weaknesses": "1. In Section 4.5 of the experiments, the parameter sizes of MLP and Transformer are significantly smaller than the parameters of Diffusion. This may introduce unfairness in the experiments. Additionally, I did not notice a detailed explanation of how the authors perform inference using MLP and Transformer. Since the authors employ a Diffusion planning paradigm as HLP, it may be worth considering a better comparison with planning-based MLP (e.g., model-based RL) and Transformer (e.g., Decision Transformer).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7yiZml58ZC", "reviewer": "Reviewer_z6e9", "summary": "In this work, the authors propose to scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long-horizon control problems conditioned on natural language instructions. Specifically, they take a hierarchical diffusion approach by training diffusion policy to plan in the latent plan (induced by a frozen low-level policy) every c steps (temporal abstraction). They leverage the language-conditioning capabilities of existing diffusion architecture to learn language-conditioned hierarchical policies.", "questions": "See “weaknesses”", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed approach achieves SOTA performance on a recently proposed language robotics benchmark\n+ The method is well-motivated and reasonable", "weaknesses": "The weakness comes from a combination of lack of originality and broadness of the experiments. While the approach is very reasonable (applying language-conditioned latent diffusion in the abstracted state and action space for high-level planning), its general idea is similar to the ones in the literature (e.g., [1]). The major differences are the use of diffusion models and the specific choice of temporal abstraction. Based on this, I would like to see more experiment evidence, e.g., beyond the CALVIN benchmark, or real-world experiments (as in SPIL).\n\n[1] Parrot: Data-Driven Behavioral Priors for Reinforcement Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C7yxLXp9wz", "reviewer": "Reviewer_FYfw", "summary": "This paper studies text-to-control problems and proposes Language to Control Diffusion models as a hierarchical planner conditioned on language (LCD). This hierarchical approach scales the diffusion model along the spatial, time, and task dimensions.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and easy to follow.\n2. The proposed hierarchical approach makes intuitive sense to scale the control diffusion models.\n3. The authors provide both theoretical guarantees and experimental results.", "weaknesses": "1. The authors stated that the proposed algorithm avoids \"the usage of a predefined low level skill oracle\". Isn't the low-level controller also adopted by LCD in this work? Do the authors refer to the imitation learning setting where we only access to the trajectories instead of the controller?\n2. Can the authors comment more on the difference between the proposed method and previous text-to-video works, such as [1, 2]? Again, the authors mentioned that \"they again avoid the issue of directly generating low level actions by using a low-level controller or an inverse dynamics model which we are able to directly solve for\", but I didn't see the the downside of leveraging low-level controllers.\n3. Besides, I would like to know if these text-to-video methods are directly comparable to LCD in experiments.\n4. It is not very clear to me what dom(P(s'|s, a)) measures and the intuitive sense behind it. For terms that are not commonly used in RL theory, it would be better to state its definition and intuition of the proof/bound. It is also not clear what the circle means in the definition of the low-level policy.\n5. The theorem comes before the practical instantiation of the algorithm. The gaps between theory and practice should be explicitly analyzed.\n\n[1] Learning Universal Policies via Text-Guided Video Generation.\\\n[2] VIDEO LANGUAGE PLANNING.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UUmTO7PzHM", "reviewer": "Reviewer_74cS", "summary": "This paper aimed to introduce the hierarchical diffusion policy into robotics control based on language instructions, while this framework faces three challenges: direct long-horizon planning, non-task-specific representation, and computational inefficiency. By proposing a diffusion-based model named Language to Control Diffusion (LCD), this paper addressed these issues.", "questions": "1. It seems low-level policy and the state encoder are coupled in HULC, is it possible to pretrain both modules separately via different objectives?\n\n2. What is the total number of parameters in LCD?\n\n3. Learning state mapping functions and latent planning models separately seems quite reasonable in RL tasks, is there a way to combine a pre-trained encoder on a more extensive visual dataset, similar to models like VQ-VAE used in Stable Diffusion, for state feature extraction, rather than pre-training the encoder solely on robot tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-structured and technically sound. The core idea is to pretrain a state encoder and then utilize the latent diffusion model to instantiate high-level policy, generating high-level goals by following the language instruction. The generated goals are then fed into the low-level policy network to obtain the final action.  The empirical results show performance improvements over the previous approaches. The authors also provided many empirical insights of the utilization of diffusion model for visual decision making tasks, which is valuable.", "weaknesses": "1. It appears that some crucial descriptions of the model are missing. For instance, there is little mention in the paper about how the state encoder is trained. After going through HULC[Mees et al., 2022a], I just noticed that the encoder is possibly the one in HULC model. Therefore, the author considers both the encoder and LLP to be pretrained. I suggest the author add the corresponding description because it is currently hard to find out this from the logical flow of the paper. It is still important to clarify the training objective and the structure of the state encoder and low-level policy.\n\n2. The novelty of the paper is limited, as it is a combination of diffusion model and HULC.\n\n3. The assumption of the dataset being optimal is very strong in real-world settings, which limits the applicability of the proposed method. \n\n4. The textual encoder -  T5-XXXL model in this paper - is quite large, which increases the inference time. Is it possible to use the textual encoder in CLIP?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)