PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play
Authors: Lili Chen
Shikhar Bahl
Deepak Pathak
Venue: CoRL 2023 Poster (2023)
Keywords: Diffusion Models
Learning from Play
Language-Driven Robotics

Abstract: Learning from unstructured and uncurated data has become the dominant paradigm for generative approaches in language or vision. Such unstructured and unguided behavior data, commonly known as play, is also easier to collect in robotics but much more difficult to learn from due to its inherently multimodal, noisy, and suboptimal nature. In this paper, we study this problem of learning goal-directed skill policies from unstructured play data which is labeled with language in hindsight. Specifically, we leverage advances in diffusion models to learn a multi-task diffusion model to extract robotic skills from play data. Using a conditional denoising diffusion process in the space of states and actions, we can gracefully handle the complexity and multimodality of play data and generate diverse and interesting robot behaviors. To make diffusion models more useful for skill learning, we encourage robotic agents to acquire a vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process. In our experiments, we demonstrate the effectiveness of our approach across a wide variety of environments in both simulation and the real world. Video results available at https://play-fusion.github.io.

PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play
Lili Chen
Shikhar Bahl* Carnegie Mellon University
Deepak Pathak
Abstract: Learning from unstructured and uncurated data has become the dominant paradigm for generative approaches in language and vision. Such unstructured and unguided behavior data, commonly known as play, is also easier to collect robotics but much more difficult to learn from due to inherently multimodal, noisy, and suboptimal nature. In this paper, study this problem of learning goal-directed skill policies from unstructured play data which is labeled with language in hindsight. Specifically, we leverage advances in diffusion models learn a multi task diffusion model to extract robotic skills from play data. Using a conditional denoising diffusion process in the space of states and actions, we can gracefully handle the complexity and multimodality of play data and generate diverse and interesting robot behaviors. To make diffusion models more useful for skill learning, we encourage robotic agents acquire vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process. In our experiments, we demonstrate the effectiveness of our approach across a wide variety of environments in both simulation and the real world. Video results available at
fusion.github. io.
Keywords: Diffusion Models, Learning from Play, Language-Driven Robotics
Introduction
Humans reuse past experience via a broad repertoire skills learned through experience that allows us to quickly solve new tasks and adapt across environments. For example, one knows how to operate and load dishwasher, many of the skills (e.g., opening the articulated door, adjusting the rack, putting objects in) will transfer seamlessly. How learn such skills for robots and from what kind is a standinn research question. Robotic skill abstraction has been studied transfer knowledge between environments and tasks [1, 2, 3]. It has been common to use primitives as actions in the options framework [4, 5], which are often hand-engineered [6. 7, 8, 9, 10, 11] or learned via imitation [12, 13, 14]. These allow for much more sample-efficient efficient control but require knowledge of the task and need to be tuned for new settings. On the other hand, there have been efforts to automatically discover skills using latent variable models [15, 16, 17, 18, 19, 20, 21, 22]. While they can work in any setting, such models are often extremely data-hungry and have difficulty scaling to the real world due to the data quality at hand.
way to
As result, real-world paradigms are based on imitation or offline reinforcement learning (RL) but both these require several assumptions about the datasets. In imitation learning, human teleoperators must perform tasks near-perfectly, reset the robot to some initial state, perform the task near-perfectly again, and repeat several times. In offline RL, data assumed to contain reward labels. which is impractical in many real-world setups where reward engineering is cumbersome. In contrast, it is much easier to collect uncurated data from human teleoperators if they instructed only explore, resulting in play data [21, 22, 23]. Learning from play (L.PP) has emerged as a viable alternative to traditional data collection methods for behavior generation. It offers several advantages: (1) it is efficient because large datasets of play can be collected without the need for setting up and executing
"Equal contribution.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

open nnO grill
reRove bread from toweter
use spatula
place toast
open
pick
plck the knni
Language-Annotated Play
Figure 1: Across multiple real-world and simulated robotic settings, we show that our model can extract semantically meaningful skills from language-annotated play data. Such data is highly multimodal and offers no optimality guarantees. Video results of PlayFusion are available at https: http://pllyuuuaaee fusion, aeeiiongittub. github. io.
perfect demonstrations, and (2) the data collected is rich and diverse because it contains broad distribution of behavior ranging from completions of complex tasks to random meandering around the environment. An important quality of such data is that it is grounded with some semantic goal that the "player" is aiming to achieve. We believe a simple abstraction for this is language instructions, which can describe almost any play trajectory.
major challenge in learning from play is that the data is highly multimodal, i.e., there are many different ways achieve specific goal, and given sample from the play data, there many different goals that could have generated it. One popular way to handle highly multimodal data is by modeling the full distribution via generative models. In recent years, there has been remarkable progress in large generative models [24, 25, 26, 27], especially in the class of diffusion models [28, 29], which have been shown to generate high-resolution images a property well suited for vision-based robotic control. In fact, diffusion models have shown to effective capturing complex, continuous actions [30, 31, 29, 32, 33] in the context of robotics. However, these diffusion model-based approaches have not been empirically shown yet work on unstructured data. We argue that the ability of diffusion models to fully capture complex data paired with their potential for text-driven generation can make them good candidates to learn from language- anguage-annotated play data.
One additional consideration is that in reality, humans only deal with a few skills. Almost every task manipulation task involves some grasping and some post-grasp movement. believe that learning discrete skills will not only make the whole process more efficient but will also allow interpolation between skills and generalizations to tasks. To address this, propose PlayFusion, diffusion model which can learn from language-annotater play data via discrete bottlenecks. We maintain the multimodal properties of our current system while allowing for more discrete representation of skills. Empirically, we show that our method outperforms state-of-the-art approaches on six different environments: three challenging weal-world manipulation settings as well as the CALVIN [34), Franka Kitchen [22], and Ravens [35, 36] simulation benchmarks.
2 Related Work
Goal and Language Conditioned Skill Learning One method of specifying the task is via goal- conditioned learning, often by using the actual achieved last state as the goal [37, 38, 39, 40J. There is also recent work on using rewards to condition robot behavior [41], J. but this requires a rewerd-labeled dataset, which makes stronger assumptions than play data. Furthermore, there is a large body of work on condittoon learning [42, 36, 43, 44, 45, 46, 47], which specifies the task through language instructions. Instead of conditioning the policy on fully labeled and curated data, take advantage of unstructured play data which is annotated with language in hindsight.

---

p(xx-1 pok, 9, s)
X
K-1
X
Xk-1
Diffussonn Model
robot exccution
s:
similarity score
codebook
place the bread on the sandwich
Overview of how PlayFusion extracts useful skills from annotated play by leveraging discrete bottlenecks in both the language embedding and diffusion model U-Net. We generate robot trajectorics via an iterative denoising process conditioned on language and current state.
Figure 2:
Learning from Play Unlike demonstrations, play data is not assumed to be optimal for any specific task as it is collected by human teleoperators who are instructed only to explore. Play-LMP and MCIL [21, 48] generate behaviors by learning motor primitives from play data using VAE [49, 50]. RIL [22] is a hierarchical imitation learning approach and C-BeT [23] generates behaviors using transformer- based policy and leverages action discretization handle multimodality. LAD [51]
incorporates diffusion for learning from play, but keeps several components of VAE-based approaches for encoding latent plans; we forgo those elements completely.
Behavior Modeling with Generative Models A promising architecture for behavior modeling with generative models is the diffusion probabilistic model [28, 52, 53, 54]. Diffuser [30], Decision Diffuser [31], Diffusion-QL. [29] and IDQL [32] apply diffusion models to the offline reinforcement learning (RL) problem. In real-world robotic applications, Diffusion Policy [33] demonstrated strong results in visuomotor policy learning from demonstrations. Different from these works, we learn from play data containing semantic labels instead of offline RL datasets expert demonstrations. Some approaches [55, 56] incorporate diffusion in robotics but not for generating low-level actions.
Discrete control A key challenge in robot learning is the exponentially large, continuous action space. Option or skill-based learning is appealing as it can circumvent this problem and allow the agent to learn in structured, countable action space [57, 58, 59, 60]. Learned action discretization [52, 23] has allowed approaches to scale to complex tasks. C-BeT [23] applies real-world robotic control with transformers [41, 61, 62] the goal-conditioned setting; [63] train dynamics model over discrete latent states. We leverage the discrete properties of VQ-VAEs and their natural connection to language-labeled skills.
3 Background
Denoising Diffusion Probabilistic Models (DDPMs) DDPMs [28] model the output generation process as a denoising process, which is often referred as Stochastic Langevin Dynamics. To generate the output, the DDPM starts by sampling = from Gaussian noise distribution. It then performs series of denoising iterations, totaling K iterations, to generate sequence of intermediate outputs, zt =- z0. This iterative process continues until noise-free output z is produced. The denoising process is governed by the following equation:
k-
aa (z k) + N(O,0"1)
Here, 6g represents the noise prediction network with a learnable parameter 0, and (O, r²)) denotes the Gaussian noise added at each iteration. This equation used generate intermediate outputs with gradually decreasing noise levels until noise-free output is obtained. To train the DDPM, the process begins by randomly selecting z from the training dataset. For each selected sample,

---

denoising iteration & is randomly chosen, and a noise ct is sampled with the appropriate variance for the selected iteration. The noise prediction network is then trained to predict the noise by minimizing:
L ||e* E0 ee k)]1?
(2)
Discrete Representations We utilize VQ- [64] inspired models in PlayFusion as they can provide a way to discretize the skill space. Given an input æ, a VQ-VAE trains an encoder to predict latent Ex)) = 2 and maintains a codebook of discrete latent codes e. The VQ layer selects as arg min; || eill, finding the closest code to the embedding, which is used to reconstruct æ. The
training loss is
LVQVAE = Crecou( ((e))) + || - sg(ee) + [|=g(2) - e,1l2
where D is the VQ-VAE decoder. The reconstruction loss is augmented with a quantization loss, bringing chosen codebook embedding vectors ej toward the encoder outputs in order to train the codebook, as well a loss to encourage the encoder to "commit" to one of the embeddings.
Learning from Play Data (LIP) the LiP setting, we are given a dataset {(s, E S A. There assumptions about tasks performed in these sequences the optimality of the data collection method. Similar to the formulation of [23], the goal is to learn policy = S x S - A where the input is the current state 8, and goal y 87. In some cases, (including ours), the goals are instead described via language annotations.
PlayFusion: Discrete Diffusion for Language-Annotated Play
Humans do not think about low-level control when performing everyday tasks. Our understanding of skills like door opening or picking up objects has already been grounded in countless prior experiences, and prior experiences successes, failures, and everything in between. PlayFusion focuses on learning these skills through (anguage-annotaate play data.
comfortably perform these in settings. Skills acquired through our
However, learning from play data is still difficult as continuous control skills are not easy to identify due to several challenges: (1) data can come from multiple modalities as there are many actions that the robot could have taken at any point, (2) want the model acquire vocabulary of meaningful skillsm and (3) want generalize beyond the training data and have the model transfer skills to new settings. To address the challenges, we leverage recent advances in diffusion model large-scale text-to-image generation. Such models [33, 30, 29] can inherently model multimodality via their iterative denoising process. To effectively transfer skills to new settings, we propose a modified diffusion model with the ability to discretize learned behavior from language-annotated data. Figure 2 shows an overview our method.
4.1 Language Conditioned Play Data
Our setup consists of language conditioned play data [21] Dplay = [({])) ( long sequences of robot behavior data containing many kinds of behaviors, collected by human operators instructed to perform interesting tasks. In this setting, we assume that there some optimality to the data, i.c. ag F(st, 2g), where 2g is latent variable that models the intention of the operator. We thus leverage language labels to estimate zg. Given sequence
{&,, a, k label with an instruction / which is passed into language model [65], Shang referring to it as ž throughout the paper. One can also use goal images, but we might not have access to these at test time. While our method can use any 2g as conditioning, assume that the play data has access to language annotations i. ( [s. 21) contains a few simple components. We use a ResNet[666]-aase visual encoder d to encode 8, (a sequence of images) and an MLP based langauge encoder di downproject the language embedding 21. The policy uses g [x)) (s:)) as conditioning the action decoder fan- Previous approaches [21, 34] use latent variable models to deal with multimodality. We find that modelling fact as diffusion process enables us to circumvent this.
Our policy 7

---

CALVIN
Franka Kitchen
Ravens
Sink
Dining Table
Cooking
Figure 3: Simulated (top row) and real-world (bottom row) environments used for our evaluations. In cach real-world setup, the robot is tasked with picking up one of the objects (e.g., plate, cup, carrot, bread, corn) and relocating it to specified location (e.g., drying rack, plate, toaster, grill, pot).
4.2 Multi-modal Behavior Generation via Diffusion
With fact, we aim to predict robot actions given the current state, using DDPM to approximate the conditional distribution P( In our setting, we additionally condition on the goal g. Formally, we train the model to generate robot actions a, conditioned on goal and current state 8t. so we modify Equations and 6 to obtain:
a, k-1 ala; yeolo, 8t. az k) + NO, o21))
(4)
co L C= - 81,
k)]l2
(5)
We the notation above for simplicity, but in practice, predict sequence of T, future actions " dr+T, instead of only the most immediate action, technique known action chunking. This is done in some recent works [33, 67] and is shown to improve temporal consistency.
4.3 Discrete Diffusion for Control
Moreover, humans often break down tasks into smaller skills, which are often repeatable. In fact, most tasks can be achieved with a relatively small set. On other hand, both latent goals that we learn as well the action diffusion process are continuous. Making sure learnt skills are discrete can only allow for better performance but also better generalization to settings. However, naively lead to suboptimal behavior. We want ensure that conditioned
enforcing discretization latent goal, y. action predictions from faut are both multimodal and yet only represent few modes. Thus, we propose discrete bottleneck instead.
For the action generation process to represent useful skill space, we want to enforce discreteness where the actions interact with latent goal. PlayFusion adds a vector quantization bottleneck in the diffusion process, specifically in the network c0(x)) ca ca(g, 8t, af + ck k)) co is U-Net which fuses the language conditioning into the action denoising. We modify the U-Net architecture with codebook of discrete latent codes cu. a discrete bottleneck for the diffusion model. Given an input x the U-Net encoder produces a latent (x), which is passed into the decoder to produce co(x:) = Ye (v) This bottleneck layer selects j as arg min; ||v: (x) - e:].. finding the closest code to embedding, which is used to reconstruct æ. To account for this, we augment the training procedure with the quantization and commitment losses, similar to VQ-VAE.
Generalization via discrete language conditioning Consider agent that has learnt skills formed from the atomic units A, B, C and C, of the form A B, C and C D. To truly extend its
5

---

capabilities beyond the initial training data, the agent must learn to interpolate and extrapolate from these existing skills, being able to perform tasks like A + that it hasn't It explicitly been trained on. Given that the action generation in the diffusion process already quantized, our hypothesis that a discrete goal space will synergestic and allow the policy to compose skills better. Thus, we maintain a codebook of discrete latent codes et for the language embeddings output by the language goal network di (21), selecting Clij which is closest to (2)). The full loss function that we use to train PlayFusion is as follows:
PlayFusion =Ile" E0(x + x=122) + B1 (1sg(()(()) Eusy )ll2 +B1 [1()(()) - sg(eus) lll
U-Mot quantization losa
U-Kat comni tmant lass
B2 [|gg(d((z)) - ens 1lI +B2 (()((1)) sg(el. 1|
lang-
quantization loss
lang- commi Ement 1o88
(6)
where 31 and 32 coefficients determine the tradeoff between covering diversity of possible behaviors and encouraging behaviors belonging to similar skills to be brought close to each other.
Sampling from PlayFusion Given a novel language instruction at test time a', we obtain the quantized encoding (z'),, combining it with the visual encoding to get conditioning g'. We sample a set of actions a ~ (0, 1), pass them through the discrete denoising process Equation 4.
5 Experiments
In this section, investigate PlayFusion and its ability to scale to complex tasks, well as generalization to new settings. We ask the following questions: (1) Can PlayFusion allow for learning complex manipulation tasks from language annotated play data? (2) Can our method perform efficiently in the real-world setup beyond the simulated environment? (3) How well can PlayFusion generalize to out of distribution settings? (4) Can PlayFusion fact learn discrete skills? (5) How do various design choices, such as quantization, language conditioning. etc., affect PlayFusion? We aim answer these through experiments in three different simulation and real world settings.
Environmental Setup We test our approach across a wide variety of environments in both simula- tions as well as the real world. For simulation, we evaluate three benchmarks: (a) CALVIN [34], (b) Franka Kitchen [22], and (c) Language-Conditioned Conditioned Ravens [35, 36). For the real-world setup, we create three different environments: cooking, dining table and sink, shown in Figure 3. More details of the environment setup are in the supplementary.
Baselines We handle task conditioning in the same way for method well all baselines, using the same visual and language encoders. We compare our method with the following baselines: (a) Learning Motor Primitives frrrreeffrom from Play (Play-LMP): Play-LMP [21] generates behaviors by learning motor primitives from play data using a VAE, which encodes action sequences into latents and then decodes them into actions. (b) Conditional Behavior Transformer (C-BeT): C-BeT [23] generates behaviors using transformer- bassed policy and leverages action discretization handle multimodality. (c)
Behavior Cloning (GCBC): GCBC [21, 68) is conditional behavior cloning.
5.1 Results in Simulation and Real World
Flsyon in simulation Table 1 shows success rates for PlayFusion, Play-LMP, C-BeT, and GCBC on the simulation benchmarks. On both CALVIN setups, we outperform the baselines by wide margin, which demonstrates the effectiveness of our method in large-scale language-conditioned policy learning from complex, multimodal play data. The baselines perform comparatively better on the Franka Kitchen environments, where the training datasets smaller and the data covers more narrow behavior distribution and the benefit of handling multimodality is smaller, however,

---

Simulation Kitchen
Real World Cooking
CALVIN
CALVIN B
Kitchen B
Ravens
Dining Table
Sink
C-BeT [23]
26.3 0.8
0.9
45.6 + 2.3
24.4 = 2.3
13.4
20.0
0.0
10.0
Play-LMP [21]
19.9 1.0
22.0
1.9 1.5
0.0 0.0
0.2
0.0
0.0
0.0
GCRC [21]
2.0
3.4+1.4
3..0+3.3
15.5 4.5
1.6
5.0
0.0
5.0
Ours
42
58.7 0.7
47.5 = 2.0
27.7 = 0.9
35.8
45.0
30.0
20.0
Table 1: Success rates for PlayFusion and the baselines simulation and real-world settings. PlayFusion consistently outperforms all of the baselines.
PlayFusion still outperforms or matches all baselines. PlayFusion also achieves significantly higher success rate than the baselines Ravens (see appendix for per-task results), which is not large- scale as CALVIN but covers large portion of the state space due to the diversity of instructions.
Long horizon tasks Using the Long Horizon CALVIN evaluation suite, we test the ability of agents stitch together different tasks, with transitioning between tasks being particularly difficult. One such long horizon chain might be "turn on the led" - "open drawer" + "push the blue block - "pick up the blue block" + "place in slider". We rollout 128 different long horizon chains containing five instructions each and record the number of instructions suc- cessfully completed. As shown in Table 2, find that PlayFusion significantly outperforms the baselines in both CALVIN A and CALVIN B. The diffusion process gracefully handles the multimodality of not only each individual task in the chain but also of the highly varied data the agent has seen of transitions between tasks.
Instructions
Av. Seq Len
AALVINA:
C-BeT
0.262
25.2
1.0
0.0
0.0
0.0
Play-LMP GCBC
0.175
16.5
1.0
0.0
0.0
0.0
0.194
19.4
0.0
0.0
0.0
0.0
CALVIN B:
C-BeT
0.272
27.2
0.0
0.0
0.0
0.0
Play-LMP
0.117
11.7
0.0
0.0
0.0
0,0
GCBC
0.291
27.2
1.9
0.0
0.0
0.0
Ours (A)
0.417
37.1
2.9
1.0
0.0
0.0
Ours
0.611
54.4
6.0
0.0
0.0
0.0
Table 2: Average sequence length on Long Horizon CALVIN and success rate for the n-th instructions.
Generalization in the real world Table shows results for PlayFusion and the baselines in our real world evaluation setups. These setups are particularly challenging for two reasons: (1) inherent challenges with real-world robotics such as noisier data and constantly changing environment conditions such as lighting, and (2) they are designed to test skill-level compositional generalization. Specifically, the agents are required to compose skills + and they might be trained on "pick up the carrot and place it in the pan" and "pick up the bread and put it in the toaster" and must generalize to "pick up the carrot and put in the toaster". Our method significantly outperforms the baselines in these settings, showcasing the ability of the diffusion model in modeling complex distributions and the emergence of learned skills via the discrete bottleneck. Video results are at https: http://laaylliaalleeeeeeeeeeaaaa fusion. gi github..o
into A D; for example,
5.2 Analysis of Discrete Representations
Learning discrete skills Table 3 studies the impact of
Bread
Pan
discrete bottlenecks (for Ravens results, see the ap-
Pinaspple
pendix). The success rate is, on average, worsened with the
removal of either the U-Net discretization and the language embedding discretization. We also qualitatively study whether semantically similar skills are actually mapped to similar areas of the latent space and should therefore brought together by the discrete bottleneck. In Figure 4, show that skills involving similar locations (c.g., pan) or objects (c.g., carrot) encoded into similar embeddings. In Figure 4, we show the embeddings
t1 oven
Carrot
Carrat
Figure 4: Visualization of the codebook beddings for various real- collworld skills.
7

---

of different trajectories. The top two rows share the first skill (which is to remove the lid from the pan) and place an object in the pan. The bottom two rows share the second skill (grasping the carrot). Embeddings that contain the same skill have a similar pattern, which further indicates that the latent skill space being learned is somewhat discretized.
Balancing the discrete bottlenecks In Table 4, we study the effects of different 31 and 82 values on CALVIN A performance, i.e., the relative weightings for the additional terms in the loss function corresponding to the U-Net discretization and language embedding discretization. We find that 81 = 32 = 0.5 results in the best performance. In general, equally weighing the four additional losses (two for U-Net and two for language) leads to improved performance over imbalanced weightings. 31 = 32 = 0.5 is also better than 31 = 32 = 1, indicating that ever-incentivizing discretization can be detrimental to diffusion model learning. Further analyses can be found in the appendix.
Methods
CALVIN A
CALVIN B
Ours
45.2 1.2
58.7 1 0.7
No U-Net discretiz. No lang discretiz.
45.3 +: 2.1
56.1 + 14
40.3 +: 1.6
54.1 + 1.2
Table 3: Effect of discrete bottlenecks.
5.3 Ablations of Design Choices
Effect of language model Although our method orthogonal the language model used, we test its sensitivity this. As shown Table 4, find that common models such as MiniLM [65]. Distil- roberta [69], MPNet [70], and BERT [71] have similar performance, showing that PlayFusion is mostly robust to this design choice. We hypothesize that the discrete bottleneck applied to the language em- beddings helps to achieve this robustness. CLIP [72] embeddings result in much lower success rates, most likely due the fact that Internet images may not contain similar "play data" instructions.
Success Rate
Effect of conditioning
Global
54.1
Conditional Noise
40,2
Visual Pre-training
38.1
Effect of language model:
ML-MinnLLM---v22
47.1
all-distilroberuu- 11-mpacct-base-22 BERT
48.4
48.8
48.
CLIP (ResNer50)
35.2
Effect of conditioning Table studies various different possibil- ities for conditioning the diffusion model generations language and vision in CALVIN A. When working with diffusion models there are multiple different ways we can approach how to feed it goals, images of the scene etc. We found that PlayFusion is mostly robust to this, with global conditioning providing benefits for smaller mod- els (such as those in the real world). We also attempted to condition the diffusion model noise the goal but found that this negatively impacted performance. For the visual conditioning, we studied the effect of initializing the image encoder with large-scale pre-trained models [73]), finding that it does not help, and PlayFusion can learn the visual encoder end-to-end from scratch.
CLIP (VIT33)
43.9
Loss weights (U-Ner & Language)
0.5 & 0.5
47.1
1
45.1
0.1 & 1
45.5
1 1 0.1 0.25 & 0.75
43.4
37.7
0.75 0.25
43.4
Table 4: Effects conditioning. language model, and loss weights.
For data scaling curves and more analyses on design choices, see the appendix.
6 Limitations and Discussion
In this paper, we introduced a novel approach for learning a multi-task robotic control policy using denoising diffusion process on trajectories, conditioned on language instructions. Our method exploits the effectiveness of diffusion models in handling multimodality and introduces two discrete bottlenecks in the diffusion model in order incentivize the model to learn semantically meaningful skills. PlayFusion does require the collection of teleoperated play data paired with after-the-face language annotations, which still require human effort despite being already less expensive and cone-consuning to collect than demonstrations. It would be interesting to label the play data with a captioning model or other autonomous method. Furthermore, there is room for improvement in our performance on our real-world setups. Additionally, our real-world experiments could be expanded even more complex household settings such study rooms, bed rooms, and living rooms. Overall, approach can significantly enhance the ability of robots to operate autonomously in complex and dynamic environments, making them more useful in wide range of applications.
8

---

References
[1] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: framework for temporal abstraction reinforcement learning. Artificial intelligence, 1 112(1-2):11122) 1999.
[2] S. Thrun, C. Faloutsos, T. Mitchell, and L. Wasserman. Automated learning and discovery state-of-the-art and research topics in rapidly growing field. Ai Magazine, 20(3):78-78, 1999.
[3] M. Pickett and A. G. Barto. Policyblocks: algorithm for creating useful macro-actions in reinforcement learning. In ICML, volume 19, pages 506-513, 2002.
[4] P.-L.. Bacon, J. Harb, and D. Precup. The option-critic architecture. In AAAI, 2017.
[5] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction reinforcement learning. Artificial Intelligence, 1999.
[6] C. Daniel, G. Neumann, O. Kroemer, and J. Peters. Hierarchical relative entropy policy search. Journal of Machine Learning Research, 2016.
[7] Stulp,
Theodorou, and S. Schaal. Reinforcement learning with sequences of motion primitives for robust manipulation. Transactions Robotics, 2012.
[8] J. Kober and J. Peters. Learning motor primitives for robotics. In ICRA, 2009.
[9] Pastor, M. Kalakrishnan, S. Chitta, E. Theodorou, and S. Schaal. Skill learning and task outcome prediction for manipulation. In ICRA, 2011.
[10] M. Dalal, D. Pathak, and R. R. Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. NeurlPS, 2021.
[11] Nasiriany, H. Liu, and Y. Zhu, Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In ICRA, 2022.
[12] K. Pertsch, Y. Lee, and J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on robot learning, pages 188-204. PMLR, 2021.
[13] S. Bahl, A. Gupta, and D. Pathak. Hierarchical neural dynamic policies. RSS, 2021.
[14] K. Pertsch, R. Desai, V. Kumar, F. Meier, J. Lim, D. Batra, and A. Rai. Cross-domain transfer via semantic skill imitation. arXiv preprint arXiv:2212.074072222 2022.
2022.
[15] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: learning skills without reward function. arXiv preprint arXiv:1802.06077 2018.
[16] A. Sharma, S. Gu, Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint NXXX::997.001555 2019.
[17] J. Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39 2020.
[18] Shankar and A. Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, pages 8624-8633. PMLR, 2020.
[19] Т. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and P. Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, pages 3418-3428 PMLR, 2019.
[20] W. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics embeddings. arXiv preprint 17X1v 2019. 2019.

---

[21] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. arXiv preprint arXiv: /903.0197,, 2019.
[22] A. Gupta, Kumar, C. Lynch, Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint 191.119900.19566 2019.
[23] Z. J. Cui, Y. Wang, N. Muhammad, L. Pinto, et al. From play to policy: Conditional behavior generation from uncurated robot data. arXiv preprint arXiv:22 10. 10047, 2022.
[24] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021.
[25] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[26] R. Rombach, A. Blattmann, D. Lorenz, P.
Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[27] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,
2020. 2020.
[28] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6855 2020.
[29] Z. Wang, J. J. Hunt, and M. Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.0611 2022.
[30] M. Janner, Y. J. B. Tenenbaum, and S. Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint Stfi::2205.09990 2022.
[31] A. Ajay, Y. Du, A. Gupta, J. Tenenbaum, T. Jaakkola, and P. Agrawal. Is conditional generative modeling all you need for meaisi?n-making? arXiv preprint arXiv:2211. 2022.
[32] P. Hansen-Estruch I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine. Idql: Implicit q-learning actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1057. 2023.
[33] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint arX1w:2303.04117, 2023.
[34] O. Mees, Hermann, E. Rosete-Beatt and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Roboties and Automation Letters, 7(3)73227-334, 2022.
[35] A. Zeng. P. Florence, J. Tompson, S. Welker, Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong. V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. CoRI., 2020.
[36] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipula- tion. In Conference on Robot Learning, pages 994-906. PMLR, 2022.
[37] L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. Citeseer, 1993.
[38] M. Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
10

---

[39] D. Ghosh, A. Gupta, A. Reddy, Fu, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals via iterated supervised learning. arXiv preprint
2019.
[40] A. Goyal, A. Friesen, A. Banino, T. Weber, N. R. Ke, A. P. Badia, A. Guez, M. Mirza, P. C. Humphreys, K. Konyushova, al. Retrieval- -uggeeene reinforcement learning. In International Conference on Machine Learning, pages 7740-7765. PMLR, 2022.
[41] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances 15097, 2021.
in neural information processing systems,
[42] C. Lynch and P. Sermanet. Grounding language in play. arXiv preprint arXiv:2005.07648, 40: 105, 2020.
[43] E. Jang, A. Irpan, M. Khansari, Kappler, F. Ebert, C. Lynch, S. Levine, and Finn. Bc-7: Zero-shot task generalization with robotic imitation learning. In Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[44] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
[45] S. Nair, E. Mitchell, K. Chen, S. Savarese, C. Finn, et al. Learning conditionnd robot behavior from offline data and crowd-sourced annotation. In Conference on Robot Learning, pages 1303-1315. PMLR, 2022.
[46] O. Mees, L.. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4):11205-113)) 2022.
[47] M. Shridhar, L.. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. Conference on Robot Learning, pages 785-799, PMLR, 2023.
[48] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07648,
2020. 2020.
[49] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1322v133 2013.
[50] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401, 1401:402.. 2014.
[51] E. Zhang, Y. Lu, W. Wang, and A. Zhang. Lad: Language augmented diffusion for reinforcement learning. arXiv preprint arXiv:2210. 15629, 2022.
[52] N. M. Shafiullah, Cui, A. A. Altanzaya, and L. Pinto. Behavior transformers: Cloning modes with one stone. Advances in neural information processing systems, 35:22955-22968, 2022.
[53] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. arXiv preprint /rXiv:/81 1.005551, 2018.
[54] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:19/2.01603. 2019.
[55] Y. Dai, M. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schuurmans, and Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111 2023.
11

---

[56] W. Liu, Y. Du, T. Hermans, S. Chernova, and C. Paxton. Structdiffusion: Language-guidd guided 304 creation of physically-valid structures using unseen objects. arXiv preprint arXiv:221 1.04604, 305:2, 2022.
[57] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang. A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, Hui, L. Sifre, van den Driessche, T. Graepel, and
D. Hassabis. Mastering the game of go without human knowledge. Nature, 2017.
[58] D. Silver, A. Huang, C. J. Maddison, Guez, L. Sifre, G. Van Den Driessche, Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
[59] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 Feb. 2015.
[60] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[61] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. Advances neural information processing systems, 14:1273-1286, 2021.
[62] Wu, A. Majumdar, K. Stone, Y. Lin, I. Mordatch, Abbeel, and A. Rajeswaran. Masked trajectory models for prediction, representation, and control, arXiv preprint htt.:2355002968. 2023.
[63] S. Ozair, Y. Li, A. Razavi, I. Antonoglou, A. Van Den Oord, and O. Vinyals. Vector quantized models for planning. In International Conference on Machine Learning, pages 3302-8313. PMLR, 2021.
[64] A. van den Oord, O. Vinyals, et al. Neural discrete representation learning, In NeurlPS, pages 6309 6318, 2017.
[65] N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv; 10084, 2019.
[66] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, Abs/1512.03388 2015. URL http://aat /ttp://axxvv.v/// ht////axxvv.///../../../..//lllt 1adss./11...000 03385.
[67] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304. 13705, 2023.
[68] Y. Ding, C. Florensa, M. Phielipp, and P. Abbeel. Goal-conditioned imitation learning. arXiv preprint 1906.09966, 2019.
[69] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint A90iv 1999. 2019.
[70] K. Song, X. Tan, Qin, J. Lu, and T.-Y. Liu. Mpnet: Masked and permuted pre- pre-training for language understanding. Advances in Neural Information Processing Systems, 13:16857-16867, 2020.
[71] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. 2018.
12

---

[72] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 1748-8763. PMLR, 2021.
[73] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint 1rXii:2223.1260/0 2022.
[74] Calvin. http://gttuubbcccoomm/s/////////
[75] Relay policy learning environments.
relayy-polieyylearnin/.
[76] Cliport.
[77] From play to policy: Conditional behavior generation from uncurated robot data. https:
//github. tot/:////a//ceepaayaooooooppp
13

---

A
Website
Video results are available at https://play- far.ag.g.uuioo...tuu...... io.
B Experimental Setup
We evaluate our method on three simulated environments. Below, we provide their details.
CALVIN [34]. The CALVIN benchmark tests a robotic agent's ability to follow language instruc- tions. CALVIN contains four manipulation environments, each of which include a desk with sliding door and drawer that can be opened and closed, as well 7-DOF Franka Emika Panda robot arm with parallel gripper. The four environments differ from each other in both their spatial composition (e.g., positions of drawers, doors, and objects) and visual features. The training data for each environment contains around 200K trajectories, from which we sample a sequence of transitions for each element of the minibatch. A portion of the dataset contains language annotations; we use this subset to train our language-conditione model. Each transition consists of the RGB image observation, proprioceptive state information, and the 7-dimensional action. The agent evaluated on its success rate in completing 34 tasks, which include variations of rotation, sliding, open/close. and lifting. These are specified by language instructions that are unseen during training in order to test the generalization ability of the agent. We evaluate two setups: (1) CALVIN A, where the model is trained and tested on the same environment (called D->D in the benchmark) and (2) CALVIN B, where the model is trained on three of the four environments and tested on the fourth (called ABC-D in the benchmark).
Franka Kitchen [22]. Franka Kitchen a simulated kitchen environment with a Franka Panda robot. It contains seven possible tasks: opening a sliding cabinet, opening hinge cabinet, sliding kettle, turning
switch, turning the bottom burner, turning on the top burner, and opening microwave door. The dataset contains 566 VR demonstrations of humans performing four of the seven tasks in sequence. Each transition consists of the RGB image observation, proprioceptive state information, and the 9-dimensional action. We split each of these demonstrations into their four tasks and annotate them with diverse natural language to create a hanguage-annotate play dataset. In our experiments, we evaluate agents on two setups within this environment, which we denote as Kitchen and Kitchen In Kitchen A, we evaluate an agent's 'ss language generalization ability at test-time by prompting it with unseen instructions asking perform one of the seven tasks. This requires the model identify the desired task and successfully execute it. Kitchen is more challenging evaluation setting, where the agent must perform two of the desired seven tasks in sequence given an unseen language instruction. In this setting, the agent must exhibit long-horizon reasoning capabilities and perform temporally consistent actions, in addition to the language generalization required in Kitchen A.
Language- Language-Conditionn Conditioned Ravens [35, 36). Ravens is tabletop manipulation environment with a Franka Panda arm. We evaluate on three tasks in the Ravens benchmark: putting blocks in bowls. stacking blocks to form pyramid, and packing blocks into boxes. The dataset consists of 1000 demonstrations collected by expert policy. Although the dataset proposed in [36] contains language instructions denoting which color block to move and the desired final location, they are not diverse like human natural language annotations would be. In order to study our model's performance on play-like language-annotan annotated dataset, we instead annotate the demonstrations with diverse natural language. At test-time, we prompt the agent with an unseen language instruction, similar to our other setups.
B.1 Real World Setup
We create multiple play environments in the real world as well. We use 7-DOF Franka Emika Panda robot arm with a parallel gripper, operating in joint action space. We have three different
14

---

environments cooking, dining table and sink. All of these tasks are multi-step. i..., in each the robot has to at least grab one object and put it in another, i.e. grab a carrot and put it inside the oven. In cooking, we test how the robot can handle articulated objects. It has to first open the oven, grill or pot, and then place an object properly inside. All of these objects have different articulations. Each of the placed objects (bread, carrot, knife, steak, spoon, etc.) have unique and different ways of being interacted with. In the sink, test very precise manipulation skills, where the robot has place objects in the narrow dish rack or hang objects (like mugs). In all of these settings, we test unseen goals (a combination of objects) that has never been seen before, well as an instruction that has never been seen before. We provide more details in the Appendix.
B.2 Additional Analysis on Discretization Bottleneck
Discretization ablation in Ravens. Table 5 studies the impact of our discrete bottlenecks on the Ravens benchmark. The success rate is, on average, worsened with the removal of either the U-Net discretization and the language embedding discretization.
Discretizing a portion of the latent. It is possible to quantize only portion of the U-Net latent representation. Table shows results of discretizing only a portion (25% or 50%) of the latent. We find that discretizing 25% of the latent resulted in better performance. Discretizing the entire latent still works well, but discretizing a portion is great way to balance encouraging skill learning and accurate denoising.
Table 5: Effect of discrete bottlenecks on Ravens tasks.
Methods
put-block-in-oowl
stack-block-pyramid
packing-box-pairs
Ours
63.6 + 2.5
20.0 + 0.0
24.0 + 1.8
No U-Net discretization
65.5 = 3.3
5.0 + 2.3
18.5 = 0.0
No lang discretization
4.1 + 0.6
3.3 + 2.7
7.5 + 2.5
Table 6: Effect of discretizing different fractions of the U-Net representation.
Methods
Success Rate
Discretize 100% of latent
45.2 + 1,2
Discretize 50% of latent
44.8 + 0.1
Discretize 25% of latent
48.7 + 0.8
B.3
Data Scaling Curves
Figure shows data scaling curves.
Effect of discrete bottlenecks. Our method scales well with more data and performs very well even at 100K trajectories, which is half the size of the CALVIN A training dataset. The removal of the language discretization results in lower success rates across almost all dataset sizes. The removal of U-Net discretization not as critical and can actually improve performance for very small datasets, but on average harmful for larger datasets.
Comparison to baselines. Our method scales well with more data while C-BeT, Play-LMP, and GCBC perform poorly for all dataset sizes.
B.4 Dataset Details
Real-world experiments. For each environment collected 250 episodes. This translates to around 15 hours of data collection. We augmented the dataset by adding 3 or 4 variations for each language
15

---

dacruntie
4
No. Same CALVIN three different envi- romments, for total of 600K training trajectories. Yes. We split each episade into the four training trajec- tories and label each of them with language. Stooes ELIE
30
0.1
0.5
1.0
2.
10
Dunset Size
Durrect Size
Figure 5: Data scaling curves. Left: effect of discrete bottlenecks. Right: comparison to baselines.
instruction (making the training dataset 750-1K - episodes). The episodes broken into smaller
annotated instructions.
Simulation experiments. We directly use the language-annotan dataset from CALVIN [34] and data generation script from CLIPort [36]. For Kitchen experiments, we used the dataset from Relay Policy Learning [22] and performed some processing and annotation to create language-annotann datasets. We provide some information in Table but note that some of the numbers estimates due to data processing procedures and refer the reader to the papers [34, 36, 22] for full details.
Table 7: Dataset details for simulation experiments.
How was play data collected?
Hours
Eps. length
No. of lang. anno- tated eps.
Is single eps. broken into smaller instructions?
CALVIN
Telcoperators in-
2.5
64
5K (instructions are repeated to ere- ale 200K training episodes)
No. Training trajectories are length- sub-episodes of length-64 episodes, structions are repeated for all sub-episndes to create total of 200K language-annen annotttte training trajectories. (How- ever, the length-64 window was sampled from long stream of play data).
structed only ex- plore. Processing into episodes and annotat- ing with language are done after- fhet-fct.
CALVIN Same CALVIN A, but three different environments.
7.5
15K (instructions are repeated to cre- ate 600K training episodes)
Kitchen
Teleoperators in- structed to perform out of possible tasks for each episode. Same Kitchen A.
1.5
200
566 (split to cre- 2.2K training episodes)
Kitchen
1.5
200
566 (split to cre- 1.6K training episodes)
Yes. We split each episode into three training trajecto- ries (one for each pair con- secutive tasks) and label each of them with language.
Ravens
Data is generated rolling an expert policy.
Up 20
1000
Depends the task. If it sequential then the instruc- tion changes throughout the episode and if it is single- step then there is one instruc- tion for the episode.
16

---

60
PlayFusion
60
PlayFusion
50
50
40
Success Rate
40
Success
30
20
20
10
10
32
512
1024
2048
4096
Action Horizon
Codebook Size
Figure 6: Effect of model design choices.
B.5 Model Design Choices
Figure studies the impact of action horizon and codebook size CALVIN A. PlayFusion is mostly robust to the action horizon Ta. We empirically found t around 20% of the overall horizon worked the best. We find that PlayFusion is relatively robust to the discrete latent codebook sizes.
Note that asymptotically, increasing the codebook size would remove the discrete bottleneck, in principle. To study whether this happens in practice, we further increased the codebook size and show CALVIN A results in Table 8. As expected, performance drops when codebook size gets very large.
Table shows the effect number of diffusion timesteps CALVIN A. We found that using timesteps works slightly better but our method is generally robust to this hyperparameter.
Table 8: Effect of further increasing the codebook size.
Codebook Size
Success Rate
2048
45.2 + 1.2
8192
46.0 1.2
16384
41.1 + 0.1
Table 9: Effect of diffusion timesteps.
Timesteps
Success Rate
50
45.2 + 1.2
100
39.9 + 1.3
25
47.4 + 0.8
B.6 Generalization to Unseen Skills
We performed an experiment where we removed one skill from the CALVIN training data. Specif- ically, we removed lift-red-block-slin
from the training data and tested the model's ability to interpolate between (1) lifting other blocks from the slider (e.g.. lift-blue- blok-block-slider, lift-pink-block- slider) and (2) lifting red blocks in other scenarios (c.g.. lift-red-bloot block-drawer, lift-red seredd-boo--tablllee We also repeated this experiment for lfft-blue-block-thbbe We find that the removal of the discrete bottlenecks results in generally worse performance in this challenging setup (see Table 10). Al- though confidence intervals do overlap bit, we find that method is average the best for both lift-red-B fift-red-block-Se block-slider and lift- slue-block-ttble.
17

---

Table 10: Performance on unseen skills.
Models
lift-red-block-slide
lift-blue-blockttblle
Ours
20.0 + 8.1
16.6 7.2
No U-Net discretization
10.0 - 4.6
3.3 2.7
No lang discretization
13.3 + 2.7
13.3 + 5.4
B.7
Ravens Experiments
Table 11 shows per-task success rates for Ravens.
Table 11: Per-task success rates for Ravens.
put-block-in-bow stack-block-pyramid
packing-bol boxiing---o--oo box-pairs
C-BeT
17.2 1.1
15.0 2.3
8.1 1.5
Play-LMP
0.0 = 0.0
0.0 + 0.0
0.8 + 0.2
GCBC
0.0 0.0
3.3 + 2.7
1.7 + 0.7
Ours
63.6 + 2.5
20.0 + 0.0
24.0 = 1.8
B.8
Implementation Details
Table 12 shows the main hyperparameters our model our simulation and real world experiments. We build off of the implementation of MCIL from CALVIN [74]. For Franka Kitchen and Ravens dataset and environment processing, implementations of the baselines, modify [77] for C-BeT and [74] for Play-LMP and GCBC.
implementations from [75] and [76], respectively. For
Where possible, we use the same hyperparameters for PlayFusion and the baselines.
Table 12: Hyperparameters of PlayFusion our simulation and real-world experiments.
Hyperparameter
CALVIN
Franka Kitchen
Ravens
Real World
Batch size
32
128
Codehook size
2048
2048
2048
2048
U-Net discretiz. wgt
0.5
0.5
0.5
0.5
Lang. discretiz. wgt
0.5
0.5
0.5
0.5
Action horizon T.
16
Context length To
2
1
1
Language features
384
384
384
384
Learning rate
le-4
2.5e-4
2.5c-4
2.5c-4
Diffusion timsteps
50
50
50
Beta scheduler
squaredcos.cap.v2
squaredcos_cam _cppoooscc_p222eeee__
squaredcos.cap..V
squaredcos.cap.v2
Timestep embed dim
256
256
128
256
18


Reviews:
[{"review_id": "c_bPkG2tmj", "reviewer": "Reviewer_aiX8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a diffusion model to generate natural-language-conditional actions. It introduces a discrete bottleneck in the diffusion model and uses discrete language conditional embedding with a VQVAE training objective. On CALVIN, Kitchen, and a real-world task, the proposed model outperforms baselines based on VAE, transformers, and behavior cloning. An ablation study on the introduced changes is included.\n", "summary_of_recommendation": "I believe this contribution is interesting to the community and can potentially inspire future work, while the significance might be limited.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "It seems the authors did not state which sampler they use for the diffusion model, so I am assuming they are using a stochastic DDPM sampler. The sampled action will incorporate randomness. Therefore without retraining the model, an error bar can perhaps be put on every result by sampling multiple times. I am curious about this randomness. I think it is especially important to have an error bar in Table 3 as the gap between different ablation seems small.\n\nAccording to the appendix, the authors used 50 steps of diffusion sampling. Have the authors attempted tuning the number of diffusion steps?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am not totally familiar with the task setting and baseline models. My review below assumes the tasks are standard and challenging to existing models and the baseline models are state-of-the-art.\n\nApplying diffusion models to this task seems like a sane choice. From my understanding, the contribution of this paper is threefold:\n- Usage of a diffusion model as an action generator\n- The discrete bottleneck in the diffusion model\n- Discrete language embedding\n\nThe authors have given a good amount of evidence to support these methods:\n- The proposed model outperforms baseline models.\n- Through ablation, discrete bottleneck and discrete language embedding both contributed to improved performance.\n\nI hope the authors can kindly answer my questions below. I also hope the reviewers could fix the typos, as there are quite a few of them (such as line 174 and 255).\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "aSuVyyIL29", "reviewer": "Reviewer_FBbz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes collecting play data, doing hindsight language annotation, and using a diffusion policy conditioned on RGB and language to perform robotics tasks. The method is evaluated on a real robot as well as three different simulation setups.", "summary_of_recommendation": "I recommend a **Weak Accept**. I think this is a strong paper overall that could be improved with some more info and/or experiments as suggested above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* It would be good to have more information on the dataset that was collected for these experiments, and the entire data collection process. Sample complexity is an important part of imitation learning algorithms so this is important context when comparing to other prior works:\n* How many hours of data were collected on the real environments? How long was each episode, and how many language annotated instructions did this result for the final dataset that was used for training? Is a single episode broken into multiple smaller language annotated instructions?\n* How was play data collected in the simulation environments? Also, same questions as above for the sim datasets, would be good to have the sizes/numbers on these datasets.\n* Data scaling curves would be interesting - how does the method compare to the baselines when training with more/less trajectories? Is there a sample efficiency gain from discrete bottlenecks and/or conditional diffusion policies?\n\n* Agree with the following in the limitations section: “our real-world experiments could be expanded to even more complex household settings such as study rooms, bed rooms, and living rooms” It would be interesting to see a larger study in more diverse scenes to push the limits of this play + language annotation + discrete diffusion policy recipe.\n* Will the datasets and code be released?\n* Minor typo on L127: “skillsm”; extra period in L289.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* The paper is very well presented and convincing.\n* Strong experimental section with proper ablations.\n* Results on multiple environments like CALVIN, Language conditioned Ravens, and Franka\n* Testing on long horizon tasks on the CALVIN evaluation suite make results more trustworthy with 128 total rollouts.\n* Very nice video results showing the PlayFusion policy running on the robot\n\nWeaknesses\n* Lacking some info on datasets and data collection process.\n* Other works have shown good results for language conditioned imitation learning. While results in this paper seem stronger than baselines, I think the result in the paper could be stronger if it can be shown that the diffusion + discretization improves sample efficiency or provides some other strong advantage. See section below requesting more information on dataset sizes/numbers.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V8AhoW1Tfiz", "reviewer": "Reviewer_wanB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes to use diffusion models to represent the different possible multimodal skills in the robotic setting. To improve the diversity of the diffusion model, the paper proposes to use a latent codebook in the denoising function of the diffusion model", "summary_of_recommendation": "For the rebuttal, please address my concerns above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Can the authors clarify Equation 7 a bit more? The symbols are a bit confusing\n\nI would like to see additional ablations on the efficacy of quantization of language and the denoising function -- it seems the performance gain of the quantization of the U-net is a bit limited and may not make sense (see my comments above). It might instead be interesting to only quantize a portion of the latent representation in the diffusion U-net or to explore instead the quantization with x0 diffusion prediction.\n\nIn principle -- increasing the codebook would eventually remove the efficacy of the discrete bottleneck (in Figure 5). As the codebook further increases, does performance drop?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I found the idea of using diffusion models to represent language anotated skills from learning to play to be natural, albeit a bit incremental (see some of the omitted related works below)\n\nThe paper demonstrates the efficacy of the approach across a wide set of different tasks and has several interesting design choices over prior work.\n\nWhile I understand the motivation of using a discrete bottleneck to enable combinatorial generation, I think this makes a lot more sense in the language encoder as opposed to the denoising function. The denoising function needs to predict the noise added to noisy trajectories, which can be a uniform Gaussian. It seems like quantizing its prediction would be actively harmful as it prevent accurate denoising as opposed to specializing skills.\n\nThe paper is missing a couple relevant other papers which also use language conditioned diffusion models to represent policies. Some examples include [1][2][3]. \n\n[1] Learning Universal Policies through Text-Conditioned Video Generation\n[2] LAD: Language Augmented Diffusion for Reinforcement Learning\n[3] StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects\n\nL127 skillsm -> skills", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "V3BTZ64BNwG", "reviewer": "Reviewer_JiUe", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work extends language conditioned learning from play, proposing that diffusion policies with a discrete bottleneck are well suited to addressing inherent multimodality in noisy human-collected data.\n", "summary_of_recommendation": "The presented work feels well executed, but with questionable research novelty. That which is claimed to be novel in the current work is weakly justified as important to success in the experiments. Even if not novel, the empirical results may provide useful signal to the community of the continued success of diffusion policies in new, less structured imitation learning scenarios.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Does the proposed discrete diffusion policy actually outperform prior continuous baselines/ablations in a statistically significant manner?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n* Well motivated: play data is typically multimodal, which motivates using richer classes of generative models to capture multiple modes in the action label distribution.\n* Clear description of methodology and setting.\n* Strong initial signs of life on recognized manipulation environments.\n* Clean presentation and writing.\n* Compelling results on real robots.\n\n## Weaknesses\n* Limited novelty. As this work points out in the related work section, Diffusion Policies (Chi et al 2023) showed the benefits of diffusion models as policies in imitation learning scenarios. This work seems to take this insight and apply it to the hindsight language annotated learning from play setting, with no modification to the LfP setting and weakly justified modification to the diffusion policy setting (see \"Weak experimental evidence\"). However, while not necessarily novel algorithmically, this may provide useful further evidence to the community that diffusion policies are a sensible model class, especially when action label distributions are multimodal.\n* Simulation results presented without confidence intervals over multiple trials. This makes it difficult to assess whether the gains over prior work are statistically significant, or if we are simply observing sampling error. Either confidence intervals should be added or claims like “PlayFusion also achieves significantly higher success rate than the baselines on Ravens” should be toned down.\n* Weak experimental evidence. Lack of confidence intervals makes interpreting important ablations difficult. L247: “The success rate (averaged over CALVIN A and B) is worsened with the removal of either the U-Net discretization and the language embedding discretization.” These ablations are what justifies the main method over continuous alternatives, but the results are presented without any confidence intervals making it hard to judge how significantly the proposed modifications are over baseline. Particularly because the success rates seem rather close (53.1 vs 51.1 vs 45.1), it is hard to say currently if the main proposed method is actually better than simpler alternatives or if we are just observing sampling error. I.e. is a 1.9% difference between discrete and continuous diffusion policy representations significant? This strikes me as a pretty major current weakness, as it seems to be justifying the only novel algorithmic components of the work.\n* Strong claims: Multiple times throughout the paper (e.g. L160), the claim is made that discrete skill representations generalize well in new settings without citation or further justification. The work would benefit from bolstering this claim somehow. What is it about discrete representations that should lead to better generalization, either theoretically or empirically? As a counterexample, BC-Z showed generalization to new combinations of skills not present in training data with continuous internal learned representations (Resnet + MLP).\nGeneralizing beyond the training distribution is presented as a major aim of the paper and discrete diffusion policies are presented as an answer. Why are there no experiments in simulation that test for generalization beyond the training distribution? This would presumably be straightforward to setup and would be far more replicable than the real world experiments.\n\n## Nit\n* Multiple references to “language-annotated play data” or “language conditioned play data” (section 4.1) without citing prior work “Language Conditioned Imitation Learning Over Unstructured Data” which focuses specifically on this setting.\nDoes Diffuser (Chi 2023) use discrete bottlenecks? If not, this seems like an algorithmic change worth mentioning as a key difference in the related section.\n* L53 “However, these diffusion model-based approaches are still limited to learning from demonstrations or reward-labeled datasets, expensive data collection methods that may introduce undesirable assumptions to real-world robotics.” Rather than saying Diffusion Policies are limited, it seems more accurate to say that Diffusion Policies have not been empirically shown yet to work on unstructured demonstration data. This seems especially true because the current work doesn’t seem to modify Diffusion Policies algorithmically, but rather applies them to a new setting.\nL59: “We believe that learning discrete skills will not only make the whole process more efficient but will also allow interpolation between skills and generalizations to new tasks.” Why would learning discrete skills make interpolation easier? Why would discrete skills generalize better? The manuscript could benefit from supporting these motivations with evidence or providing some further justification.\nL127: typo, “skillsm”", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose using diffusion models to model multimodal skills in a robotic manipulation setting, where a policy is trained on hindsight labeled play data. They propose a novel latent codebook in the diffusion denoising step in order to improve the diversity of the model. They evaluate the resultant policy on both real and simulated environments and show that their method outperforms related work.

The authors presented a thorough and detailed set of experiments during the rebuttal phase and addressed a number of reviewer concerns (e.g. missing references, additional ablations, explanations for various design choices, notation clarifications, etc). As a result reviewers did not increase their score, but the overall recommendation was still (slightly) leaning towards acceptance (and was so initially). It also should be noted that the reviewer proposing a weak reject score, did not engage in discussion after the author's rebuttal, yet the authors were quite convincing in their responses.

One potential concern not raised by reviewers (but as a suggestion from the AC) is that while the fractional improvement of their method over baselines is impressive, the final real-world success rates are never-the-less very low (e.g. 20% success on the Sink task). For future work I would consider verifying that the “improved policies” are actually qualitatively performing the tasks and not just “getting lucky” and triggering success thresholds more often. Perhaps a fractional reward metric (i.e. a dense reward) might better highlight the policy performance over baselines.
Decision: Accept (Poster)