Learning Interactive Real-World Simulators
Authors: Sherry Yang
Yilun Du
Seyed Kamyar Seyed Ghasemipour
Jonathan Tompson
Leslie Pack Kaelbling
Dale Schuurmans
Pieter Abbeel
Venue: ICLR 2024 oral (2024)
Keywords: Generative simulator
simulating real-world interactions
planning
reinforcement learning
vision language models
video generation
Primary area: applications to robotics, autonomy, planning
TL;DR: We learn an interactive real-world simulator from broad data rich in different axes that enables long-horizon interactions with humans, vision language models, and reinforcement learning agents.

Abstract: Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different axes (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, UniSim can emulate how humans and agents interact with the world by simulating the visual outcome of both high-level instructions such as “open the drawer” and low-level controls such as “move by x,y” from otherwise static scenes and objects. There are numerous use cases for such a real-world simulator. As an example, we use UniSim to train both high-level vision-language planners and low-level reinforcement learning policies, each of which exhibit zero-shot real-world transfer after training purely in a learned real-world simulator. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience in UniSim, opening up even wider applications.

Published as a conference paper at ICLR 2024
LEARNING INTERACTIVE
SIMULATORS
Sherry Yang, 1.2 Yilun Du
Kamyar Ghasemipour Jonathan Tompson?
Leslie Kaelbling Dale Schuurmans
Pieter Abbeel'
UC Berkeley Google DeepMind 'MIT *University of Alberta
sherryye (erryeebeeeeee.edee google. com]
ABSTRACT
Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative mod- els is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning real-worle simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the over- all experience, we can simulate the visual outcome of both high-level instructions such as "open the drawer" and low-level controls such as "move by Ax, Ay" from otherwise static scenes and objects. We use the simulator to train both high-level vision- langugge policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simu- lation. We also show that other types of intelligence such as video captioning mod- els can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at http::/umiiveraalssmmmldi gittt..........................
INTRODUCTION
Generative models trained on internet data can now produce highly realistic text (OpenAI, 2023).
age (Ramesh et al., 2022), and video (Ho et al., 2022a). Perhaps the ultimate goal of generative models is to be able to simulate the visual effects of a wide variety of actions, from how cars are driven on a street to how furniture and meals are prepared. With a real-world simulator, humans can "interact" with diverse scenes and objects, robots can learn from simulated experience without risking physical damage, and vast amount of "real-world" like data can be simulated to train other types of machine intelligence.
One roadblock to building this simulator lies in the datasets - different datasets cover different information that have to be brought together to simulate realistic experience. For instance, paired text-image data from the internet contains rich scenes and objects but little movement (Schuhmann et al., 2022; Zhai et al., 2022), video captioning and question answering data contain rich high- level descriptions but little low-level movement detail (Xu et al., 2016; Krishna et al., 2017), human activity data contains rich human action but little mechanical motion (Miech al., 2019; Grauman et al., 2022), and robotics data contains rich robot action but are limited in quantity (Dasari et al., 2019; Mandlekar al., 2018). Since different datasets are curated by different industrial research communities for different purposes, divergence in information is natural and hard to overcome, posing difficulties real-world simulator that seeks capture all visual aspects of the world.
In this work, we propose to combine a wealth of data in a conditional video generation framework instantiate universal simulator (UniSim)'. Under unified action-in-video-nn interface, the simulator enables rich interaction through fine-grained motion control of otherwise static scenes and objects. To support long-horizar repeated interactions, we formulate the simulator as an observation
Note that by "universal", we mean the model can simulate through the unified interface of actions and videos, as opposed to being able to simulate everything. Sound, for instance, is not heing simulated.

---

Published as a conference paper at ICLR 2024
Internet texts, images, videos
Internet
Panaramna scans
scenes
Internet
Simulated navigation
robots
Unistr
Human
Real-world
manipulation
navigation
Robot
Simulated manipulation
manipulation
Figure I: A universal simulator (UniSim). The simulator of the real-world learns from broad data with diverse information including objects, scenes, human activities, motions in navigation and manipulation, panorama scans, and simulations and renderings.
prediction model that be rolled out autoregressively support consistent simulation across video generation boundaries.
While the potential applications of the simulator broad, demonstrate three specific use cases. We first show how the simulator enables a vision-language policy to perform long-horizon goal-conditioned tasks through hindsight relabeling of simulated experience (Andrychowicz al., 2017). In addition to learning high-level vision-language policies, we illustrate how the simulator can enable learning low-level control policies by leveraging model-based reinforcement learning (RL) (Sutton, 1988). Both the high-level vasion-language policy and the low-level control policy, while trained purely in simulation, can generalize to real robot settings. This is enabled by using the simulator that is nearly visually indistinguishable from the real world, achieving one step towards bridging the sim-to-real gap in embodied learning (Rusu et al., 2017). Furthermore, we can simulate rare events where data collection is expensive or dangerous (e.g., crashes in self-driving cars). Such simulated videos can then be used to improve other machine intelligence such as rare event detec- tors, suggesting broad applications of UniSim beyond embodied learning. The main contributions can be summarized as follows:
We take the first step toward building universal simulator of real-world interaction by combin- ing diverse datasets rich in along different dimensions - e.g., objects, scenes, actions, motions, language, and motor controls in unified action-in- iition----vieooouu generative framework.
We formulate the action-in vetion-in-video-our video-out framework as an observation prediction model conditioned finite history and parametrized by a video diffusion model. We illustrate that the observation prediction model can be rolled out autoregressively to obtain consistent and long-horizon videos. We illustrate how the simulator can enable both high-level language policies, low-level control policies, and video captioning models to generalize to the real world when trained purely in sim- ulation, thereby bridging the sim-to-real gap.
2 LEARNING AN INTERACTIVE REAL-WORLD WORLD SIMULATOR
We define simulator of the real world as a model that, given some state the world (e.g., an image frame), can take in some action as input, and produce the visual consequence of the action (in the form of a vidco) as output. Learning such a simulator is hard, since different actions have different formats leng..lagguage instructions, robot controls, camera movements) and videos have different frame rates. Nevertheless, we propose specific strategies for processing each type of data to unify the action space and align videos of variable lengths to actions in Section 2.1. With unified action space, we then train an action-con conditiond video generation model to fuse information across datasets through a universal interface relating actions to videos in Section 2.2.
2.1 ORCHESTRATING DIVERSE DATASETS
Below, we highlight diverse information in different datasets and propose ways to process actions into a common format (see all datasets used to train UniSim in Appendix B).
2

---

Published as a conference paper at ICLR 2024
Dp
11
0
0
Universal Simulator
Ax Aoi, Axy, Aai
wipe table
<200m?
4g
a1
Figure 2: Training and inference of UniSim. UniSim is video diffusion model trained to predict the next (variable length) set of observation frames (ox) given observations from the past (e.g., ot-1) and ac- tion input at-1- UniSim handle temporally extended actions in various modalities such motor controls (Az1, Awi, Ar2,..) language descriptions ("wipe table"), and actions extracted from camera motions and other sources. Each dotted arrow indicates concatenating the initial noise sample for the next video segment with the previous frame.
Simulated execution and renderings. While annotating actions for real- videos is expen- sive, simulation engines such as Habitat (Savva et al., 2019) can render wide variety of actions. We use datasets previously collected from these simulators, i.e., Habitat object navigation with HM3D (Ramakrishnan et al., 2021) and Language Table Data from Lynch et al. (2023) to train UniSim. We extract text descriptions as actions when available. For simulated continuous con- trol actions, encode them via language embeddings and concatenate the text embeddings with discretized control values.
Real rohot data. An increasing amount of video data of real-robot executions paired with task de- scriptions such as the Bridge Data (Ebert et al., 2021) and data that enabled RT-1 and RT-2 (Brohan al., 2022) are becoming increasingly available. Despite low-level control actions often being different across robots, the task descriptions can serve as high-level actions in UniSim. We further include discretize continuous controls actions when available similar simulated robotics data. Human activity videos. Rich human activity data such as Ego4D (Grauman et al., 2022), EPIC- KITCHENS (Damen al., 2018), and SomeehhingSSomethine curated. Different from low-level robot controls, these activities are high-level actions that hu- mans take to interact with the world. But these actions are often provided as labels for video classification activity recognition tasks (Goyal et al., 2017). In this case, we convert the video labels into text actions. In addition, we subsample the videos to construct chunks of observations a frame rate that captures meaningful actions.
V2 (Goyal et al., 2017) have been
Panorama scans. There exists a wealth of 3D scans such as Matterport3D (Chang et al., 2017). These static scans do not contain actions. We construct actions (e.g., turn left) by truncating panorama scans and utilize the information of camera poses between two images.
Internet text-image data. Paired text-image datasets such as LAION (Schuhmann et al., 2021) contain static images of variety of objects without actions. However, the captions often contain motion information such as person walking". To use image data in UniSim, we treat individual images as single-frame
frame videos and image captions as actions.
For each of these datasets, we process text tokens into continuous representations using T5 language model embeddings (Raffel et al., 2020) concatenated with low-level actions such as robot controls. This serves as the final unified action space of our simulator.
2.2 SIMULATING HORIZON INTERACTIONS THROUGH OBSERVATION PREDICTION
With observations from different environments that have been converted videos, and actions of different formats that have been converted to continuous embeddings, we can formulate interactions with many real-world environments as interacting with universal simulator. We then formulate the universal simulator as an observation prediction model that predicts observations conditioned on actions and previous observations as shown in Figure 2. We finally show that this observation prediction model can be parametrized using video diffusion.
Simulating Real-World Interactions. We define an observation space O and an action space A which capture the videos and actions described in Section 2.1. At a specific interactive step t, an agent, having observed set of history frames h E O, decides on some temporally extended ac- tion 0 E A, which can be resolved into sequence of low-level robot commands to be executed in
3

---

Published as a conference paper at ICLR 2024
the real world. During the execution, the next set of video frames OG E O are captured from the real world. The goal of a simulator is to predict or from h4-1 and dz-1- We can formulate this prediction problem learning an observation prediction model p(o,|//--, 0.-1) While an ideal predictive model should condition on all information of the past, i.e., (Oo, 4n 04-2, ot-1), through some recurrent state, we found conditioning on finite set of frames (e.g., frames from the most recent interaction, 0r-1) greatly simplifies the modeling problem. To simulate long interactions, we can sample from the observation prediction model p(o (1-1))+1)) autoregressively conditioned on the previously sampled observations. One advantage of this observation prediction model is that the simulator stays the same across all tasks and can be used in combination with any reward function, which can be separately learned. The learned reward function can then be used to optimize policies T (ag]/re using existing decision making algorithms such as planning and RI, as we will illustrate in Section 4.1 and Section 4.2.
Parametrizing and Training the Simulator. We parametrize p(oy 14-1, az-1) using diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) as an instantiation of UniSim outlined in Figure 2. Specifically, the reverse process learns denoising model co(o, , -k) 0--1) that, conditioned on the history, generates the next observationfrom initial noise samples using K de- noising steps. In practice, we only use previous video frames and omit previous actions history, and concatenate previous video frames with initial noise samples of [² ~ NN I) channelwise to serve conditional inputs to the denoising model. To condition on an action at-1, we leverage classifier-free guidance (Ho & Salimans, 2022). The final (ou 0t-1) is parametrized by the variance schedule:
(R)
co (o (4)) 1-/h-1, a--1) = (1 + c)(o (k)
,i/h-1,, nco k(/4-1),
(1)
where 77 controls action conditioning strength. With this parametrization, we train Eo by minimizing
LUSE = ||e co (V1 3(0)), + 3B0,), k/1x-1,0-1111""
where ~ (O 1), and g(k) E R are a set of K different noise levels for each k E [], Given the learned ea, an observation or can be generated by sampling from the initial distribution
of (K) ~N I) and iteratively denoising according to the following process for & from K to 0 (k-1) = a (o( - 7 (1) co(o (k)
0i
,K ,k((4-1,04-1)) + E, 8~ (~,(00,f1),
(2)
where (2) is the denoising step size, (*) is linear decay on the current denoised sample, and o is a time varying noise level that depends on (*) and g(())
Architecture and Training. We use the video U-Net architecture (Ho et al., 2022b) implement UniSim by employing interleaved temporal and spatial attention and convolution layers in both the downsampling and upsampling passes. For history conditioning, replicate the conditioning frames at all future frame indices, and concatenate the conditioning frames with the noise sample for each of the future frame to serve as input to the U-Net. UniSim model has 5.6B parameters and requires 512 TPU-v3 and 20 days to train on all data. See more details in Appendix C.
SIMULATING REAL- WORLD INTERACTIONS
We now demonstrate emulating real-world manipulation and navigation environments by simulating both action-rich and long-horizon interactions for both humans and robots.
3.1 ACTION-RICH, HORG-HORIZON, AND DIVERSE INTERACTIONS
Action-Rich Simulation. We first demonstrate action-rich interactions through natural language actions. Figure 3 shows simulation of human manipulation and navigation starting from the same initial observation (left-most column). We can instruct a person in the initial frame to perform various kitchen tasks (top left), press different switches (top right), or navigate scenes (bottom). The model only trained on generic internet data, without action-rich manipulation data such as EPIC- KITCHENS (Damen et al., 2018), fails to simulate action-rich manipulations (Appendix F).
Long- Horizon Simulation. Next, we illustrate 8 sequential interactions in Figure 4. We condition the simulation of each interaction on previous observations and new language action as described in Section 2.2. UniSim successfully preserves objects manipulated by previous instructions (e.g., the orange and can are preserved in the drawers in Columns 4, 5, 7, 8 after being put in the drawers). See additional long-horizon interactions in Appendix A.
4

---

Published as a conference paper at ICLR 2024
Wash
Press
hands
left
Pick up
Press
bowl
middle
Cut
Press right
carrots
Dry hand
Plug in cable
Left
Left
Right
Right
Up
Down
7oom
7oom
out
in
Action-rich simulations. UniSim support manipulation actions such "cut carrots", "wash hands", and "pickup bowl" from same initial frame (top left) and other navigation actions.
Figure 3:
Pick up can Close put on top bottom drawer middle drawer
Open
Put orange
Close
Open top drawer
7. Put can top drawer
8. Close top drawer
middle drawer middle drawer
Figure 4: Long-horizon simulations. UniSim sequentially simulates 8 interactions autoregressively. The simulated interactions maintain temporal consistency across long-horizon interactions, correctly preserving objects and locations (can counter in column 2-7, orange in drawer in column 4-5).
Condition
FID +
FVD 4.
IS
CLIP
Habitat: navigate TV
frame
59.47
315.69
3.03
22.55
distant
34.89
237
3.43
22.62
Navigate TV
recent 34.63
211.3
3.52
22.63
Table 1: Ablations of history conditioning using
Figure 6: Simulations of low-data domains using the Habitat object navigation using HMMD cataset (Ra- makrishnan et al., 2021) with only 700 training exam- ples. Prefixing language actions with dataset identifier leads to video samples that complete the action (top).
FVD, FID, and Inception score, and CLIP score on Ego4D. Conditioning on multiple frames better than on a single frame, and recent history has an edge over distant history.
Diversity and Stochasticity in the Simulator. UniSim can also support highly diverse and stochas- tic environment transitions, e.g., diverse objects being revealed after removing the towel top (Figure left), diverse object colors and locations (cups and pens in Figure 5 right), and real-world variabilities such as change in camera angles. Flexibility in diffusion models promotes simulation of highly stochastic environments that cannot be controlled by actions, so that policy can learn to only control the controllable part (Yang al., 2022).
3.2
ABLATION AND ANALYSIS
Frame Conditioning Ablations. We ablate over choices of past frames condition using validation split of the Ego4D dataset (Grauman et al., 2022), which contains egocentric movement requiring proper handling of observation history. We compare UniSim conditioned different
5

---

Published as a conference paper at ICLR 2024
Uncover Pen
Toothpaste
Spider
Plate
Bottle
Pickup
Put cup 2. Pen 3. Apple
Figure 5: Diverse and stochastic simulations. On the left, we use text to specify the object being revealed by suffixing "uncovering" with the object name. On the right, we only specify "put cup" or "put pen", and cups and pens of different colors are sampled as a result of the stochastic sampling process during video generation.
numbers of past frames Table 1. Conditioning on frames is better than conditioning on a single frame, but conditioning on history that is too far in the past (4 frames with exponentially increasing distances) can hurt performance. Increasing the number of conditioning frames beyond 4 did not further improve performance on Ego4D, but it could be helpful for applications that require memory from distant past (c.g., navigation for retrieval).
Simulating Low-Data Domains. During joint training of UniSim on diverse data, we found that naïvely combining datasets of highly varying size can result in low generation quality in low-data domains. While we found that attaching a domain identifier such as the name of the dataset to the actions being conditioned on improves generation quality in low-data domains, shown in Figure 6. While such domain identifier improves in-distribution generation quality, we found domain-specific identifiers hurt generalization to other domains, and should only be applied with the test domain in distri- bution of the training domain.
increase the weight of these domains in the data mixture during training,
APPLICATIONS OF UNISIM
We now demonstrate how UniSim can be used to train other types of machine intelligence such as lasgun-language policies, RL agents, and vision-language models through simulating highly realistic experiences.
4.1 TRAINING LONG-HORIZON VISION-LANGUUAGE POLICIES THROUGH HINDSIGHT LABELING.
Language models and vision language models (VLM) have recently been used as policies that can operate in image text based observation and action spaces (Du et al., 2023b; Driess et al., 2023; Brohan et al., 2023). One major challenge in learning such agents lies in the need for large amounts of language action labels. The labor intensity in data collection only increases tasks increase in horizon and complexity. UniSim can generate large amounts of training data for VLM policies through hindsight relabeling.
Setup and Baseline. We use data from the Language Table environment (Lynch & Sermanet, 2020) table. We train an image-goal conditioned VLM policy to predict language instructions and the motor controls from the start and goal images using the PALM-E architecture (Driess et al., 2023) (See data and model details in Appendix D.I). For the baseline, the goal is set to the last frame of the original short-horizon trajectories. During each evaluation run, we set the long-horizon goal by modifying the location of 3-4 blocks, and measure the blocks' distance to their goal states after executing instructions using the VLM policy. We define the reduction in distance to goal (RDG) metric as
for learning geometric rearrangements of blocks
|050
Sgaal
||8 8poml
RDG
||50
Sgoal la
where 8y represents the underlying block locations after executing the policy, 8q and 8yaal represents the initial and goal block locations.
Generating Hindsight Data with the Simulator. To use the simulator for draw inspiration from hindsight relabeling (Rauber et al., 2019). Specifically, we create total of 10k long-horizen trajectories from the simulator by doing rollouts in the simulator 3-5 times per trajectory, where each rollout corresponds to one scripted language instruction. We then use the
tasks, we

---

Published as a conference paper at ICLR 2024
Start
Language plans generated by the VLM policy
Goal
Move the blue eube to the bottom right
Slide green circle to the top left
yellow circle to the bottom left
Simulated video plans
Real-robot execution
Figure 7: Long-horizon simulation. A VLM poliy generates high-level language actions (first row) which are executed in the simulator (middle row) similar how they are executed in the real world (bottom row) using the Language Table robot. The VLM trained on data from the simulator complete long- onng-orizzon tasks by successfully moving three blocks (blue, green, yellow) to match their target location in the goal image.
RDG (moved)
RDG (all)
Suce. rate (all) Succ. rate (pointing)
VLM-BC
0.11 + 0.13
0.07 + 0.11 0.34 + 0.13
VL.A-BC Simulator-RI.
0.58
0.12
Simulator-Hindsigh
0.34 +0.13
0.81
0.71
Table 2: Evaluation of long-horizon actions. Re- duction in distance to goal (RDG) defined in Equa- tion across 5 evaluation runs of VLM trained using simulated ong-horizon data (bottom row) compared VI.M trained original short-horizon data (top row). Using simulator performs much hetter hoth in RGD of moved blocks (left) and RGD in all blocks (right).
Table 3: Evaluation of RL policy. Percentage of successful simulated rollouts (out of 48 tasks) using the VLA policy with and without RL finetuning on Language Table (assessed qualitatively using video rollouts in the simulator). Simulator-Ri RL improves the overall performance, especially in pointing tasks which contain limited expert demonstrations.
pointing-based based
final frame from each long-horizon rollout as goal input and the scripted language instructions as supervision for training the VLM policy.
Despite the VLM policy only being trained on simulated data, it is able to produce effective high-level language actions given an initial and goal image from the real Language Table domain where the data for training the simulator was collected. The simulator can simulate video trajectories from the initial real observation, from which robot actions are recovered using an inverse dynamics model and executed on the real robot. Figure 7 shows that the language actions produced by the VLM, the generated videos from the simulator according the language actions, and the executions on the real robot. We see that the simulated video trajectory successfully translated robot actions in the real world. See additional results from the long- horizon VLM policy in Appendix A.2.
Results on Real-Robot Evaluation.
Results on Simulated Evaluation. In addition to testing the language instructions and simulated video by converting video trajectory into robot actions executed on the real robot, we also conduct simulator based evaluation to compare the reduction in distance goal (RDG) of the VLM policy using generated long-horizon data to using the original short-horizon data in Table 2. The VLM trained using long-horizon generated data performs 3-4 times better than using the original data in completing long- horizoo coul-conditioned tasks.
4.2
SIMULATOR FOR REINFORCEMENT LEARNING
Reinforcement learning (RL) has achieved superhuman performance on difficult tasks such as play- ing Go and Atari games (Silver et al.; Mnih et al., 2015), but has limited real world applications due, among other reasons, to the lack of a realistic environment simulator (Dulac-Arnol) et al., 2019). enable effective training of RL agents by providing the
We investigate whether the simulator agent with realistic simulator that can be accessed in parallel.
Setup. We finetune the PaLI 3B vision- laniunnlanguage model (Chen al., 2022b) to predict low-level control actions (joint movements in Ar, Ay) from an image observation and a task description (e.g., "move the blue cube to the right") using behavioral cloning (BC) serve as the low-level control policy and the baseline, which we call the vision-language-action (VLA) policy similar Brohan et al. (2023). Because UniSim can take low-level control actions input, we can directly conduct model-based rollouts in the simulator using control actions generated by VLA policy. To acquire reward information, we use the number of atessto-ccompletoon from the training data proxy reward to train a model that maps the current observation to learned reward. We then use the

---

Published as a conference paper at ICLR 2024
Simulated rollout from Ax, Ay moving left, right, down, up
Simulated rollout from År, Ay moving diagonally
Real-robot execution of "move blue cube green circle"
Figure 8: [Top] Simulation from low-level controls. UniSim supports low-level control actions as inputs to move endpoint horizontally, vertically, and diagonally. [Bottom] Real-robot execution of an RL policy trained in simulation and zero-shot onto the real Language Table task. The RL policy can successfully complete the task of "moving blue cube to green circle".
REINFORCE algorithm (Williams, 1992) to optimize the VLA policy, treating the rollouts from the simulator as the on-policy rollouts from the real environment and use the learned reward model to predict rewards from simulated rollouts. See details of RI. training in Appendix D.2.
Results. We first do sanity check on simulating real-robot executions by applying low-level control actions (c.g., Ar = 0.05, dy = 0.05) repeatedly for 20-30 environment steps to move the endpoint left, right, down, up, and diagonally in Figure 8 (top two rows). We see that the simulated rollouts capture both the endpoint movements and the physics of collision. To compare the RL policy trained in simulation to the BC policy, we qualitatively assessed the simulated rollouts in the simulator. Table 3 shows that RL training significantly improves the performance of the VLA policy across wide set of tasks, especially in tasks such as "point blue block". We then directly deploy the RL policy trained in the simulator onto the real robot in zero-shot, and observe successful task executions as shown in Figure (bottom row). Additional results on real robot can be found in Appendix A.3.
4.3 REALISTIC SIMULATOR FOR BROADER VISION-LANGUAGER TASKS
UniSim can generate training data for other machine-leamingg subproblems. This especially useful when natural data is rare or difficult to collect (e.g., footage of crimes or accidents). We provide such proof-of-concept by training vision-language models on purely generated data from UniSim, and observe significant performance benefits in video captioning.
Activity MSR-VTT
VATEX
SMIT
No finetune Activity Simulator
15.2
21.91
13.31
9.22
54.90
24.88
36.01
16.91
46.23
27.63
40.03
20.58
Table 4: VLM trained in UniSim to perform video captioning tasks. CIDEr scores for PaLI- X finetuned only on simulated data from UniSim compared to no finetuning and finetuning on true video data from ActivityNet Captions. Finetuning only on simulated data has large advantage over no finetuning and transfers better to other tasks than finetuning on true data.
Setup. We finetune PaLI-X (Chen et al., 2023), a VLM with 55B parameters pretrained on broad set of image, video, and language tasks, to caption set of videos generated by UniSim using texts from the training split of ActivityNet Captions (Krishna et al., 2017). We measure the CIDEr score of the finetuned model on the test split of ActivityNet Captions as well as other captioning tasks following the same setup Chen et al. (2023). See finetuning details of PaLI-X in Appendix D.3.
Results. We compare PaLI-X finetuned on purely generated videos to pretrained PaLI-X without finetuning and PaLI-X finetuned on original ActivityNet Captions in Table 4. Purely finetuning on generated data drastically improves the captioning performance from no finetuning at all on Activ- ityNet (15.2 to 46.23), while achieving 84% performance of finetuning PaLI-X finetuned on generated data transfers better to other captioning tasks such as MSR-VTT (Xu al., 2016), VATEX (Wang al., 2019), and SMIT (Monfort et al., 2021) than PaLI-X finetuned on true data, which tends to overfit to Activity Net. These results suggest that UniSim can serve as an effective data generator for improving broader vision-language models.
true data. Furthermore,
5 RELATED WORK
Internet-Scale Generative Models. Language models trained on internet text succeed at bext- tasks (OpenAI, 2023; Anil et al., 2023) but not physical tasks, which requires perception and con- trol. Internet-scale generative models can synthesize realistic images and videos (Wu et al., 2021; Ho et al., 2022a; Singer et al., 2022; Yang et al., 2023; Blattmann et al., 2023), but have mostly
8

---

Published as a conference paper at ICLR 2024
been applied to generative media (Zhang et al., 2023) opposed to empowering sophisticated agents capable of multi-turn interactions. Du et al. (2023a) shows video generation can serve as policies. but the major bottleneck for policy learning often lies in limited access to real-world en- vironments (Dulac-Arold et al., 2019). We focus on this exact bottleneck by learning universal simulators of the real world, enabling realistic and unlimited "environment" access for training so- phisticated agents interactively.
Learning World Models. Learning accurate dynamics model in reaction to control inputs has been a long-standing challenge in system identification (L.jung & Glad, 1994), model-based re- inforcement learning (Sutton, 1991), and optimal control Aström & Wittenmark (1973); Bertsekas (1995). Most systems choose to learn one dynamics model per system in the lower dimensional state space as opposed in the pixel space (Ferns et al., 2004; Achille & Soatto, 2018; Lesort al., 2018; Castro, 2020), which, despite being a simpler modeling problem, limits knowledge sharing across systems. With large transformer architectures, learning image-based based world models has become plau- sible (Hafner et al., 2020; Chen et al., 2022a; Seo et al., 2022; Micheli et al., 2022; Wu et al., 2022; Hafner et al., 2023), but mostly in games or simulated domains with visually simplistic and abundant data. In generative modeling of videos, previous works have leveraged text prompts (Yu et al., 2023; Zhou et al., 2022), driving motions (Siarohin et al., 2019; Wang et al., 2022), 3D geometries (Weng et al., 2019; Xue et al., 2018), physical simulations (Chuang et al., 2005), frequency information (Li et al., 2023), and user annotations (Hao et al., 2018) to introduce movements into videos. However, they foeus on generating domain specific videos (e.g., for self-driving) as opposed to building universal simulator that can be used to further improve other agents. The amount of control over generated videos in these existing work is also limited, as they do not treat video generation dynamics modeling problem like in our work.
LIMITATIONS AND CONCLUSION
We have shown it is possible to learn a simulator of the real world in response to various action inputs ranging from texts robot controls. UniSim can simulate visually realistic experiences for interacting with humans and training autonomous agents. We hope UniSim will instigate broad in- terest in learning and applying real-world simulators to improve machine intelligence. Our simulator has a few limitations that call for future work:
Hallucination. When an action unrealistic given the scene (e.g., "wash hands" is given to a tabletop robot), we observe hallucinations (e.g.. the table turns into a sink or the view turns away from the tabletop robot and sink shows up). Ideally, we want UniSim to detect actions that are not possible to simulate as opposed to hallucinating unrealistic outcomes.
Limited memory. The simulator conditioned on a few frames of the recent history cannot capture long-term memory (e.g., an apple in drawer could disappear when the drawer is opened if putting the apple in the drawer is not a part of the history for conditioning). How much history to condition on depends the application of the (e.g., whether the simulator will be used for policy learning in a hear-Markon setting or question answering that requires long-term memory).
Limited out-of-domain generalization. This is especially true for domains that are not repre- sented in the training data. For instance, the simulator is mostly trained on 4 robot morphologies, and its ability to generalize to an unseen robot is limited. Further scaling up training data could help, as the training data is nowhere near all the video data available on the internet.
Visual simulation only. Our simulator is not suitable for environments where actions do not cause visual observation change (e.g., different forces in grasping static cup). A true universal simulator should capture all aspects of the world beyond visual experience (e.g., sound, sensory, etc).
REFERENCES
Alessandro Achille and Stefano Soatto. A separation principle for control in the age of deep learning. Annual Review of Control, Robotics, and Autonomous Systems, 1:287-307, 2018
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision- language navigation: Interpreting visually- navigation instructions in real environments. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 3674-3683, 2018
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In

---

Published as a conference paper at ICLR 2024
Advances in Neural Information Processing Systems. 2017.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint arXiv:2305.1040) 2023.
Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 1995.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. arXiv preprint arXiv:2304.08818 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint (tw.ii::22220661111 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Vision-language-atiiss models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15811 2023.
Pablo Samuel Castro. Scalable methods for computing state similarity in deterministic markov decision processes. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 10069-10076, 2020.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arKiv: (0o::::099055555 2017.
Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn. Transdreamer: Reinforcement learning with transformer world models. arXiv preprint vrXiv:2202.09481. 2022a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209. 06794, 2022b.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo, Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, et al. Pali-x: On scaling up multilingual vision and language model. arXiv preprint arXiv:2305. 18565, 2023.
Yung-Yo Yu Chuang, Dan B Goldman, Ke Colin Zheng, Brian Curless, David Salesin, and Richard Szeliski. Animating pictures with stochastic motion textures. In ACM SIGGRAPH 2005 Papers, pp. 853-860. 2005.
Özgün Çięek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Comput- ing and Computer-Assisterr InterventionnMICCAL 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part II 19, pp. 424-432. Springer, 2016.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella. Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In Proceedings of the European conference on computer vision (ECCV), pp. 720-736, 2018
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning- arXiv preprint arXiv: 1910. 2019.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint 2rXiv:2303.03377 2023.
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schu- urmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023. URL https:/aarxii org/abs/2302.0011 2023a.
10

---

Published as a conference paper at ICLR 2024
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. arXiv preprint arXiv:2302.06692, 2023b.
Gabriel Dulac-Amolo Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprini arXiv: /2901, 2019.
Frederik Ebert, Yanlai Yang, Karl Schmeckpeper, Bernadette Bucher, Georgios Georgakis, Kostas Danillidis, Chelsea Finn, and Sergey Levine. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396 2021.
Norm Ferns, Prakash Panangaden, and Doina Precup. Metrics for finite markov decision processes. In UAI, volume pp. 162-169, 2004.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference computer vision, pp.
2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 18995-19012, 2022.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 1,00444 2023.
Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7854- 7863, 2018
Jonathan Ho and Tim Salimans. arXiv:2207. 12598, 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXi::2210002303, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.0344 2022b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp- 706-715, 2017.
Timothée Lesort, Natalia Diaz-Rodr,,uez, Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Networks, 108:379-392. 2018.
Zhengqi Li, Richard Tucker, Noah Snavely, and Aleksander Holynski. Generative image dynamics, 2023.
Lennart Ljung and Torkel Glad. Modeling of dynamic systems. Prentice-Hall, Inc., 1994.

---

Published as a conference paper at ICLR 2024
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07644 2020.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive language: Talking to robots in real time. IEEE Robotics and Automation Letters, 2023.
Ajay Mandlckar, Yuke Zhu, Animesh Garg, Jonathan Booher, Max Spero, Albert Tung, Julian Gao, John Emmons, Anchit Gupta, Emre Orbay, al. Roboturk: A crowdsoureing platform for robotic skill learning through imitation. In Conference on Robot Learning, pp. 879-888888 893. PMLR, 2018.
Vincent Micheli, Eloi Alonso, and François Flcuret. Transformers are sample efficient world models. arXiv preprint 220.:22000000880 2022.
Antoine Miech, Dimitri Zhukov, Jean-Bapteee Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 2630-2640, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 3187540):529-5333 2015.
Mathew Monfort, Sou Young Jin, Alexander Liu, David Harwath, Rogerio Feris, James Glass, and Aude Oliva. Spoken moments: Learning joint audio-visual representations from video descrip- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14871-14881 2021.
OpenAI. Gpt-4 technical report, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21((140):167,, 2020.
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg. John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Maattaatmatterport 3d dataset (HM3d): 1000 large- scale 3d environments for embodied AI. In Thirty-fifth Conference on Neural Information Pro- cessing Systems Datasets and Benchmarks Track, 2021. URL https: /tttss::/rreee /ttr/v.vvvv./////////e/////aa org/ a
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.061 06122,,,, 2022.
Paulo Rauber, Avinash Ummadisingu, Filipe Mutz, and Jürgen Schmidhuber. Hindsight policy gradients. In International Conference on Learning Representations, 2019.
Andrei A Rusu, Matej Večerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. Conference on robot learning, pp. 262-270. PMLR, 2017.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339-9347, 2019.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2225278--55944 2022.
12

---

Published as a conference paper at ICLR 2024
Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel. Reinforcement learning with action-free pre-training from videos. In International Conference on Machine Learning, pp- 19561-19579. PMLR. 2022.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animat- ing arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2377-2386, 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou. Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre. Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm. arXiv preprint arXivv1712.0083
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-videos Text-to-videe generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynanics. In International Conference on Machine Learn- ing. pp. 2256-2265. PMLR, 2015.
Richard S. Sutton. Learning predict by the methods of temporal differences. Machine Learning. 1988.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 264):160-163, 1991.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1rXiv:8801.006900 2018.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Dara Bahri, Tal Schuster, Steven Zheng, et al. U12: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations, 2022.
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6847-6857, 2021.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4581-4591, 2019.
Yaohui Wang, Di Yang, Francois Bremond, and Antitza Dantcheva. Latent image animator: Learn- ing to animate images via latent space navigation. arXiv preprint 220.:22000009443 2022.
Chung-Y: Yi Weng, Brian Curless, and Ira Kemelmacchr--hhiizermnn Photo wake-up: 3d character animation from single photo. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5908-5917, 2019.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine learning, 8:229-256, 1992.
Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.18866 2021.
Ziyi Wu, Nikita Dvomik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Unsupervised visual dynamics simulation with object-centric models. arXiv preprint arXiv:2210.05861. 2022.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
13

---

Published as a conference paper at ICLR 2024
Tianfan Xue, Jiajun Wu, Katherine L Bouman, and William T Freeman. Visual dynamics: Stochastic future generation via layered cross convolutional networks. IEEE transactions on pattern analysis and machine intelligence, 41(9):2236-222 2250, 2018.
Mengjiao Yang. Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Sepa- rating what you can control from what you cannot. arXiv preprint UrXiv:2210.134333 2022.
Mengjiao Yang, Yilun Du, Bo Dai, Dale Schuurmans, Joshua B Tenenbaum, and Pieter Abbeel. Probabilistic adaptation of text-to-video models. arXiv preprint arXiv:2306.01872, 2023.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18456-18466, 2023.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022.
Chenshuang Zhang, Chaoning Zhang, Mengehun Zhang, and In So Kweon. Text-to-image diffusion model in generative survey. arXiv preprint urXiv:2303.07900 2023.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXhv:2211.11018 2022.
Karl J. Áström and Björn Wittenmark. Adaptive control of linear time-invariant systems. Automat- ica, 96)5551-564, 1973.
14

---

Published as a conference paper at ICLR 2024
Appendix
In this Appendix we provide additional qualitative results on long-horizon simulation of human and robot interactions (Section A.I), long-horizon VLM policies (Section A.2), and low-level RL policies (Section A.3) that work on real robot. We also provided details on the dataset used to train UniSim in Section B, the model architecture and training details of UniSim in Section C, and the details of the three experimental setups for applications of UniSim in Section D. Finally, we provide failed examples when UniSim is not jointly trained on broad datasets (Section F). Video demos can be found at anonymous sappper-submisssinn.. iittuiio
A ADDITIONAL RESULTS
A.1 ADDITIONAL
INTERACTION
1. Close
2. Open top drawer
3. Put bottle in
4. Close top 1. Move sponge Move
3. Knock can over
bottom drawer
drawer
drawer
close to chips close to chips
Hotr
Figure 9: Additional results long horizon interaction with humans and robots similar Figure 4. UniSim generate consistent video rollouts 3-4 high-level language actions.
15

---

Published as a conference paper at ICLR 2024
A.2 ADDITIONAL REAL-ROBOT RESULTS FOR LONG-HORIZON LANGUAGE POLICY
Language plans generated the VLM policy
Goal
Move the blue cube to botton left
Slide the circle to the top left
Move the yellow circle to the top right
Simulated video plans
Real-robot executions
Start
Goal
Language plans generated the VLM polic
Hove cube to the bottom left.
slide green circle to the top left
Simulated video plans
Real-robot exccutions
Figure 10: Additional results (similar Figure 7) on applying UniSim to train vision-language policies to complete long-horizon tasks. VLM finetuned with hindsight labeled data is able to generate long-horizon instructions that moves two or three blocks successfully to match their location in the goal image.
16

---

Published as a conference paper at ICLR 2024
A.3 ADDITIONAL RESULTS ON LEARNING RL POLICY UNISIM
Place your hand above the blue cube
Push the red circle towards center right
Slice yellow bexagon bit right
Move the red towards the red circle
Move the red star right up
Push the blue cube closer to the red circle
Push the yellow heart above blue trianglo
Slightly move the green circle downwards
your arm left of the red star
Separate the circle from green star
Push the yellow beart bottom the green star
Move the red circle into the green star
yellow hexagon the top left the board
green the botton of the blue cube
Separate the star from the red circle
Separate green circle from green slar
Real first observation of each trajectory
Simulated last observation of each trajectory
Place your hand above the blue eube
Push the red circle towards center right
Slice yellow hexagon
Move the star towards the red cirole
bit right
Move the red right and up bit
Push the blue cube closer to the red circle
Push the yellow heart above the blue triangle
Slightly move green circle downwards
Place your arm to the left of the red
Soparate the green circle from yreen
Push the yellow the bottom of the
cincle into the green star
Move the yellow hexagon the top left of the board
Place the green circle to
Separatee the red star from red circle
Sepurate green circle from green star
Figure First real observations and last simulated observations rolling out the RL policy trained in UniSim.
17

---

Published as a conference paper at ICLR 2024
Put the red star towards the blue cube
Move the blue cube next to the green cirele
Put the yellow pentagon towards the blue cube
Move the blue cube close the green circle
Slide the red circle the yellow pentagon
Put the ycllow pentagon the blue cube
Real first observation of each trajectory
Learned reward model output
Put the red star lowards the blue cube
Move the blue cube next to the green circle
Put the yellow pentagon
towards the blue cube
Move the blue cube close to the green circle
Slide the red circle
Put the ycllow pentagon to the blue cube
next to the yellow pentagon
Real-robot last observation of each trajectory
Put the red towards the blue cube
Move blue cuhe the green circle
yellow pentagon towards the blue cube
Move the hlue cube close the green circle
Slide the red circle the yellow pentagon
yellow pentagon the blue cube
Figure 12:
First observations and observations of executing RL policy trained from UniSim the real world in zero-shot. Middle plot also shows the output of the learned reward model isteps-to-completion) during policy execution, where step corresponds to the top plot (initial observation) and step 70 corresponds the bottom plot (final observation).
18

---

Published as a conference paper at ICLR 2024
B DATASETS
We provide the datasets used train UniSim below, including dataset name, number of training examples (approximate), and weight in the data mixture. Miscellaneous data are collections of datasets that have been published. Some of these datasets have been processed into train and validation split, hence the number of training examples may differ from the original data size. When text are available in the original dataset, we use T5 language model embeddings (Raffel et al., 2020) to preprocess the text into continuous representations. When low-level controls are available in the original dataset, we encode them both as text and normalize then discretize them into 4096 bins contatenated with language embeddings (if present). The choice of mixture weights are either 0.1 or 0.05 without careful tuning. How data mixture weights affect simulation performance an interesting line of future work.
Dataset
Examples
Weight
Habitat HM3D (Ramakrishnan al., 2021)
710
0.1
Simulation
Language Table sim (Lynch & Sermanet, 2020)
160k
0.05
Bridge Data (Ebert et al., 2021)
2k
0.05
RT-1 data (Brohan et al., 2022)
70k
0.1
Real Robot
Language Table real (Lynch & Sermanct, 2020)
440k
0.05
Miscellancous robot vidcos
133k
0.05
Ego4D (Grauman al., 2022)
3.5M
Something Sooeriinng (Goyal et al., 2017)
160k
0.1
Human activities
SPIC-KITCHENS (Damen al., 2018)
25k
0.1
Miscellaneous human videos
50k
0.05
Panorama scan
Matterport Room-to toe-tt--Reee Roomooo scans (Anderson et al., 2018)
3.5M
0.1
LA1ON-400M (Schuhmann et al., 2021)
400M
0.05
Internet text-image
ALIGN (Jia et al., 2021)
400M
0.05
Internet video
Miscellaneous vidcos
13M
0.05
Table Dataset name, number of training examples, and mixture weights used for training UniSim.
19

---

Published as a conference paper at ICLR 2024
C ARCHITECTURE AND TRAINING
We the 3D U-Net architecture (Çiçek al., 2016; Ho al., 2022b) parametrize UniSim video model. We apply the spatial downsampling pass followed by the spatial upsampling pass with skip connections to the downsampling pass activations with interleaved 3D convolution and tention layers as in the standard 3D U-Net. The video models in UniSim consist of one history conditioned video prediction model as the base and two additional spatial super-resolution models similar to Ho et al. (2022a). The history conditioned base model operates at temporal and spa- tial resolution [16, 24, 40). and the two spatial super-resolution models operate at spatial resolution [24, 40] + [48, 80] and 48, 80] + [192, 320], respectively. To condition the base video model on history, we take frames from the previous video segment and concatenate them channelwise to the noise samples inputted to the U-Net. We employ temporal attention for the forward model to allow maximum modeling flexibility but temporal convolution to the super-resolution models for efficiency reasons similar to Ho et al. (2022a). The model and training hyperparamters of UniSim are summarized in Table 6.
Hyperparameter
Value
Base channels
1024
Optimizer
Adam (8) 0.9, 82 0.99)
Channel multipliers
1,
Learning rate
0.0001
Blocks per resolution
3
Batch size
256
Attention resolutions
6, 12, 24
Num attention heads
16, 16, 8
Conditioning embedding dimension
4096
Conditioning embedding MLP layers: 4
Conditioning token length
64
EMA
0.9999
Dropout
0.1
Training hardware
512 TPU-v3 chips
Training steps
1000000
Diffusion noise schedule
cosine
Noise schedule log SNR range
[-20, 20]
Sampling timesteps
256
Sampling variancee interpolation
0.1
Weight decay
0.0
Prediction target
Table 6: Hyperparameters for training UniSim diffusion model.
20

---

Published as a conference paper at ICLR 2024
D DETAILS OF EXPERIMENTAL SETUPS
D.1 DETAILS OF LEARNING LONG-HORIZON POLICY
Language Table Dataset and environment. The Language Table (Lynch & Sermanet, 2020) dataset consists of 160k simulated trajectories and 440k real trajectories where each trajectory con- tains a language instruction (e.g.. "move blue cube to the right"), sequence of visuomotor controls, and a sequence of image frames corresponding to the execution of the task. The original trajectories have short horizons (e.g., only moving one block).
PALM-E VLM Policy. We modify the original PALM-E 12B model (Driess et al., 2023) to condi- tion on a goal image as additional input before decoding the text actions. The VLM is finetuned on either the original short horizon data or the long horizon simulated data using 64 TPUv3 chips for day. The supervision for thori-horizon baseline is the single step language instruction in the original data, whereas the supervision for long-horizon UniSim data is the scripted long-horizon language instructions chained together that generated the video data. Other model architecture and training details follow Driess et al. (2023).
Simulated evaluation. In setting up goal in the simulated environments, subset of 3-4 blocks (randomly selected) are moved by 0.05, 0.1, or 0.2 along the x.y axes (randomly selected). The original observation space has E [0.15, 0.6] and y E [-0.3048, 0.3048). So the modification of goal location corresponds to meaningful block movements. For executing the long-horizon VLM policy trained on UniSim data, we first sample one language instruction from the VLM, predict video of 16 frames, and use a separately trained inverse dynamics model similar to Du et al. (2023a) to recover the low-level control actions, which we found slightly outperform directly regressing on control actions from language outputs of the VLM. We execute 5 instructions in total, and measure the final distance to goal according to the ground truth simulator state. We evaluations each with different random seed for sampling the initial state and resetting the goal, and report the mean and standard error in Table 2.
D.2 DETAILS OF RL POLICY TRAINING
Stage 1 (Supervised Learning) Model Architecture The PaLI 3B model trained on Language- Table uses a Vision Transformer architecture G/14 (Zhai et al., 2022) to process images, and the encoder-decoder architecture of UL.2 language model (Tay et al., 2022) for encoding task descrip- tions and decoding tokens which can represent language, control actions, or other values of interest (described below). Objectives In the first stage of training, using a dataset of demonstrations, we finetune the pretrained PaLI 3B vision language model checkpoint (Chen et al., 2022b) with the following tasks:
Given observations and task instruction, predict the demonstration action. The continuous actions of the Language-Table domain are discretized into the form "+1 -5", and represented using extra tokens from the PaLI model's token vocabu- lary. As an example, "+1 -5" is represented by the token sequence (<extra.id.65>, cextra.id.653
Behavioral Cloning:
<extra.id.1>
<extra.id.5>)
Timestep to Success Prediction: Given observations and task instruction, predict how many timesteps are left until the end of episode (i.e. success). Similar to actions, the number of steps remaining is represented via extra tokens from the PaLI model's token vocabulary.
Given the first and last frame of an episode, predict the task in- struction associated with that episode.
Instruction Prediction:
We use learning rate 0.001, dropout rate 0.1, and batch size 128 to finetune the PalI 3B model for 300k gradient steps with Ik warmup steps both the simulated and real Language Table dataset similar to RT-2 Brohan et al. (2023).
Stage 2 (RL Training) Reward Definition As mentioned above. during Stage 1, given an ob- servation and goal, the PaI.I model finetuned to predict how many timesteps are left until the demonstration episode reaches a success state. Let us denote this function by d(o, g). The reward we use during RL training is defined as r(or, ae, 0s41, = (d(00+1.g)) - d(o:g)) C, where C > 0 is small constant used to stabilize training (C = 5e 2 in this work). Intuitively, this reward tracks from timestep t to t ++ 1 the policy arrived closer to accomplishing the desired goal. Before starting Stage 2, we make copy of the Stage I model checkpoint and keep it frozen to use as the reward model for RL training. Environment Definition To implement video gener- ation as environment transitions, we expose the inference interface of the video generation model
21

---

Published as a conference paper at ICLR 2024
through remote procedure call, and use the DeepMind RL Environment API (also known DM Env API) (Tassa et al., 2018) to wrap the remote procedure call in the step function of the en- vironment. When the environment is reset to start new episode, goal instruction is randomly sampled from the ones available in the dataset of demonstrations used in Stage 1. RI Method We initialize the RI. trained policy using the Stage 1 checkpoint, which mentioned trained with a Behavioral Cloning objective. A collection of actor processes perform policy roll- outs in the video generation environment, and add rewards the trajectories using the reward model defined above. The policy is updated using the REINFORCE (Williams, 1992) objective, L... S(r,, oz at.g) g) = , log ( or, g) E/= r(o,, a4 0++1, 9) where L(o,, (r,g) repre- sents the loss associated with the observation pair (ot, at) an episode with the goal g. The actors are rate limited to prevent generated trajectories from being very off- off-policy. We report the hyperparameters associated with RL training in Table 7.
also
Hyperparameter
Value
Max steps per episode
100
Number of actor processes
64
Number of image history stack
2
Learner batch size
64
Discounting factor ?
0.9
Table 7: Hyperparameters for training the VLA RL policy using the ACME framework.
D.3 DETAILS OF VIDEO CAPTIONING
Note that even though UniSim video based simulator trained to condition on past history, we achieve text-only conditioning by inputting placeholder frames such white images while increasing the classifier-free guidance strength on text. We found this to work well in generating videos purely from captions of ActivityNet Captions. For generating data to train VLMs, we take the training split of ActivityNet Captions which consists of 30,740 text-video examples after the 50/25/25% train/vall/val2 split as in Chen et al. (2023). For each of the 30,740 text, we generate 4 videos from UniSim, and use the text labels as supervision in finetuning PaLI-X. As a result, we have 4X amount of the original training data (in terms the number of videos). In addition, we found the generated videos to generally align better semantically than the original ActivityNet Captions videos, which could contain noise and ambiguous videos that could be labeled differently. We use ground truth temporal proposals at evaluation following Chen et al. (2023) and Krishna et al. (2017). Following Chen et al. (2023) and Wang et al. (2021), we use the vall split for validation and val2 split for testing.
E ADDITIONAL ABLATIONS
E. ABLATIONS OF DATASETS
We conduct ablations on dataset used UniSim by computing the FVD and CLIP scores over 1024 samples from the test split. We observe that including internet data and various activity and robot data performs the best. Removing the internet data led to significantly worse FVD, highlighting the importance of using internet data in UniSim.
Dataset
FVD 4
CLIP f
Internet only
219.62
22.27
Without internet
307.80
21.99
Universal simulator
211.30
22.63
Table 8: Ablations datasets using FVD and CLIP score on the held-out test split. Including internet data and diverse human activity and robot data in UniSim achieves the best FVD and CLIP scores.
E.2 ABLATIONS OF MODEL SIZE
We conduct ablations on model size by computing the FVD and CLIP scores over 1024 samples from the test split. We found that while increasing the model size improves the video modeling performance, the amount of improvement measured by FVD plateaus as the model gets bigger, which is slightly disappointing from scaling point of view.
22

---

Published as a conference paper at ICLR 2024
Model size
FVD 4
CLIP t
500M
277.85
22.08
1.6B
224.61
22.27
5.6B
211.30
22.63
Table 9: Ablations of model size using FVD and CLIP score on the held-out test split. The largest model achieves the best FVD and CLIP scores.
23

---

Published as a conference paper at ICLR 2024
F FAILED SIMULATIONS WITHOUT JOINT TRAINING
Figure 13: Failed environment simulation from the action "uncover bottle" without training broad data as in UniSim. Top vidcos generated from only training SSV2. Bottom two videos are generated from only training generic internet data (without SSV2, EpicKitchen, Ego4D, and various robotics dataset).
24

---

Published as a conference paper at ICLR 2024
Pick up can and put on lop
Close bottom drawer
Open middle drawer
orange middle drawer
Close middle drawer
Open top drawer
Put in drawer
Close top drawer
Figure 14: When the text-to-video model behind UniSim is only trained on data from Brohan et al. (2022) as opposed incorporating broad data from the intermet and other manipulation datascts, long-horizon interaction simulations fail half of the time (red text).
25


Reviews:
[{"review_id": "ihrdy34PtF", "reviewer": "Reviewer_ZLpe", "summary": "In this work, the authors propose to learn a universal simulator (UniSim) of real-world interaction through generative modeling (a diffusion model for outputting the next frame given the previous frame and the input actions). They achieve so by careful orchestration of diverse datasets, which are rich along completely different axes (e.g., some videos have object-level diversity, some have densely labeled language instructions, and some have scene-level diversity). They show applications of the proposed simulator such as training long-horizon embodied planners and low-level object manipulators.", "questions": "See “weaknesses”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Reasonably scalable approach to collect training data for the proposed simulator \n+ The use of diffusion models to fuse different aspects of the diverse datasets with decent results is impressive\n+ Particularly the sim-to-real transfer is a promising direction for using the proposed real-world simulator.", "weaknesses": "While this work shows great promise in a range of downstream applications. I believe it might need more experimental evidence to support the claim that it can simulate low-level actions well. Specifically, section 4.2 only shows results for a relatively simple object (mostly blocks) re-arrangement (without grasping, e.g.) on a table. What about grasping objects, pulling objects (e.g., opening a drawer), etc? It will give us insights as to how fine-grained the controls are supported by the proposed simulator, even if it cannot simulate low-level actions perfectly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2qaG3iXNkU", "reviewer": "Reviewer_kkjZ", "summary": "- The paper presents a video diffusion model that does conditional next (few) frame prediction. It conditions on previous frames and either a text description of the video, or more granular actions (robotic movements, or camera angles). The focus is on its use in robotics contexts.\n- The novelty is in the mix of data trained on. Rather than focusing on a single environment or even single action space, the model (UniSim) is trained jointly on 14 common datasets, from the text-image LAION dataset (often used for image generation), to the Something-somethingV2 video dataset (often used for video classification). Significant compute is used (512x TPUs)\n- A limited ablation is conducted on how previous observations should be conditioned upon.\n- Three use cases are explored:\n1) A separate vision-language model is first trained to predict language instruction and actions, given a start and end observation, on a robotics control task in the Language Table environment. It is then finetuned using simulated trajectories from data synthetically generated by UniSim (longer to those in the original dataset).\n2) A separate policy model is first trained via BC on the Language Table environment, then finetuned with RL using simulated trajectories from data synthetically generated by UniSim (itself trained on Language Table data).\n3) A separate video-to-caption model is found to benefit when finetuned on data synthetically generated by UniSim, for producing captions on ActivityNet.\n\n___\nFollowing the rebuttal, I upgrade my ratings as follows: soundness 2$\\to$3, overall rating 5$\\to$8, and confidence 4$\\to$5. The main remaining weakness is that the mixture of datasets and modalities (a key contribution of the work) appears to be of limited benefit on the tasks assessed by the paper. But there are enough positives in the paper for me to downweight this issue.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper will undoubtedly draw a lot of attention and excitement from researchers working in several areas, including RL, robotics, large models, and diffusion models.\n- It represents a major effort in training a cross-domain model, with emphasis on its use for robotic control.\n- I welcome this kind of larger scale-up work being submitted to an academic conference, since a recent trend has seen similar works restricted to industry-lab preprints.\n- The first few pages motivating the work are quite inspiring.\n- Effort has been made to explore a range of use cases.\n- Overall it represents a very promising direction towards foundational models for control.", "weaknesses": "I expect that this paper will comfortably clear the bar for acceptance. However, there are two main issues I believe should first be addressed. I've set my score relatively low because of these, but anticipate increasing it following a revised version.\n\n1) Whilst it's difficult to accuse the paper of overclaiming in any specific place, the writing and framing risk feeling a little showy. The title is very general, and applies to any paper on model-based RL for the real-world rather than something specific to this paper, and naming the method a \"universal simulator\" feels grandiose. (Happy to collect other revewiers' opinions on this.) The connection between POMDP's and action-conditioned video generation is more-or-less assumed by any world model paper (e.g. [1]), and shouldn't be highlighted as a main contribution of the paper.\n2) One of the recurring claims throughout the paper is that the major novelty is UniSim's \"orchestration of diverse datasets, each providing a different aspect of the overall experience\" into a single model. Yet no hard evidence is given for this combination being important -- aside from two vague figures in Appendix E. At a minimum, it would be important to train a version of UniSim on say, datasets from the Language Table environment _only_, and report numbers for when synthetic data was generated from this, in Table 2 and 3. This would help support the claim that dataset diversity is valuable.\n\nOther issues (in decreasing priority)\n- I think it'd be useful to investigate how entwined the effect of actions is with the dataset distribution. For example, could camera commands (zoom in etc) successfully be applied to kitchen scenes as in Figure 3? The fact that the name of the dataset had to be included as part of the action during training, makes me suspect actions may not be able to generalise well to new kinds of video. This would not be a dealbreaker for the paper's acceptance, but is important to show readers how general this data mixing is.\n- A lack of strong baselines might be expected for this kind of scale-up work. But in their absence, ablations become more important, to verify that the various components of the model were all necessary. The paper only presents a brief study of which frames to condition on.\n- The model section is poorly written. The use of $\\mathcal{T}$ is (I think) slightly misleading -- usually the transition fn of a POMDP is defined as operating on the states, $\\mathcal{T}(s_t,a_t) \\to s_{t+1}$, and there is a separate emission function producing the observations, $p(o_t|s_t)$. Eq. 1 implicitly combines these -- I might recommend renaming it $f$ or $g$ or whatever. I didn't follow why $o_l$ notation needed to be introduced, since it's immediately unrolled into $[o_t, o_{t+1}]$ and never referred to again. I also didn't understand why the model conditions on the noised, rather than clean, previous observations. It's said the last four frames are concatenated from $o_{t-1}$, which confused me -- does $o_{t-1}$ represent four frames, or should it read $o_{t-1:t-4}$ or similar?\n- It's a shame to give the model details only in the Appendix C, as I believe many readers would be interested in them. I hope some of these can be shifted to the main body, particularly key details around the diffusion architecture (such as the core and super-resolution modules) and the amount of compute required.\n- Any algorithmic or model novelty is light (more or less straightforward video diffusion).\n- The two main experiments were conducted on environments that were within the training distribution of UniSim. It would have been more impressive to investigate the performance on new environments.\n- The wordy description of all datasets in 2.1, I felt was much better summarized by Table 5 in the Appendix (perhaps with the addition of a column explaining how an action space is defined and handled), and might be swapped. (Optional!)\n\nMinor issues/questions\n- Appendix says 1M steps on 512 TPUs with batchsize 256 -- this seemed a low ratio of training updates to available compute. Did performance saturate beyond this?\n- What was the wall clock time of the model training?\n- How many parameters were in the model?\n- Will the model weights be open-sourced?\n\n[1] Transformers are Sample-Efficient World Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DBZbFbd302", "reviewer": "Reviewer_b2Jg", "summary": "This paper presents UniSim, a video prediction and generative model aiming for serving as a universal simulator of diverse scenarios conditioned on input language-described actions. It devotes a big effort in combining dataset with different modalities and information axes, trained a unified generative model, and shows the trained model can be used for downstream policy learning.", "questions": "See above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Very cool and impressive research direction and proposed method\n- Huge effort devoted in unifying multiple large scale datasets\n- Experiments demonstrated effectiveness for downstream policy learning", "weaknesses": "I think the paper presents a very important step towards learning a universal video predictive world model. One of my questions is, the shown demo looks like generally still in distribution, in terms of generalization across different embodiment: the generated video contaiing robot are very similar to robotic dataset, and in more complex scenes training using human videos the model seems only handling human hands. How does it work in those complex scenes when the model is commanded to predict outcomes given a robot action input?\nAlso, when it comes to low level control input, the paper seems only handling delta motion in the cartesion space. Does it handle more general end-effector action in SE3 space? (joint space seems out of reach for this family of method since it's not observable) Is it true that for predicting outcomes conditioned on robot action, the robot arm needs to be visible in the first place?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "neyFV0WvE8", "reviewer": "Reviewer_ihUZ", "summary": "This paper introduces a universal simulator (UniSim) that aims to simulate how humans and agents interact with the world. The proposed framework combines various types of datasets, including internet text-image pairs and robotics data, with the motivation that existing datasets are useful along different axes. The paper uses a video diffusion model as an interactive simulator of the world. UniSim can simulate both high-level instructions and low-level control, which show zero-shot transferability to real-world scenarios, addressing the sim-to-real transferability problem. The authors highlight the potential for UniSim to be used in broader applications, such as video captioning and rare event detection.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- This is an interesting paper that presents some exciting results.\n- The paper is well organized and well-written.", "weaknesses": "- It would be nice if the paper delved more into the limitations of the models. The paper has shown that exciting results can be obtained, but it's useful for the community to know the limits of the generalization capabilities, especially if people want to use this in the future for various applications. \n- For reproducibility, it would be helpful if the authors could release the code and some example pre-trained checkpoints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)