Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
Authors: Hongtao Wu
Ya Jing
Chilam Cheang
Guangzeng Chen
Jiafeng Xu
Xinghang Li
Minghuan Liu
Hang Li
Tao Kong
Venue: ICLR 2024 poster (2024)
Keywords: Visual Robot Manipulation
Video Generative Pre-Training
Causal Transformer
Primary area: applications to robotics, autonomy, planning

Abstract: Generative pre-trained models have demonstrated remarkable effectiveness in language and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We introduce GR-1, a GPT-style model designed for multi-task language-conditioned visual robot manipulation. GR-1 takes as inputs a language instruction, a sequence of observation images, and a sequence of robot states. It predicts robot actions as well as future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre-trained on a large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and a real robot. On CALVIN benchmark, our method outperforms state-of-the-art baseline methods and improves the success rate from 88.9% to 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. In real robot experiments, GR-1 also outperforms baseline methods and shows strong potentials in generalization to unseen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https://GR1-Manipulation.github.io

Published as a conference paper at ICLR 2024
UNLEASHING LARGE-SCALE VIDEO GENERATIVE PRE-TRAINING FOR VISUAL ROBOT MANIPULATION
Hongtao Wul, Ya Jing', Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, Tao Kong* ByteDance Research {wuhongtao. 123, kongt.sojebyyeecccee.com
ABSTRACT
Generative pre-trained models have demonstrated remarkable effectiveness in lan- guage and vision domains by learning useful representations. In this paper, we extend the scope of this effectiveness by showing that visual robot manipulation can significantly benefit from large-scale video generative pre-training. We intro- duce GR-1, a GPT--ttlle model designed for multi-task language-cone conditiond vi- sual robot manipulation. GR-1 takes inputs language instruction, sequence of observation images, and a sequence of robot states. It predicts robot actions well future images in an end-to-end manner. Thanks to a flexible design, GR-1 can be seamlessly finetuned on robot data after pre- pre-trained on large-scale video dataset. We perform extensive experiments on the challenging CALVIN benchmark and real robot. On CALVIN benchmark, our method outperforms state-of-the-ar baseline methods and improves the success rate from 88.9% 94.9%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. real robot experiments, GR-1 also out- performs baseline methods and shows strong potentials in generalization to un- seen scenes and objects. We provide inaugural evidence that a unified GPT-style transformer, augmented with large-scale video generative pre-training, exhibits remarkable generalization to multi-task visual robot manipulation. Project page: https: /RR Man pu 1 at on. www.oo..iittoooo github.is ic
INTRODUCTION
Recently, generative pre-trained models have demonstrated impressive performance in both natural language processing (NLP) and computer vision (CV) (Brown al., 2020 Chen et al. 2020 He et al., [022 Touvron et al., 2023). They model sequences of inputs in generative manner and pre-train on large-scale datasets before finetuning on specific tasks. Large-scale pre-training allows these models to learn general patterns from large datasets and thus enables them to easily general- ize to related finetuning tasks with inherited generalizability and robustness. Robotics data is also generative in sense that the observation is only revealed after the action is taken. However, like NLP and CV, robot data is scarce, as its collection often requires costly and cime-consuming human demonstrations. Moreover, robot data is multi-modal, including images, robot states, tions, and language instructions. To address these challenges, prior research has delved into diverse pre-training methods, aiming enhance the learning capabilities of robots (Nair et al. 2022 Ra- dosavovic et al. 2022 2023 Seo et al. 2023 Parisi et al. 2022 Shah et al. 2023 Lin et al. 2023
Kumar al. 2022 Liu al. 2022 Jen-Chen
2020
In this paper, we adapt similar generative pre-training paradigm for tackling the problem of multi-
task language-con conditionnd visual robot manipulation. We argue that video generative pre- pre-training a closely related task to robot action learning for the reason that robot trajectory itself contains video sequence. The ability forecast future images based on past images and language instructions empowers the robot to anticipate forthcoming events, thereby facilitating the generation of relevant
1 Equal contribution. *Corresponding authors.

---

Published as a conference paper at ICLR 2024
CALVIN Benchmark Dataset
Inputs
Leamaale Tokens
Outputs
Tremee
M/ loten
book drowwer
Image
335 lotan
imoge
"panrigiite slik
54 shoer
Robot Stows
Deploy
Pre-train
Fine-fune
>>>>>>
Copy Welghts
GR-1 I
GR-1
>>>>>>
Wil Wlls
Wlls
WM
Deploy
Large Scale Video Datase!
Real Rabot Dataset
the sporge"
caly the
Isto
Figure I: Overview of GR-1. GR-1 is first pre-trained on the task of video prediction with a large- scale video dataset. It is then finetuned robot data learn multi-task visual robot manipulation.
and suitable actions. To this end, we propose to leverage large-scale video generative pre-training for efficient and effective learning of multi-task visual robot manipulation. We introduce GR- (Fig. [ GPT-style model which takes input a language instruction, sequence of observation images, and sequence of robot states and predicts robot actions and future images an end-to-end manner. GR-1 is first pre-trate rre-triiedd on video prediction using lurge-scale video dataset (Grauman al. 2022 and subsequently undergoes seamless fine-tuning with robot data.
We perform extensive experiments the challenging CALVIN benchmark, which contains 34 dif- ferent manipulation tasks with language instructions. Results show that our method outperforms state-of-the the-att baseline methods and improves 1) the success rate from 88.9% to 94.9% and 2) the average length (average number of completed tasks in a row of 5) from 3.06 to 4.21. More impor- of 77.8% while that
tantly, when trained on 10% data of the full dataset, GR-1 achieves success of the best baseline method is 66.8%. In the setting of zero-shot unseen scene generalization, GR-1 improves the success rate from 53.3% to 85.4%. When evaluated on unseen language instructions, GR-1 also outperforms all the baseline methods. We perform real robot experiments of end-to-end object transportation and articulated object manipulation verify the performance of GR-1 1 in the real world. GR-1 outperforms the comparing state-of-ther -fhte--rtt baseline methods and shows promis- ing potentials in out-of-distribution settings, including generalization to unseen scenes and unseen objects. To the best of our knowledge, GR-1 for the first time shows that unified GPT-style trans- former, augmented with large-scale video generative pre-training, is able to effectively generalize to multi-task visual robot manipulation. Key contributions of the paper include:
We show that large-scale video generative pre-training is able to effectively benefit visual robot manipulation learning.
We present a flexible GPT-style transformer model, GR-1, 1, which allows large-scale video generative pre-training and robot data finetuning with a unified model. Therefore, the model trained on large-scale video datasets can be directly used for robot policy learning.
We conduct extensive experiments in both simulation and the real world to study the per- formance of GR-1 in various settings.
RELATED WORK
2.1
LANGUAGE-CONDITIONN VISUAL ROBOT MANIPULATION
Language-condinng conditioned visual robot manipulation is a flexible and intuitive way for non-expert hu- mans to instruct the robot to perform different tasks (Brohan et al. 2022 2023 Shridhar al. 2023
2

---

Published as a conference paper at ICLR 2024
2022
Ahm al. 2022 Driess al.] 2023; Du et al.) 2023 Guhur et al.; 2023; Jang et al. 2022
Lynch & Sermanet [020; Lynch et al., 2022 Mees et al. 2022a cj Ha et al. 2023). J. Some existing methods (Driess et al. 2023 Ahn et al. [202 Huang al. 2022) exploit large language models (L.LMs) to plan over task domain and pass instructions to low-level action policies to generate robot actions. Ha et al. (2023) uses LLMSs scale up data generation and leverages the diffusion pol- icy (Chi et al. (2023) to learn a langunge-conditioned policy. Recently, RT-2 (Brohan et al., (003)) proposes to co-fine-tune robot trajectory data and vision-language data together and achieves great generalization capabilities on novel objects and instructions. CLIPort (Shafiullah et al. 2022 and
PerAct (Shridhar al.) 2023) learn language-conditen conditioned manipulation by leveraging CLIP (Rad-
ford et al.) 2021) to connect visual signals and language instructions. Hiveformer (Guhur et al.) 2023) proposes to jointly model languages, multiple views, and history with a unified transformer model. These methods predict sparse end-effector keypoint poses and rely on motion planners to plan trajectories, making them less flexible compared to predicting continuous actions. Another line of work learns Lunguage-conditione policies from unstructured play data which contains demon- strations with and without language labels (Lynch & Sermanet 2020) Mees et al.) a.) These weweenncet--eqqueeee
conditional variational auto enttoer to generate latent plans which are used to condition the action policy. In contrast, our method uses only demonstrations with language labels and leverages a simple GPT-style transformer to model the trajectory.
methods leverage
2.2 SEQUENTIAL. DECISION MAKING WITH TR TRANSFORMERRS
There is a growing interest in using transformers Vaswani et al. 2017) to model sequential decision- making problems. Decision Transformer (DT) (Chen al.) 2021) takes inputs a sequence of return-to-ge to-go, observations, and actions in the past, and outputs actions auto-regressively by leverag- ing causal transformer. VIMA (Jiang et al.] 2022) enables multi-modal prompting interface for generalist robot manipulation with an encoder-decoder transformer architecture. GATO (Reed et al.) 2022) uses decoder-only transformer learn a multi-modal, multi-task, and tintimemb generalist policy. RoboCat (Bousmalis et al.] (203) proposes to use heterogencous robot data to develop a self-improving agent for robot manipulation. predicts both actions and future images. However, RoboCat at does not perform video pre-training and it is goal condicoond instead of language-conditionel
2.3
PRE-TRAINING FOR ROBOT LEARNING
Recently, pre-training for robot learning has been an active research topic (Radosavovic et al.
2022
2023
Nair al. 2022 Xiao et al. 2022 Liu et al. 2022 Seo et al. 2023
Lin al. 2023
Kumar
et al.
2022:
Shah et al. 2023 Jing et al. 2023 Parisi et al. 2022: Escontrela et al.) 2023
Laskin
let al.
2020
Brohan et al. [003]] Liu et al.] 2022 Thomas al. [2023] Schwarzer et al. [202]]
Sun
et al.
2023
Yen-Chen et al. 2020 Mendonca et al. 2023 Ma et al.
2022 Bhateja et al. 2023
Karamcheti al. 2023; Yang al.] 2023). Some methods aim to learn useful visual representations
by masked image modeling (Xiao et al.][2222 | Radosavovic et al. [022] Seo et al. 2023 Karamcheti
cct al. (223) and contrastive learning (Nair et al. 2022 Sermanet et al. 2018; Jing et al., [023]]
Laskin et al. 2020). Another line of work aims first learn a world model and then train RL agent with the learned model (Seo et al., 2023 Hafner et al., 2020). SpawnNet (Lin et al. 2023) introduces two-stream architecture that fuses pre-trained networks with separate network for learning generalizable policies. VPT (Baker et al.] 2022) trains an inverse dynamics model from small amount of labeled videos and uses it label large amount of unlabeled videos for action learning in Minecraft. VIPER (Escontrela et al.) 2023) leverages pre-trained video prediction models as reward signals for reinforcement learning. The video data in VPT and VIPER are both in-domain data from task environments. In contrast, our method is pre-trained on large-scale non-robotic data which out-on-domain. Model-based approaches propose to learn video prediction model and leverage model predictive control (Finn & Levine 2017, Gupta et al. 2022) or learn inverse dynamics model (Du et al. 2023) to infer actions. Our method different from these approaches as it uses a unified model for both video prediction and action prediction. Recently, RPT (Radosavovic et al. 2023) proposes a self-supervised sensorimotor pre-training approach to learn a model of the physical world by predicting masked-out tokens of different modalities. Our method is different in that it adapts video prediction tasks (Yan et al. 202T 2023) for large-scale pre- pre-training and focuses on aanguag-coonditione multi-tasking.
3

---

Published as a conference paper at ICLR 2024
3 METHOD
3.1 PROBLEM FORMULATION
Video Generative Pre-Training We leverage language-condttinned video prediction as the video generative pre-training task. Specifically, we pre-train model Tp to predict the video frame at timestep t At given the language description I the video and a sequence of video frames hz-h: from timestep t h to t:
Tu (1, ha) Or+
(1)
We assume access to dataset containing pairs of video and language description of the video:
(1, 01, 02, 07}
Multi-Task Visual Robot Manipulation Finetuning. We formulate multi-task language- conditioned visual robot manipulation as learning a model an that maps a language instruction / and sequence of observation images Ou- and states -t- from timestep & - h to the current timestep to an action a,. addition, we add future image prediction similar to the video prediction in the pre-training phase:
(1, O St-h:t) + Ort
(2)
The language instruction I describes the task that the robot is instructed to accomplish, egg. "slide left the red block". The observation sequence Oz-Aut contains visual observation images from the environment. The state sequence St-ixt denotes the robot states, i.e. the end-effector pose and the binary gripper status. different tasks D {T)) Each trajectory consists of language instruction and sequence of observation images, robot states, and actions:
also assume access to a dataset containing N expert trajectories
{1, 01, $1, a1, O2, 82, a2 Sr, ar)
As will be shown in Sec, 3.3 the model pre-trained by video prediction can be directly extended to finetuning robot data for learning visual robot manipulation.
3.2
ARCHITECTURE
GR-1 (Fig. [ is simple GPT-sttle transformer (Radford et al. 2018) which is able to take different modalities as inputs and outputs future images and actions.
3.2. INPUT
Lannuage input. The language is encoded via a text encoder (Fig. aa) Following (Shridhar
ct al. 2022 2023). we employ CLIP (Radford et al. 202I) as the language encoder.
Visual input. Visual observations o are encoded via a Vision Transformer (ViT) which has been pre-trained with MAE (He et al. 2022) (Fig ()). The output CLS token cCSS is used as global representation the image. The output patch tokens 2Pt are used as local representations which are further processed with perceiver resampler (Jaegle et al.) 2021) to reduce the token number.
Robot state input. The robot state contains the 6D pose of the robot end-effector Sarm (E)33) and a binary status of the gripper Sgripper E {0, 1}. We use linear layers to encode them (Fig. (b).
misstedd
Fercelier Brsompler
Vidaen Cecodeer
c Prezes
brcc"
ploper
GB-1
Figure 2: Encoders and Decoders. (a) Language encoder. (b) Robot state encoder. (c) Image encoder. (d) Image decoder. (e) Action decoder.
4

---

Published as a conference paper at ICLR 2024
Before being fed into the causal transformer, the embeddings of all modalities are passed through linear layers to align the dimension. For action prediction, we learn an action prediction token embedding to predict the arm and gripper actions. For brevity, refer to it as [ACT] For vidco prediction, we learn several observation prediction token embeddings to predict future frames. For brevity, refer them [OBS]. During video generative pre-training, the tokens are sequenced as:
(1, Oz-h: [OBS], o [OBS], ...., ....,,,00 0,, [O3S])
When finetuning on robot data, the tokens are sequenced as:
(1, St-h, O [OBS], [ACT], (, s: St, Ot, [OBS], [ACT])
Note that the language token is repeated in each timestep to avoid being overwhelmed by other modalities. To inject temporal information, add learned relative timestep embeddings to the tokens. All modalities at timestep share the same timestep embedding.
3.2.2 NETWORK
We follow the causal attention mechanism typically used in generative pre-trained models, except that all the [ACT] and [OBS] tokens are masked (Fig. [ That is, during pre-training, token can attend to all the tokens in the previous positions except the [OBS] tokens; during finetuning, token can attend to all the tokens in the previous positions except the [ACT] and [OBS] tokens.
We refer the reader to Radford et al. (2018) for more details about causal transformers.
3.2.3 OUTPUT
For video prediction, we attach a transformer decoder consisting of self-attention blocks and multi- layer perceptrons (MLPs). The decoder operates on the outputs corresponding the [OBS] tokens and mask tokens (Fig. d)) Each mask token is shared and learnable embedding added with corresponding positional encoding. The output corresponding to mask token reconstructs a patch of the predicted future image. Following He et al.] (2022), the loss function T-viden computes the mean squared error (MSE) between the reconstructed and original images in the pixel space. The outputs from the [ACT] tokens are passed through linear layers to predict the arm and gripper Smooth-LI loss Larm for training.
actions (Fig. (e)). Since the arm action is continuous, Gripper actions are optimized using Binary Cross Entropy (BCE) loss Lgripper..
3.3 TRAINING
We first pre-train GR-1 on video prediction and then finetune it on robot data (Fig. D. In both pre-training and finetuning phases, we freeze the CLIP text encoder and MAE image encoder.
Pre-Training. The data for the large-scale video generative pre-training are sourced from the re- cently proposed Ego4D dataset (Grauman al. (022) which contains massive-scale human-object interactions. Ego44) contains more than 3,500 hours of data. Each video clip also contains a natural language annotation describing the behavior of the person in the video. We crop short clip with a duration of 3s from each video. With this strategy, a total of 800,000 video clips, containing 8M frames, are collected. During pre-training, we randomly sample video sequences and train GR-1 to predict o (see the left part of Fig. TT. The network is optimized with causal video prediction loss Lvideo-
Robot Data Finetuning, The pre-trained model is finetuned by randomly sampling sequences from the robot dataset and optimizing GR-1 end-to-end with causal behavior cloning loss and video pre- diction loss:
Lúnetune
-arm + Lgripper + Livideo
(3)
4 EXPERIMENT
We perform experiments the challenging CALVIN benchmark (Mess et al. 2022c) and real robot. We aim to answer three questions: 1) Is GR-1 effective on visual robot manipulation? 2) Does GR-1 work on real robots? 3) Can GR-1 handle challenging settings including small dataset,
5

---

Published as a conference paper at ICLR 2024
Table 1: CALVIN Benchmark Results.
Tasks completed in row
Method
Experiment
Avg. Len.
MCIL
ABCD-1
0.373
0.027
0.002
0,000
0.000
0.40
RT-1
ABCD+D
0.844
0.617
0.438
0.323
0.227
2.45
HULC
ABCD
0.889
0.733
0.587
0,475
0.383
3.06
MT-R3M
ABCD DD
0.752
0.527
0.375
0.258
0.163
2.08
GR-1 (Ours)
ABCD -D
0.949
0.896
0.844
0.789
0.731
4.21
MCIL
ABC-DD
0.304
0.013
0.002
0.000
0.000
0.31
RT-1
ABC-
0.533
0.222
0.094
0.038
0.013
0.90
HULC
ABC-DD +D
0.418
0.165
0.057
0.019
0.011
0.67
MT-R3M
ABC-DD +D
0.529
0.234
0.105
0.043
0.018
0.93
GR-1 (Ours)
ABC-D +D
0.854
0.712
0.596
0.497
0.401
3.06
RT-1
10% data
0.249
0.069
0.015
0.006
0.000
0.34
HULC
10% data
0.668
0.295
0.103
0.032
0.013
1.11
MT-R3M
10% data
0.408
0.146
0.043
0.014
0.002
0.61
GR-1 (Ours)
10% data
0.778
0.533
0.332
0.218
0.139
2.00
RT-1
unseen lang
0.494
0.222
0.086
0.036
0,017
0.86
HULC
unseen lang
0.715
0.470
0.308
0.199
0.130
1.82
MT-R3M
unseen lang
0,512
0,249
0.106
0.040
0.017
0.92
GR-1 (Ours)
unseen lang 0.764
0.555
0.381
0.270
0.196
2.17
"push the sliding door to the right side
'Iff the red block from sliding cabinet"
"press the button turn the led ligh!"
"uull the handle to open the drawer"
"take the blue
"grasp the pink block"
block rotate to the right"
Figure 3: CALVIN Benchmark Results. We show examples multi-task learning trained on ABCD-> split.
generalization to unseen scenes, generalization to unseen objects, and generalization unseen lan- guages? We also perform ablation studies to understand how different modules of GR-1 help visual robot manipulation learning. Ablation studies and more results can be found in the appendix. Videos are available on the project page: https:: / /GR Man i inmanippllad pulat ion withhu.... io
4.1 CALVIN BENCHMARK EXPERIMENT
CALVIN a challenging benchmark focusing on learning language-conditioned policy for long- horizon robot manipulation (Fig. 3}. contains 34 tasks and features unconstrained language structions. The environment contains a Franka Emika Panda robot with parallel-jaw gripper and desk with a sliding door, drawer that be opened closed, blocks with different colors, an LED and light bulb that can be turned on or off.
Experiment Setup. For action prediction, similar to Mees et (2022c), we train GR-1 predict delta XYZ positions and delta Euler angles for arm actions and binary gripper actions. The training dataset contains over 20k expert trajectories paired with language instruction labels. Note that the CALVIN dataset contains 24 hours of teleoperated undirected play data. To simulate real-world scenario, only 1 % the data contains crowd-sourced language instruction labels, on which we train our method. We perform experiments stand for four different enviromments (Fig. The four environments are different in desk colors and object configurations. In ABCD-> evaluate in environment D. ABC->D, models are trained with data from environments A, B, and C and evaluated in environment D which is unseen during training.
two splits of data: ABCD-D and ABC-D. A. B, C, and
train models with data from all four environments and

---

Published as a conference paper at ICLR 2024
Baseline Methods. We compare with four baseline methods: MCIL (yynch & Sermanet 2020), RT-1 (Brohan et al. (022). HULC (Mees et al.) 2022b). and multi-task version of R3M (Nar eet al. (022). RT-1 (Brohan et al. 2022) is a state-of-the-art method that uses convolution layers and transformers to generate actions in an end-to-end manner. It uses FiLM layers to condition the convolu- tion layers with the pre-trained embedding of the language instruction. MCIL and HULC take hierarchical approach which first generates latent plans and conditions the plans on the policy to predict actions. These two methods take as inputs language instructions and observa- tion images taken from the static and gripper cameras. To showcase the effectiveness of video generative pre-training, we compare with another pre- pre-training method R3M (Nair et al. 2022) which is also pre- trained on Ego4D dataset. We use R3M to encode the observation images and leverage a GPT--tylee transformer to output actions. We freeze the R3M image encoder during training in Nair et al. (2022). We denote this multi-tasking method as MT-R3M. MCIL and HULC trained on the full CALVIN dataset containing data with and without language annotations. RT- 1, MT.R3M, and our method are trained on the data with language annotations which accounts for 1% of the full dataset. We refer the reader Mees al. (2022c) for more details on the dataset and language annotations.
Env
Env
Env C
Env
Figure 4: The positions
of the sliding door, LED, bulb, light switch, and but- ton are different across the four environments.
Multi-Tast Learning. We first perform experiments on the ABCD +D split. Quantitative results are shown in Tab. Qualitative results are shown in Fig. and g GR-1 outperforms all the baseline methods on sequentially completing 1, 2. 3, 4, and 5 tasks in row. The average length in the last column, computed by averaging the number of completed tasks in a row of 5 in all the evaluated sequences, shows the long-horizon capability a comprehensive way. GR-1 outperforms all the comparing baseline methods and improves the best baseline method from 3,06 to 4.21. This demonstrates 1) the superiority of our method on long-horize hong-horizon multi-tasking and 2) the effectiveness of video generative pre-training.
Zero-Shot Unseen Scene Generalization. We also perform experiments on the ABC-D split to evaluate the capability of zero-shot unseen scene generalization. Results are shown in Tab. [ GR- substantially improves the performance in terms of success rate and average length. Specifically, GR-1 achieves success rate of 85.4%, while that of the best baseline method is 53.3%. This demon- strates that GR-1 possesses strong zero-shot unseen scene generalization capability. We hypothesize that the zero-shot capability of our method is owing to pre-training on large-scale egocentric video clips with rich human-object interactions. This provides consistent and robust visual-textual align- ment across different environments for generalization.
Data Efficiency. Robot data is expensive and scarce compared to vision vision-language data. To study the data efficiency, we train on 10% data of the full training dataset from ABCD-D split. Specifi- cally, we sample 66 trajectories for each of the 34 tasks, i.e. 2244 trajectories, from the total 22,966 training trajectories. Results are shown in Tab. [ The performance of all methods degrades com- pared to training on the full dataset. The best baseline method, i.e. HULC, achieves success rate of 66.8% and an average length of 1.11. GR-1 significantly outperforms all the comparing bascline methods, achieving success rate of 77.8% and an average length of 2.00. This highlights that GR-1 is efficient when it comes to data. And this is very important as it allows GR-1 to quickly learn skills without collecting large amount of data.
Zero-Shot Unseen Language Generalization. To investigate whether GR-1 can generalize to un- seen language instructions, we use GPT-4 (OpenAT, 2023) to generate 50 synonymous instructions for each of the 34 tasks and randomly sampled from them during evaluation (Li et al. 2023). See Tab. 6 for some examples. For GR-1 and all the comparing baseline methods, we use the model trained on ABCD-D split for evaluation and test on the same sampled set of unseen language instructions. The results are shown in Tab. I The performance of all the methods drops when eval- uated on unseen language instructions. GR-1 outperforms all the comparing baseline methods. We hypothesize that this generalization capability attributes to 1) being exposed to diverse languages in the large video dataset during pre-training and 2) freezing the strong CLIP text encoder during the whole training process to retain its powerful text encoding capability.
7

---

Published as a conference paper at ICLR 2024
Object Transportation
Articulated Object
Seen Objects
Unseen Instances
Unseen Cotegories
Manipulation
bull pepper
the sggalen the plole"
the dosk"
piole
onto the plate"
"opan the
Figure 5: Real Robot Experiments. We perform real robot experiments of object transportation and articulated object manipulation.
Table 2: Real Robot Experiment Results.
Object Transportation
Articulated Object
Method
Seen Objects
Unseen Instances
Unseen Categories
Manipulation
RT-1
0.27
0.13
0.00
0.35
MT-R3M
0.15
0.13
0.10
0.30
GR-1 (Ours)
0.79
0.73
0.30
0.75
4.2
REAL ROBOT EXPERIMENT
To evaluate how GR-1 works in the real world, we perform real robot experiments of object trans- portation and articulated object manipulation (Fig. 55. More details and results can be found in the appendix.
4.2.1 OBJECT TRANSPORTATION
Experiment Setup. In this experiment, the basic scene for training contains a plate and three ob- jects: eggplant, broccoli, and bell pepper (leftmost figure in Fig. [5). We collected 1775 demonstrations of transporting one of the three objects from the plate to the desk or vice versa, with the HTC Vive VR system. We evaluate in three settings. In the first setting, denoted as Seen Objects, the robot is instructed to transport the three objects appeared in the training data. Besides evaluating scene that only contains these three objects as the training data, we also evaluate in two more unseen scenes with disturbance. In the first disturbed scene, we add a tomato, a corn, and a yellow peach as distractors; in the second one, further change the background by adding a wooden board and bowl. These two scenes help us evaluate the robustness of GR-1 against disturbance. In the second setting, denoted as Unseen Instances, we evaluate the capability of generalization to unseen object instances. The robot is instructed to transport novel set of eggplant, broccoli and bell pepper which are unseen in the robot training data. In the last setting, denoted as Unseen Categories, we evaluate the capability of generalization to unseen categories. That is, the categories of the trans- ported objects are unseen in the robot data. The robot is instructed to transport a tomato and yellow peach. Example instructions can be found in Fig. 5 and Fig.
Results. Quantitative results are shown in Tab. Qualitative results are shown Fig. 9 GR-1 outperforms the comparing baseline methods in all the three settings. In all settings, RT-1 and MT- R3M typically fail with picking the wrong object and incorrect placing. Another failure mode of RT-1 is collision with the plate or the desk. GR-1 achieves high success objects. And its performance only drops modestly in the setting of unseen instances. This showcases that GR-1 possesses powerful generalization to unseen instances. the most challenging setting of unseen categories, typical failure mode of GR-1 is that it sometimes mixes up the bell pepper with the peach which has a similar color.
in the setting of
4.2.2 ARTICULATED OBJECT MANIPULATION
In this experiment, aim to evaluate contact-rich articulated object manipulation. We evaluate GR-1 on manipulation of drawer (rightmost figure in Fig. 5}. We collected 2856 trajectories of opening and closing the drawer for training. Quantitative results shown in Tab. and qualitative
8

---

Published as a conference paper at ICLR 2024
"grasp the blue block and rotate it right"
"put the broccoli onto the plate"
"open the drawer"
Figure 6: Video Prediction Results. The images in green boxes are ground -ruth images; the images in blue boxes predicted images trained by setting At 1.
results are shown in Fig. GR-1 outperforms the two baseline methods by a large margin. Typical failure modes of GR-1 include 1) failing to completely close the drawer in the closing task and 2) failing to engage with the drawer handle when pulling it the opening task.
4.3 QUALITATIVE ANALYSIS ON VIDEO PREDICTION
We investigate the video prediction performance of GR-1 finetuned on CALVIN and real robot data. Qualitative results shown in Fig. 6 And more results can be found in the appendix. GR-1 is able to reconstruct the future image correctly on both CALVIN data and real robot data, although some details (e.g. occluded objects) are missing. The video prediction signal can serve strong guide for action prediction.
5 CONCLUSION
In this paper, we propose to leverage large-scale video generative pre-training for enhancing visual robot manipulation learning. We present GR-1, a GPT-style transformer that takes as inputs a lan- guage instruction, a sequence of observation images and robot states, and outputs actions and future images an end-to-end manner. GR-1 is first pre-trained on language-conditioned video prediction with large-scale video dataset. Owing to flexible design, it then be seamlessly finetuned on robot data to predict actions and future images. Extensive experiments were performed on both CALVIN benchmark and real robot verify the performance of GR-1. Results show that GR-1 improves state-of-the-ar methods in the settings of multi-task learning, zero-shot unseen scene gen- cralization, small dataset and zero-shot unseen language generalization on CALVIN benchmark. In addition, GR-1 outperforms comparing bascline methods in real robot experiments. By incorporat- ing large-scale video data, we showcase that GR-1 able to perform robustly in scenes which are disturbed heavily from those in the training data. More importantly. GR-1 is able to generalize to unseen object instances and categories in zero-shot manner. the future, we hope to combine video data both with and without languages in training to further enhance the robustness and gen- eralization capability of GR-1. Also, we want to explore the difference of pra-training on videos of any kind v.s. only videos that more relevant manipulation. In addition, plan scale up the robot data by increasing both the number robot trajectories in diverse environments and the number of manipulation skills.

---

Published as a conference paper at ICLR 2024
Acknowledgement We would like to express our gratitude the authors of Ego4D and CALVIN benchmark. Furthermore, we would like to extend our gratefulness to our colleagues at ByteDance Research for their support throughout this project.
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do As I Can, Not As Say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
Bowen Baker, Iige Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video Pre PreTraining (VPT): Learning to Act by Watch- ing Unlabeled Online Videos. Advances in Neural Information Processing Systems, 35:24639- 24654, 2022.
Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar. Robotic Offline RL from Internet Videos via Value- Function Pre- Trainngg arXiv preprint arXiv:2309. 13041, 2023.
Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X Lee, Maria Bauza, Todor Davehev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. RoboCat: A self-improving foundation agent for robotic manipulation. arXiv preprint arXiv:2306.
2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishhan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. RT-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068/ 2022.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choroman- ski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. RT-2: Vision- sisioo-languag--atttoon models transfer web knowledge to robotic control. arXiv preprint ar arXiv:2307.15816 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement learning via se- quence modeling. Advances in Neural Information Processing Systems, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691- 1703. PMLR, 2020.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion Policy: Visuomotor policy learning via action diffusion. arXiv preprint 1.Xiv:2303.04137, 2023.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, al. PaLM-E: An embodied multi- modal language model. arXiv preprint 2rXiv:2303303376 2023.
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv preprint arXiv:2302.00111 2023.
Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Young- woon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforce- ment learning. arXiv preprint arXiv:2305. 14343, 2023.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference Roboties and Automation (ICRA), pp. 2786-2793. IEEF, 2017.
10

---

Published as a conference paper at ICLR 2024
Kristen Grauman, Andrew Westbury, Eugene Bymne Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 18995-19012, 2022.
Pierre-Louis Guhur, Shizhe Chen, Ricardo Garcia Pinel, Makarand Tapaswi, Ivan Laptev, and Cordelia Schmid. Instruction-driven history-aware policies for robotic manipulations. Con- ference Robot Learning, pp. 175-187. PMLR, 2023.
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin and Li Fei-Fei. Visk Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 1894, 2022.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guider robot skill acquisition. arXiv preprint arXiv:2337. 14535, 2023.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02199, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner Monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05660, 2022.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International Conference on Machine Learning, pp. 4651 1664464. PMLR, 2021.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. BC-Z: Zero-shot task generalization with robotic imitation learning. In Con- ference on Robot Learning, 2022.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang. Yonggiang Dou, Yanjun Chen, Li Fei- Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. VIMA: General robot manipulation with multimodal prompts. arXiv preprint arXiv:2210.03094 2022.
Ya Jing, Xuelin Zhu, Xingbin Liu, Qie Sima, Taozheng Yang, Yunhai Feng, and Tao Kong. Ex- ploring visual pre-training for robot manipulation: Datasets, models and methods. In JEEE/RSI International Conference on Intelligent Robots and Systems, 2023.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. arXiv preprint arXiv:2302.127666 2023.
Aviral Kumar, Anikait Singh, Frederik Ebert, Yanlai Yang, Chelsea Finn, and Sergey Levine. Pre- training for robots: Offline RL enables learning new tasks from handful of trials. arXiv preprint arXiv:2210.05178, 2022.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. CURL: Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, pp. 5639- 5650. PMLR, 2020.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. lasiunnl foundation models effective robot imitators. arXiv preprint 1oXix:3311.01331 2023.
Xingyu Lin, John So. Sashwat Mahalingam, Fangchen Liu, and Pieter Abbeel. Spawnnet: Learning generalizable visuomotor skills from pre-trained networks. arXiv preprint arXiv:2307.03567, 2023.
11

---

Published as a conference paper at ICLR 2024
Fangchen Liu, Hao Liu, Aditya Grover, and Pieter Abbeel. Masked autoencoding for scalable and generalizable decision making. arXiv preprint arXiv:2211.12740, 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0300 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:171..05100 2017.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprini arXiv:2005.07648 2020.
Corey Lynch, Ayzaan Wahid, Jonathan Tompson, Tianli Ding, James Betker, Robert Baruch, Travis Armstrong, and Pete Florence. Interactive Language: Talking to robots in real time. arXiv preprint arXiv:2210.06407, 2022.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via aalue-implicit pre-training. arXiv preprint arXiv: 10.00003 2022.
Oier Mees, Jessica Borja-Diaz. and Wolfram Burgard. Grounding language with visual affordances over unstructured data. arXiv preprint arXiv:2210.01911 2022a.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4):1 1205- 11212, 2022b.
Oier Mees, Lukas Hermann, Erick Rosete- Beat, and Wolfram Burgard. CALVIN: A benchmark for language-condttineed policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L). 733):7327-7334 2022c.
Russell Mendonca, Shikhar Bahl, and Deepak Pathak. Structured world models from human videos. arXiv preprint arXiv:2308. /0901, 2023.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A univer- sal visual representation for robot manipulation. In 6th Annual Conference Robot Learning, 2022.
OpenAl. GPT-4 technical report. arXiv, pp. 2303-08774, 2023.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference Machine Learning, pp. 17359-17371. PMLR, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, llya Sutskever, al. Improving language under- standing by generative pre-training, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. pp. 8748-8763, 2021.
Ilija Radosavovie, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In Conference Robot Learning, 2022.
Ilija Radosavovic, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. Robot learning with sensorimotor pre-training. arXiv preprint arXiv:2306. 10007, 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Marth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061 22.22000..75... 175, 2022.
12

---

Published as a conference paper at ICLR 2024
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, pp. 12686-12699, 2021.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pp. 1332- 1344. PMLR, 2023.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 1134-1141 IEEE, 2018.
Nur Muhammad Shafiullah, Zichen Cui, Ariuntuya Arty Altanzaya, and Lerrel Pinto. Behavior Transformers: Cloning & modes with one stone. Advances in Neural Information Processing Systems, pp. 22955-22968, 2022.
Dhruv Shah, Blažej Osiński, Sergey Levine, et al. LM-Nav: Robotic navigation with large pre- trained models of language, vision, and action. In Conference on Robot Learning, pp. 492-504. PMLR, 2023.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. CLIPort: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906, 2022.
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Perceiver-Actor Actor: A multi-task transformer for robotic manipulation. In Conference on Robot Learning, pp. 785-799, 2023.
Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor.
SMART: Self-supervised multi-task pretraining with control transformers. arXiv preprint arXiv:2301.09811 2023.
Garrett Thomas, Ching-An Cheng, Ricky Loynd, Vibhav Vineet, Mihai Jalobeanu, and Andrey Kolobov. PLEX: Making the most of the available data for robotic manipulation pretraining. arXiv preprint 2rXiv:2333.0870 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 1
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Informa- tion Processing Systems, 2017.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint 2203022003.0673.. 2022.
Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
Wilson Yan, Danijar Hafner, Stephen James, and Pieter Abbecl. Temporally consistent transformers for video generation, 2023.
Taozheng Yang, Ya Jing, Hongtao Wu, Jiafeng Xu, Kuankuan Sima, Guangzeng Chen, Qie Sima, and Tao Kong. Moma-force: Visual-force imitation for real-world mobile manipulation. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6847-6852. IEEE, 2023.
Lin Yen-Chen, Andy Zeng, Shuran Song, Phillip Isola, and Tsung-Yi Yi Lin. Learning to see be- fore learning to act: Visual pre-training for manipulation. IEEE International Conference on Robotics and Automation (ICRA), pp. 7286-7293. IEEE, 2020.
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 NETWORK AND TRAINING DETAILS
The causal transformer of GR-1 contains 12 layers and 12 heads with hidden size of 384. In total, GR-1 contains 195M parameters, in which 46M of them are trainable. The output layer for action prediction is a three-layer MLP which the last layer contains two heads for predicting arm actions and gripper actions, respectively. The output layer for video prediction is a transformer consisting of self-attention blocks and linear layers. Following/He He et al. (2022), the prediction target is normalized patch-wise. Random shift augmentation is applied to the images. In pre-training, spaced frames from video clips in Ego4D dataset to train GR-1, The duration between consecutive frames is 1/3 seconds to ensure that they have sufficient visual difference. At each timestep, GR-1 is trained to predict the image at the next timestep in the sampled frame sequence, i.e. At = in Eq. I Robot data is denser compared to the sampled video frame sequences in pre-training. Therefore, when finetuning on robot data, we set At 3. And we train the network to predict captured images from the static camera and the gripper camera. The input sequence length is 10. We compare on different future steps in [A.4 We apply dropout and use AdamW (Loshchilov & Hutter (017) with cosine learning rate decay (Loshchilov & Hutter, 2016) optimize the network. Hyperparameters for pre-training and finetuning on CALVIN data are shown in Tab
sample equally-
Table 3: Training Hyperparameters
Hyperparameters
Pre-training
Finetuning
batch size
1024
512
learning rate
3.6e-4
le-3
dropout
0.1
0.1
optimizer
AdamW
AdamW
learning rate schedule
cosine decay
cosine decay
warmup epochs
5
training epochs
50
20
A.2 CALVIN BENCHMARK EXPERIMENTS
We follow the evaluation protocol in CALVIN (Mes et al. 2022c) and evaluate 1000 unique se- quence instruction chains. For each sequence, the robot aims to continuously solve up to 5 tasks by understanding a series of language instructions in a row. If task is not completed within 360 timesteps, it is considered failure. The robot receives the next task only if the current one is cessfully completed. After each sequence, the robot is set to neutral position. Fig. 8 shows more rollouts of GR-1 in different settings.
A.3 REAL ROBOT EXPERIMENTS
In real robot experiments, we use a 7-DoF Kinova Gen2 robot mounted with a RealSense camera on its end-effector. A Kinect Azure camera is used to provide the static view of the scene. In the articulated object manipulation experiment, there are tasks, i.e. opening and closing the drawer. In the object transportation experiment, we evaluate in three settings. In total, there are 10 tasks. They are "put the OBJECT onto the desk" and "put the OBJECT onto the plate". OBJECT=(eggplant, bell pepper, broccoli, tomato, yellow peach). For all the three scenes in the setting of Seen Objects (Fig. 3... the robot instructed to transport the three seen objects in the training data, i.e. an eggplant, broccoli, and a bell pepper. The settings of Unseen Instances and Unseen Categories share an identical scene which includes a tomato, yellow peach, and a novel set of eggplant, broccoli, and bell pepper. The robot is instructed to transport the unseen eggplant, broccoli, and bell pepper in the
14

---

Published as a conference paper at ICLR 2024
(a)
(b)
G8:1 mo Vidzo Presiction Predraiiing
GR wit Video Prodiction Pretra
GR-1 wio Vidso Pre.craiingg
(Rt1 MD Volee Meltraneg
6R:1
Succies Rate
0.8
Avg. Len.
0.4
Multi-lesk Learning
Generalization to Unseen Scenes
Data Efficienty
Picking
Transpartig
Figure 7: Ablation Studies. (a) We show the average length (averaged number of completed tasks row of 5) on CALVIN benchmark. (b) We show the success rates of picking and transporting in real robot experiments.
setting of Unseen Instances and the tomato and yellow peach in the setting of Unseen Categories, respectively. In both experiments, we use the same training setting as in the CALVIN experiments except that the batch size and training epochs are changed 64 and 30, respectively. See Fig. 9 for real robot rollouts.
Table 4: Ablation Studies.
Tasks completed in a row
Pre-Training
Video Prediction
Data
2
3
4
5
Avg. Len.
x
X
ABCD-D
0.889
0.775
0.661
0.549
0.459
3.33
X
ABCD+D
0.918
0.833
0.761
0.685
0.619
3.82
V
ABCD +D
0.949
0.896
0.844
0.789
0.731
4.21
X
X
ABC-D
0.823
0.609
0.425
0.318
0.225
2.40
X
ABC-D
0.815
0.651
0.498
0.392
0.297
2.65
/
ABC-D
0.854
0.712
0.596
0.497
0.401
3.06
X
X
10% data
0.526
0.288
0.138
0.061
0.022
1.04
X
10% data
0.698
0.415
0.223
0.133
0.052
1.52
10% data
0.778
0.533
0.332
0.218
0.139
2.00
A.4 ABLATION STUDIES
Video Prediction Pre-training. In this section, we perform ablation studies to study how video generative pre-training helps GR-1 on learning visual robot manipulation. We investigate two vari- ants of GR-1. The first variant, denoted as GR-1 w/o Video Prediction Pre-training, is trained from seratch and does not predict videos. That is, [CBS] tokens removed from the input token sequence. The second variant, denoted as GR-1 1 w/o Video Pre-training, is trained from scratch and retains video prediction. On CALVIN benchmark, we perform experiments on ABCD-DD split, ABC-D split, and 10% training data (Fig. aa) and Tab. 4. We also perform real-robot experi- ments in pick-and-place setting which uses the training scene in Fig. The robot is trained to perform picking an object on the desk/plate and placing it onto the plate/desk. Results are shown Fig. ()). GR-1 outperforms both variants all experiments. We hypothesize that this is be- cause the large-scale video pre-training helps GR-1 learn an accurate video prediction model which helps the robot understand what shall happen future steps given the language instruction and previous observations. And this information acts strong signpost for the robot to generate per- tinent actions for rolling out trajectories. Without pre-training, the video prediction of GR-I w/o Video Pre-training may not be as robust. On CALVIN benchmark, it outperforms GR-1 w/o Video Prediction & Pre-training but achieves a lower success rate in real robot experiments.
Different Future Predictions. We compare the effectiveness of predicting images different future steps (i.e. 1, 3, and 5) on CALVIN benchmark. Results are shown in Tab, [5] Pre-training is not used in this ablation. It is observed that increasing the step from to improves success rates. This may
15

---

Published as a conference paper at ICLR 2024
be because consecutive frames very similar and predicting frames that are farther away from the current step helps the robot to understand more about the future. But the improvement saturates soon. We hypothesize that this is because the model is trained to predict local actions and predicting frames that are too far into the future may not be able to provide good guidance for immediate local action prediction.
Table Ablation Studies Video Prediction.
Tasks completed row
Future Step
5
Avg. Len.
0.895
0.802
0.710
0.643
0.562
3.61
0.918
0.833
0.761
0.685
0.619
3.82
0.909
0.806
0.719
0.649
0.583
3.67
A.5 TASK SUCCESS RATES
To probe into how video generative pre-training helps visual robot manipulation learning, com- pare task success rates of GR-1 with GR-1 w/o Video Prediction & Pre-traing. Results are shown in Tab. 7 Tasks with large improvements mostly involve block manipulation. They difficult be- cause the robot needs to first grasp the correct block and then manipulate it according to the language instruction. With video generative pre-training, the performance of these tasks improves.
We also compare the task success rates with training on 10% data from ABCD-D split to investigate the effect of different data sizes. Similar the pattern found in the above, when increasing the data size, tasks involving block manipulation show large improvement. Also, the success rate of turning on/off lightbulb also increases substantially.
A.6 MORE RESULTS
Table 6: Examples of Unseen Language Instructions Generated by GPT-4 (OpenAl 2023 for the Zero-Shot Unseen Language Generalization Experiment on CALVIN.
Original
Generated
"use the switch to turn off the light bulb"
"use the switch to stop the light source"
"slide the block that it falls into the drawer"
"Move the block ensuring it goes into the drawer" "Acquire a grip on handle slide the drawer out"
"pull the handle to open the drawer" "lift the pink block from the sliding cabinet"
"Hoist up the pink block kept in the sliding cabinet"
"store the grasped block in the sliding cabinet"
"Hide the item gripped in sliding stash"
"take the red block and rotate the right"
"Twist the red object to the right"
"press the button to turn on the led light"
"press the switch to turn on the glowing LED" "grasp firmly and raise the blue block"
"grasp and lift the blue block"
16

---

Published as a conference paper at ICLR 2024
Table 7: Task Success Rates (%).
GR-1 w/o Video
GR-1 trained on 10% data
Task
GR-1
Prediction Pre-training
from ABCD split
rotate blue block right
94.9
71.2
51.6
slider right
99.3
99.6
86.9
lift red block slider
98.5
91.7
50.6
turn off led
100
100
95.6
push into drawer
82.9
79.8
74.7
lift blue block drawer
93.8
80.0
lift pink block slider
97.8
93.8
56.4
place in slider
91.3
89.1
34.8
open drawer
99.4
100
94.2
rotate red block right
98.6
81.9
45.3
lift red block table
97.7
76.7
36.5
lift pink block table
94.1
72.0
73.8
move slider left
99.2
99.5
90.7
turn on lightbulb
99.4
98.6
79.8
rotate blue block left
97.1
71.2
66.7
push blue block left
84.1
72.7
56.1
close drawer
99.5
98.8
91.3
turn off lightbulb
99.3
100
79.6
turn led
100
98.7
95.6
stack block
80.1
45.7
43.2
push pink block right
61.8
60.3
50.0
push red block left
82.3
77.8
54.7
lift blue block table
97.1
66.2
64.9
place in drawer
98.9
98.6
95.0
rotate red block left push pink block left
95.3
70.5
61.5
89.6
84.5
73.8
lift blue block slider
97.0
90.1
54.7
push red block right
54.2
49.3
43.6
lift pink block drawer
100
81.8
50.0
rotate pink block right
91.5
79.1
51.7
unstack block
100
84.4
100.0
rotate pink block left
96.4
70.4
66.0
push blue block right
53.6
50.0
33,9
lift red block drawer
100
100
87.5
17

---

Published as a conference paper at ICLR 2024
Long- Horizon Multi-Tasking: ABCD-
"store the grasped block in the sliding cabinet"
"pull open drawer"
"grosp and Iit the red block"
Zero-Shot Generallzation: ABC -D
"psh slidingg
"use the switch to turn off the light bulb"
"remove the stacked block"
Figure 8: Rollouts on CALVIN benchmark.
18

---

Published as a conference paper at ICLR 2024
"pul the broccoli onto the plate"
"put eggplant onto the plate
"put the bell pepper plate"
"put the bell pepper onto the desk"
"put the yellow peach onlo the plate
"open
"close the drawer"
Figure 9: Real Robot Rollouts. The first five show object transportation rollouts. The last two rows show articulated object manipulation rollouts.
19

---

Published as a conference paper at ICLR 2024
"go slide the pink block the right"
"grasp the blue block and rotate right"
"turn off the light"
"toggle the light switch to turn off the yellow light"
Figure 10: Video Prediction Results on Calvin Benchmark. The images green boxes are ground-truth images; the images in blue boxes are predicted images trained by setting At 1.
20

---

Published as a conference paper at ICLR 2024
"put the bell pepper onto the desk"
"put the eggplant onto the plate"
3
"close the drawer"
"opnn the drawer"
Figure 11: Video Prediction Results of Object Transportation and Articulated Object Manip- ulation Experiments in See. 4 The images in green boxes are ground-truth images; the images in blue boxes are predicted images trained by setting At 1.
21

---

Published as a conference paper at ICLR 2024
"place the picked object onto the desk"
"place the picked object onto the plate"
"pick the eggplant the plate"
"pick the broccoli the desk"
Figure 12: Video Prediction Results of Pick-and-Place Experiments in Ablation Studies in Sec. A.4 The images in green boxes are ground-truth images; the images in blue boxes are pre- dicted images trained by setting At 1.
22


Reviews:
[{"review_id": "mEXq4YK2RA", "reviewer": "Reviewer_QyTe", "summary": "The authors introduce GR-1, a GPT-style decoder-only transformer for language-conditioned robot manipulation. The model predicts both actions and future observations; the authors pretrain on Ego4D predicting future observations, followed by finetuning on robot data predicting both future observations and actions. The resulting model shows strong performance on the CALVIN benchmark as well as real-world language-conditioned robot manipulation on several pick-and-place tasks.", "questions": "- For the Ego4D pretraining, did you use the entire dataset, or some more relevant subset (e.g., hands and objects)?\n- Why was $\\Delta t = 1$ used during pretraining? If I understand correctly, this means that the next frame in Ego4D was always predicted, which is temporally very close considering that Ego4D videos are fairly high FPS.\n- Why is GR-1 w/o pretraining so much better than the baselines in the CALVIN zero-shot setting?\n-  \"283M parameters, in which 46M of them are trainable\" -- to clarify, this means that the majority of the parameters are the frozen MAE and text encoder parameters?\n- What is the control frequency of the real robot during the execution of the policy?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The paper is well-written, well-presented, and easy to follow. The idea is simple and easy to understand, which is a good thing. While the basic idea of training a GPT-style transformer in this manner is not exactly brilliant or surprising, it hasn't been done in prior work (as far as I know), and the design of the model is principled and well-executed. The CALVIN results are strong, improving significantly on the state-of-the-art.", "weaknesses": "The value of GPT-style video pretraining is the core narrative of this paper, and the primary way that the authors claim GR-1 improves on prior work. Given that, I would say the main weaknesses of this paper are related to a lack of sufficient evidence and discussion surrounding this claim.\n\n- Training details/hyperparameters/etc are missing for the Ego4D pretraining phase, which should definitely be included given their importance to the method.\n- The real robot experiments are a bit weak. There are not very many tasks, they are fairly simple, and the tested level of generalization is not very significant. Breaking down the tasks into separate pick and place phases provides some inherent supervision that makes the tasks  easier; I would have liked to see slightly more end-to-end tasks (e.g., \"put the broccoli on the plate\"). Given how broad the pretraining dataset is, if it really does help, it seems like better zero-shot generalization should be possible: e.g., slightly different objects not found in the robot data.\n- The baselines are severely lacking. For example, the authors claim that \"compared with VIMA, our method is capable of dealing with much more complex 6-DoF long-horizon language-conditioned robot manipulation tasks\". A statement like that should be supported by a comparison to some sort of VIMA-like method (i.e., an encoder-decoder architecture with cross-attention). There is also *tons* of prior work on video pretraining for robotics, much of which is cited in Section 2.3: e.g., R3M, VIP, MVP, and many more. Many of these also pretrain on Ego4D and finetune on robot data. It is impossible to evaluate the strength of GR-1 without comparison to these other methods.\n\nI would be willing to increase my score if more comparisons were added in both the simulation and the real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wp9bjCUonG", "reviewer": "Reviewer_SaZc", "summary": "The paper develops an approach for multi-task visual robotic manipulation by pre-training for video prediction on non-robotic datasets, and then fine-tuning on robot data. The main contribution of the paper is to demonstrate that pre-training for the task of video prediction on human datasets enables sample-efficient fine-tuning for manipulation, with some generalization across tasks. The experiments involve manipulation tasks on a simulation environment, and pick/place tasks in the real world.", "questions": "Please refer to the weaknesses above for reference. \n\n- Is it possible to show generalization results (similar in line to the zero-shot results) but for more tasks and variations within tasks, like different scenes, different objects etc.? This is expecially important because the current simulation benchmark seems saturated with 88% baseline performance, and only marginal improvement to 94% with the proposed approach. \n\n- For the real-world experiments, is it possible to show benefits in tasks beyond pick/place and those that frequently occur in the pre-training Ego4D dataset like articulated object manipulation, scooping, pouring etc.?\n\n- Why have ABC-> D style result instead of training on all the available tasks (except the held-out one)? I might be missing something here, so would appreciate a clarification, but it seems to me that the zero-shot result in the paper doesn't seem to be training a unified model across all but held out tasks, which seems a bit odd. \n\n- Is it possible to have comparisons to prior works that also train on datasets similar/identical to Ego4D but have different pre-training objectives? (please refer to weaknesses above for details)\n\n- Is it possible to show qualitative results for the video-prediction (similar to Fig 6 and Fig 11) but for more real-world evaluations? If possible it will be helpful to show some *random* generations to get a sense of how much \n\n- Is there no forgetting in the pre-training followed by fine-tuning regime (as opposed to co-training)? My sense is that after fine-tuning on the robot dataset, the information acquired from pretraining will be largely lost due to domain shift in the two datasets. One simple way to test this is to look at video generation results of the model on Ego4D clips, just after pre-training, and after pre-training+finetuning. My sense is that the latter will be much worse than the former. I am curious if the authors considered any co-training strategy, where for fine-tuning, the dataset consists of some clips from the pre-training dataset as well as the robot data?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The task of video prediction on non-robot datasets for as a pre-training step for downstream manipulation is interesting, and novel to the best of my understanding. Importantly, this pre-training task can ingest all language-annotated video datasets, can hence can be potentially scaled to diverse videos on the web, beyond the Ego4D dataset in the paper. \n\n- Three external baselines are implemented for comparison, and experiments show some evidence of real-world manipulation, which strengthens the claims of the paper. \n\n- The vision transformer architecture is without significant bells and whistles, and has standard GPT-style encoder-decoder structure, making it easy to implement and train. In my understanding, the fine-tuning on robot data is very similar to the pre-training step in terms of training details, and doesn't require additional considerations. \n\n- Overall, the paper is well-written, easy to understand, and the approach is clearly described.", "weaknesses": "- The results doesn't quite succeed in showing any generalization benefits of pre-training for video prediction on diverse data. This seems to be the main claim of the paper, and requires more thorough experiments for validation. The simulation environments seem to have all the tasks in a similar table-top setting, with very little scene variation across tasks. The zero-shot result is interesting, but it is shown for only a single task.\n\n- The real-world experiments are in very simple pick and place tasks. The Ego4D videos used for pre-training contain such rich skills like articulated object manipulation, scooping, pouring etc. and so for properly showing the benefits of pre-training it is important to demonstrate that the approach can indeed tackle some of the interesting contact rich tasks that involve motions/skills beyond pick and place. In addition, the generalization results are limited to minor distractor and background variations, without variations in the task objects themselves. For reference, several recent robotics papers tackle such tasks and generalizations [1,2] \n\n- The baselines are missing all relevant papers that also pre-train on human videos / image datasets. Many of these are cited in the related works, but some comparison is necessary to actually demonstrate that pre-training for video prediction has benefits compared to pre-training for other objects like masked prediction, contrastive learning etc. (which is a claim of the paper). For example, a multi-task version of R3M [3] can be a relevant baseline that pre-trains on Ego4D but to learn visual representations, and then fine-tunes with behavior cloning. \n\n[1] Walke, Homer, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao et al. \"Bridgedata v2: A dataset for robot learning at scale.\" arXiv preprint arXiv:2308.12952 (2023).\n\n[2] Bharadhwaj, Homanga, Jay Vakil, Mohit Sharma, Abhinav Gupta, Shubham Tulsiani, and Vikash Kumar. \"Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking.\" arXiv preprint arXiv:2309.01918 (2023).\n\n[3] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3M: A universal visual representation for robot manipulation. In 6th Annual Conference on Robot Learning,\n2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fCpGl1THTd", "reviewer": "Reviewer_Cjar", "summary": "The paper presents an approach for large-scale video pretraining for robot manipulation. The presented method, termed GR-1, first pre-trains a causal GPT-style transformer on actionless video data, and then finetunes the representation on a smaller number of robot demonstrations with actions. The key aspects of the approach are (1) using a GPT-style transformer for the embodied foundation model and (2) predicting the video tokens/generating future video frames as part of the training process. The approach is tested on the CALVIN simulation benchmark and on a real-world robot platform, outperforming the considered baselines.", "questions": "1. How different is the D simulation setting from A, B, and C?\n2. What is the $\\Delta_t$ used in the experiments (e.g., 0.1 s)?\n3. Why is only one action predicted at a time rather than a receding horizon style prediction? [B] found the latter to work well.\n4. Is there multimodality in the distribution that can be modeled for the video frame prediction task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The general problem of making use of actionless human video data is of interest and importance to the research community.\n* The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.\n* The paper is well-written and easy to follow.\n* The figures are informative and effectively illustrate the benefits of the proposed approach.\n* The experiments consider both simulation and real robot evaluation, as well as an ablation study, demonstrating GT-1's superior performance as compared to the considered baselines and support for GT-1's design choices.\n* The discussion did a good job of describing the weaknesses and failure modes of the proposed method as well as the baselines.", "weaknesses": "* The literature review is missing a number of relevant works. The baselines considered are also not necessarily state-of-the-art. This year, a number of works have come out that would be prudent to compare against.\n  * V-PTR: similar high-level motivation of using video-based, prediction-focused pre-training and then action-based finetuning. This should have likely served as a baseline for the proposed method. \n    * [A] Bhateja, Chethan, et al. \"Robotic Offline RL from Internet Videos via Value-Function Pre-Training.\" arXiv preprint arXiv:2309.13041 (2023).\n  * Diffusion policy: diffusion policy has shown very good results in terms of multi-task, low-data regime performance. \n    * [B] Chi, Cheng, et al. \"Diffusion policy: Visuomotor policy learning via action diffusion.\" arXiv preprint arXiv:2303.04137 (2023). \n    * [C] Ha, Huy, Pete Florence, and Shuran Song. \"Scaling up and distilling down: Language-guided robot skill acquisition.\" arXiv preprint arXiv:2307.14535 (2023).\n  * Pretrained video representations, such as R3M, VIP, and Voltron consider transformers, MAEs, and temporal video frames similarly to the proposed work. Taking a policy learning approach with a pre-trained representation such as the ones listed would have been another good option for a baseline. \n    * [D] Nair, Suraj, et al. \"R3M: A universal visual representation for robot manipulation.\" arXiv preprint arXiv:2203.12601 (2022). \n    * [E] Ma, Yecheng Jason, et al. \"VIP: Towards universal visual reward and representation via value-implicit pre-training.\" arXiv preprint arXiv:2210.00030 (2022). \n    * [F] Karamcheti, Siddharth, et al. \"Language-driven representation learning for robotics.\" arXiv preprint arXiv:2302.12766 (2023).\n* Given above works, I am not sure that the statement \"GR-1 for the first time shows that a unified GPT-style transformer, augmented with large-scale video generative pre-training is able to effectively generalize to multi-task visual robot manipulation.\" is accurate/precise.\n* I found section 3.2.2 to be a bit confusing in describing the masked tokens. The OBS token is masked to predict the future video frames, is that correct? This sentence: \"all the tokens, including the [OBS] tokens, can only attend to all the language and observation tokens but not the [OBS] in the past\" was particularly unclear for me. Why specifically the tokens in the past? The notation of [OBS] and observation tokens adds to the confusion.\n* In the experimental setup, I did not quite understand the actual size of the training set. My assumption is that GT-1 does not handle data both with and without language.\n* It would be helpful to have some measure of statistical significance for the results (e.g., standard error) to understand whether the differences in performance are meaningful.\n* Future work description should be more insightful/in-depth than 'combine more robot data and video data in training'.\n\nSome typos and points of confusion are listed below:\n\n1. Often 'causal' is typoed as 'casual'.\n2. In Sec. 4.1, '[an] LED and a light bulb'? \n3. In Sec. 4.1, 'datset'.\n4. In Sec. 4.1, 'This highlight[s]' ... 'without collecting [a] large amount of data'.\n5. The references should be proofread (e.g., to ensure the year is not entered twice, appropriate acronyms are capitalized (e.g., 'rl'), conference name formatting is consistent in terms of capitalization, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RXgjGv3ro4", "reviewer": "Reviewer_H1qB", "summary": "This paper proposes GR-1, a GPT-like autoregressive Transformer model that leverages large-scale video generative pretraining and downstream robotic dataset finetuning to solve language-conditioned robotic manipulation tasks. Specifically, the model is pretrained on Ego4D to reconstruct a future observation from the current observation, and then finetuned on robotic datasets with a combination of video prediction and robotic action prediction losses. Experiments on the Calvin Benchmark and on real robots show that the proposed method achieves better language-conditioned robot manipulation success rates on different task variations.", "questions": "See \"weaknesses\"", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is clearly presented and easy to read.\n- The proposed video generative pretraining approach, combined with downstream finetuning on task-specific robotic manipulation datasets, achieves better language-conditioned robotic manipulation performance than previous baselines like RT-1.\n- The effectiveness of the proposed approach not only holds in simulation, but also transfers to real robot tasks.\n- The proposed approach shows promises in scaling up to larger-scale video pretraining datasets and more robot manipulation tasks in the future.", "weaknesses": "- Is there a plan to open-source the code and checkpoints? Making them available to the community would greatly facilitate efforts to scale up pretraining for robotics.\n- Comparing Table 1 and Table 2, the performance on the Calvin Benchmark significantly falls when only using 10% of robotic dataset for finetuning (2k trajectories) compared to using 100% of finetuning dataset (20k trajectories). According to the table, it looks like the primary reason is that the success rate primitive skills (i.e., success rate for \"1 task completed in a row\") falls from 94.9% to 77.8%. Thus it would be helpful to provide a further analysis into this phenomenon along with the failure modes.\n- For zero-shot generalization experiments conducted in the paper, looks like the experiments are primarily conducted on object position variations, color variations, and background variations. It would be helpful to provide an analysis on whether the policy can (1) generalize to longer horizon tasks than the length of the tasks seen during training (e.g., train the policy on 1-5 tasks in a row, test on 6-10 tasks completed in a row); (2) generalize to novel language instructions that are semantically similar to those seen during training but with different wordings (e.g., instead of \"lift the red cube\", use \"pick up the red cube\" as instructions; instead of \"push the sliding door to the right\", use \"open the door to the right side\" as instructions)\n- It would be helpful to provide an ablation on the effect of different quantities of Ego4D pretraining datasets on downstream robot manipulation task performance. Also is there a (possibly smaller) subset of Ego4D dataset for pretraining that is especially helpful for downstream robot manipulation tasks?\n- The number of visual features per image from the MAE encoder can be large, and as a result, the current policy might not generalize to very long horizon tasks. It would be helpful to explore using e.g., Perceiver Resampler, to downsample the number of input image features.\n\nFor the second to the last weakness points listed above, I wouldn't expect authors to address all of them given the limited rebuttal time, but it would be great if authors could address most of them.\n\n\n\n\n\n**Update Nov. 23**: Thanks authors for the rebuttal! All of my concerns have been addressed, and I believe that the paper will be very valuable for the vision language & robotics community to scale up generative pretraining efforts for large-scale, open-world robotic manipulation skill learning. The authors also promise to release the code and checkpoints. Thus I've increased my rating and confidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)