Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulation
Authors: Yang Tian
Sizhe Yang
Jia Zeng
Ping Wang
Dahua Lin
Hao Dong
Jiangmiao Pang
Venue: ICLR 2025 Oral (2025)
Keywords: Robotic Manipulation ; Pre-training ; Visual Foresight ; Inverse Dynamics ; Large-scale robot dataset
Primary area: applications to robotics, autonomy, planning

Abstract: Current efforts to learn scalable policies in robotic manipulation primarily fall into two categories: one focuses on "action," which involves behavior cloning from extensive collections of robotic data, while the other emphasizes "vision," enhancing model generalization by pre-training representations or generative models, also referred to as world models, using large-scale visual datasets. This paper presents an end-to-end paradigm that predicts actions using inverse dynamics models conditioned on the robot's forecasted visual states, named Predictive Inverse Dynamics Models (PIDM). By closing the loop between vision and action, the end-to-end PIDM can be a better scalable action learner. In practice, we use Transformers to process both visual states and actions, naming the model Seer. It is initially pre-trained on large-scale robotic datasets, such as DROID, and can be adapted to real-world scenarios with a little fine-tuning data. Thanks to large-scale, end-to-end training and the continuous synergy between vision and action at each execution step, Seer significantly outperforms state-of-the-art methods across both simulation and real-world experiments. It achieves improvements of 13% on the LIBERO-LONG benchmark, 22% on CALVIN ABC-D, and 43% in real-world tasks. Notably, it demonstrates superior generalization for novel objects, lighting conditions, and environments under high-intensity disturbances. Code and models will be publicly available.

Published as a conference paper at ICLR 2025
PREDICTIVE INVERSE DYNAMICS MODELS ARE SCAL- ABLE LEARNERS FOR ROBOTIC MANIPULATION
Yang Tian Sizhe Yang..1, Jia Zeng'
Ping Wang
Dahua Lin
Hao Dong? Jiangmiao Pangl.
1 Shanghai AI Laboratory
CFCS, School of CS, Peking University
3 National Engineering Research Center for Software Engineering, Peking University
School of Software & Microelectronics. Peking University
5 Key Laboratory of High Confidence Software Technologies (PKU), Ministry of Education
Chinese University Hong Kong
Equal contribution, Author ordering determined by coin flip
Corresponding authors
Project page: https: / /ttt:s///mmmtt//// gigtgithi Lo/Seer/
ln
440%
Large-scale
Robot
Behavior Cloning Policy
Epiendes
prodiction
+10%
End-to-End
Pre-training
Visual Prediction Module
Visusl Precidion Model
Vision
LIBERD
Action
+21% +12%
Inverse Dynamics Module
Limited
Inverse Dynemios Model
Downstream Task DatA
SOTA
Swer PT)
Fine-tuning
(b) Two-Stage PIDM
(c) End-to-End PIDM
Beer (w PTy
Figure 1: In contrast previous methods that (a) conduct end-to-end naive behavior cloning from large-scale robotic data or (b) use decoupled visual prediction and inverse dynamics models to set goals and guide actions, present end-to-end Predictive Inverse Dynamics Models (PIDM) that closes the loop between vision and action. Seer, the model we built, surpasses previous states of the art and demonstrates consistent improvements over the ablated version without pre-training.
ABSTRACT
Current efforts to learn scalable policies in robotic manipulation primarily fall into two categories: one focuses on "action," which involves behavior cloning from ex- tensive collections of robotic data, while the other emphasizes "vision," enhancing model generalization by pre-training representations or generative models, also re- ferred world models, using large-scale visual datasets. This paper presents end-to-end paradigm that predicts actions using inverse dynamics models condi- tioned the robot's forecasted visual states, named Predictive Inverse Dynamics Models (PIDM). By closing the loop between vision and action, the end-to-end PIDM can be better scalable action learner. practice, we use Transformers process both visual states and actions, naming the model Seer. It is initially pre- trained on large-scale robotic datasets, such as DROID, and can be adapted to real- world scenarios with a little fine-tuning data. Thanks to large-scale, end-to-end training and the synergy between vision and action, Seer significantly outperforms previous methods across both simulation and real- world experiments. It achieves improvements of 13% on the LIBERO-LONG benchmark, 21% on CALVIN ABC-D, and 43% in real-world tasks. Notably, Seer sets new state-of-the-art CALVIN ABC-D benchmark, achieving an average length of 4.28, and exhibits superior generalization for novel objects, lighting conditions, and environments under high-intensity disturbances on real-world scenarios. Code and models are publicly available at attps:://///.........ttt https:///qithut www com./openkkoootlaa//ee.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Learning scalable and generalizable policies has become a main focus in robotic manipulation. Re- cent efforts primarily fall into two categories: one focuses on "action," " like RT-1 (Brohan et al.) 2022). Octo (Ghosh et al. 2024), and OpenVLA (Kim et al. 2024), which perform naive behav- 10r cloning from large-scale robotic data such as Open X-Embodiment and DROID (Padalkar et al.) 2024 Khazatsky et al. 2024). The other emphasizes "vision" and may learn representations through discriminative or generative ways and integrate with the control policy in a two-stage manner. For example, R3M (Nair al.
2022) and MVP (Xiao al. 2022) learn discriminative representa- tions from large-scale video datasets such as Ego4D (Grauman et al. 2022), while UniPI (Du et al.) 2024), Susie (Black et al., 2023), and CLOVER (Bu et al. 2024) develop generative models as "world models" to facilitate manipulation policies. Apparently, the scaling laws in robot learning are still evolving, with researchers exploring strategies through diverse data and methods.
We revisit these approaches and propose that a scalable manipulation policy should integrate vision and action in closed loop. This integration is natural and necessary. as humans typically coordinate their hands and eyes to manipulate objects. Therefore, closing the loop during training and inference are both necessary for a better scalable action learner.
This paper achieves this by introducing a simple yet effective end-to-end Predictive Inverse Dy- namics Models (PIDM) that can unify the advantages of previous methods. As shown in Figure [ predicts actions using Inverse Dynamics Models (IDM) conditioned on the "Predictive" visual states of the robot. During training. both the visual prediction module and the inverse dynamics module optimized synergistically in an end-to-end manner. During inference, PIDM ensures continuous synergy between vision and action at each execution step. In contrast to previous methods that use IDM, our approach is the first optimize vision and action in an end-to-end manner. Throughout this paper, unless otherwise specified, PIDM is assumed to be end-to-end.
In practice,
Transformers process both visual states and actions and name the model Seer. Seer benefits from PIDM by simultaneously leveraging visual, temporal, and action information from large-scale datasets, and can be more scalable due to the Transformer architecture. We intro- duce a foresight token to predict future RGB images and an action token to estimate intermediate actions between current and predicted future observations. Both tokens are fused with input RGB images, robot state, and language tokens through a multi-modal encoder. Importantly, we design a unidirectional attention mask that allows the action token to deeply integrate past and future predic- tive information, facilitating end-to-end training.
We conduct extensive experiments on both simulation and real- world benchmarks. On two widely adopted simulation benchmarks, LIBERO-LONE (Liu et al.) 2024) (10 tasks) and CALVIN ABC- (Mees et al., 2022) (34 tasks), our method demonstrates a 10.4% improvement success rate and 0.75 increase in average task completion length compared to state-of-the-an baselines. Our results further indicate superiority in long-horizon task completion, unseen scene generalization, and data efficiency. Additionally, We evaluate our method on six challenging real-world tasks with over 900 trials. Leveraging the public large robot dataset DROID (Khazatsky et al. 2024), our method consistently shows robustness, even under disturbances and with limited fine-tuning data.
2 RELATED WORK
Action-Centric Pre-training for Manipulation. Recent advancements in action-centric pre- training have significantly enhanced manipulation policies. Approaches like SMART (Sun et al. 2023) and DualMind (Wei et al. (003) emphasize understanding the dynamics within environments. Some studies (Agrawal et al.) 2016 Brandiobbrener et al.) (2224) integrate current and goal infor- mation to extract effective features or serve as an auxiliary objective. Subsequently, standard behavior cloning approach is applied during downstream task implementations. Additionally, RT- X (Padalkar et al., 2024) and Octo (Ghosh et al., 2024) focus on pre-training robot policies using diverse datasets to facilitate extensive generalization capabilities. Recently, vision-language mod- els (VLMs) have demonstrated considerable common-sense knowledge about the world and strong capabilities in understanding both language and images. VLe (Kim et al.) (024) further pre- trains VLMs using robotic data, leveraging their prior knowledge to achieve robust performance on downstream language-conditioned manipulation tasks. While these methods primarily supervise
2

---

Published as a conference paper at ICLR 2025
actions, they do not fully exploit the rich visual and temporal information inherent in robot demon- strations. In contrast, we pre-train policies by integrating conditional visual foresight and inverse dynamics prediction, allowing for comprehensive utilization of robotic data.
Vision-Centric Pre-training for Manipulation. Extensive research has focused on visual pre-
training for visuomotor control (Karamcheti al.,
2023 Zeng et al. 2024). One major direction
involves representation learning using techniques such as masked image modeling (iao et al. 2022
Radosavovic et al. 2023 Seo et al. 2023), contrastive learning (Nair al., [022] Ma et al. 2022
2023), and generative video pre-training (Wu el al., 2024). Another line of work focuses on visual expectations guiding actions, termed the Predictive Inverse Dynamics Model (PIDM) (Bharadhwaj
iet al. [224] Wang et al.) 2024: Soni et al.) 2024 Chen et al.) 2024). Firstly, a video generation model predicts future visual sub-goals and pre-trained on general visual datasets. Then, an in- verse dynamics (goal-conditioned) low-level policy is trained on downstream robot data to predict the intermediary actions. Compared to these two-stage PIDM, we propose an end-to-end PIDM paradigm that leverages large-scale robot data for pre-training, showing better performance.
Pre-training Datasets for Manipulation. High-quality, large-scale, and diverse pre-training data crucial for acquiring manipulation skills. Image datasets (Deng et al. 2009). video datasets (Damen jet al., 2018 Goyal et al., 2017); (Grauman et al., 2022). and robot datasets are commonly utilized for this purpose. Image datasets provide rich semantic information, while video datasets contain temporal information. Both enhance visual representations for manipulation, however, their lack of action labels and robot states limits their utility in decision-making. Some studies focus collecting robot behavior data (Mandlekar et al., 2019; Walke et al.) 2023] Dasari et al.) 2019 Bahl et al. 2023 Jang al. 2022), but the data diversity remains relatively constrained. Recent efforts aim to Turther scale and diversify robot datasets. For instance, the Open X-Embodiment dataset (Padalkar
al. 2024) aggregates data from 22 different robots across 21 institutions, covering 527 skills and 160,266 tasks. DROID (Khazatsky et al., (024) includes 76,000 trajectories collected across 564 scenes and 86 tasks. In this work, we leverage DROID to pre-train policies for real-world validation, demonstrating that rich behavioral data significantly enhances success rates in downstream tasks.
METHOD
In this section, we describe Seer in detail. We begin with a brief problem formulation (Section 3.1 Next, we discuss keys in our end-to-end prediction (Section 3.2). enabling Seer to forecast the future and adjust actions accordingly. We then elaborate on the carefully designed model architecture (Section 3.3), through which we formulate Seer in an end-to-end manner. Finally, we provide implementation details (Section 3.4).
visual foresight and inverse dynamics
3.1 PROBLEM FORMULATION
Given a large-scale dataset of diverse manipulation demonstrations D1 = {(1, or, St, 04) o0) i= and a smaller downstream dataset D2 = {(l, ot, 8t, at) o=) (where Ni >> N2), our goal is to enhance downstream task performance through effective pre-training on D1. followed by fine- tuning D2. Each trajectory {(1,o,, ot, $t, ) provides the time step t, language instruction RGB images oe from the eye-on-hand and eyc-on-base views, robot states s; and robot actions a,, which include arm actions aamn (6D pose) and gripper actions Ogripper (open close). It is important to note that current large pre-training robot data may contain incomplete language annotations / and task-agnostic actions a,, such random exploration in the environment (Mees et al.) (0022). However, Seer could handle this scenario effectively due to the following specific design choices.
N.
3.2 END-TO- TO-END PIDM
Vision: Conditional Visual Foresight. A key insight is that informative future states guide actions. Therefore, we propose conditional visual foresight ftore to effectively anticipate future visual repre- sentations. Seer takes as input a goal g in the form of language instructions or robot states, along with historical observations hy, and predicts the RGB images at the time step t t+n, denoted by ô
Öttn n Store (g, he)-
3

---

Published as a conference paper at ICLR 2025
Language
Robot State S
Foresight Token
Action Token
[FRS]
[INV]
1
Current RGB image O
...
...
1
Unidirectional Attention
Multi-Modal Encoder
add Mask Token
Action
Conditional Visual AssioncAAction
Inverse Dynamics
Foresight
Prediction
Predicted Image
Figure 2: Pipeline of Seer. Seer consists of three parts: Multi-Modal Encoder, Conditional Visual Foresight and Inverse Dynamics Prediction. In Multi-Modal Encoder, Seer incorporates the fore- sight token [FRS] and the action token [INV]. Both tokens attend to the RGB images, language tokens, and robot state tokens, with [INV] also attending [FRS]. In Conditional Visual Foresight, the encoded [FRS], along with new mask tokens, aims to reconstruct future RGB images. In Inverse Dynamics Prediction, the encoded [INV] and other tokens speculate intermediary actions.
The historical observations h2 consist of the RGB images Ot-m+1:: and robot states 81m++1+ the last m time steps. Due to the rich information contained in RGB images, their abundance, and easy accessibility, we select them as future representatives. Following (He al.] (022), the loss function Lfore computes the mean squared error (MSE) at the pixel level
Lior = fl h4) o.-n112
(2)
Inverse Dynamics Prediction. Given two temporally ordered observations or and Or+1, inverse dynamics prediction estimates the intermediate action à, Here, we extend inverse dynam- ies fin to predict the action sequence â given goal g. historical observations h; and Ottn- Specifically, we replace ground truth Ot+o with the predicted representation of in the latent space
Action:
a finv f4, d....).
(3)
The loss function Lime comprises the action loss farn and the gripper action loss Lgripper
Lin Lum Xgripper
(4)
where Larm is Smooth-L1 loss, Lyriper is Binary Cross Entropy (BCE) loss and À is set to 0.01.
Close the Loop between Vision and Action. Seer integrates conditional visual foresight with in- verse dynamics prediction effectively through training, enabling full utilization of both vision and action information robot data. In detail, ftore incorporates a clear goal g and historical observa- tions ft to predict future RGB images 0z+ A latent representation 0/4, (leading to Ôt+n) and he facilitate action prediction via fav. Due to the model design of Seer, all these processes are executed in an end-to-end to-end manner. The overall training loss L comprises Liore and Lin
L = C + Linv:
(5)
where a is a hyperparameter set to 0.5. Compared to single-step action prediction, predicting mul- tiple steps provides temporal action consistency and robustness to idle actions (Chi et al.) (023). During inference, we can either discard actions beyond the first step or apply temporal ensemble techniques to compute weighted average of the multi-step actions.
3.3 MODEL ARCHITECTURE
Input Tokenizers. As illustrated Figure 2 the model processes three types of inputs: language, images, and robot states. We use different encoders to tokenize each modality accordingly. For language inputs, we first tokenize the text and then use a CLIP text encoder (Radford et al., (202I) to obtain text embeddings, which are subsequently projected into latent space using linear layer. For
4

---

Published as a conference paper at ICLR 2025
image inputs. the images are first patchified and passed through a pre-trained Vision Transformer (ViT) (He et al. 2022) to generate visual embeddings. Since the ViT produces hundreds of em- beddings per image, imposing significant computational burden on the transformer backbone, and much of the visual information is irrelevant to the manipulation task, we employ a perceiver resam- pler (Alayrac et al., 2022) extract task-relevant visual features and reduce the number of image tokens. For the robot state, we encode it into state tokens using a multi-layer perceptron (MLP).
Multi-Modal Encoder. The multi-modal encoder in our model is based on GPT-2 style trans- former architecture. Before feeding the sequential language-image-state pairs into the transformer, we append readout tokens [INV] and [FR]] each timestep. These readout tokens attend to em- beddings from different modalities, serving as image and action latents used for conditional visual foresight and inverse dynamics prediction. To incorporate temporal information, we also add learnable position embedding to the tokens for each timestep.
The [FRS] token aims to facilitate conditional visual foresight, corresponding to aforementioned oe It attends to language, historical image and state tokens. Conversely, the [INV] token performs inverse dynamics prediction conditioned on the predicted visual foresight, attending to the input tokens and, crucially, the foresight token [FRS]. This special unidirectional attention mask in transformer encoder, highlighted in Figure 2 brings two benefits. First, this will help the [INV] token deeply integrate both past and future predictive information within multi-layer network. Second, this enables an end-to-end training paradigm through the fusion in the latent space.
Readout Decoders. Encoded by the multi-modal encoder, the action and image latents generated by the [INV] and [FRS] readout tokens are fed into the readout decoders to predict images and actions. The action decoder utilizes an MLP transform the action latent into the action vector dr. For image decoding, we employ a vision transformer (ViT) as the image decoder, following (He et al.) 2022). The image decoder takes the image latent along with masked tokens input. Processed by ViT. the output corresponding to each masked token represents a specific patch of the image.
3.4 IMPLEMENTATION DETAILS
Training. The training objectives, conditional visual foresight and inverse dynamics prediction, remain consistent between pre-training and fine-tuning. Notably, two key differences in model con- figurations exist between these phases. First, missing language instructions are common in robotic pre-training datasets. In such cases, during pre-training, the robot state token at the future time step + n + 1 acts as goal. The [FRS] would attend to it instead of the language token, ensuring [FRS] to acquire unambiguous information. Second, pre-training data may include random meaning- less behaviors, such as environmental exploration. Consequently the [INV] and [FRS] tokens do not attend previous image and robot state tokens to prevent overfitting to any specific behaviors.
Inference. During inference, the complete language instruction 1, robot states 8, and image obser- vations o are provided as inputs. The [FRS] token attends the historical image, state, and language instruction tokens to perform conditional visual foresight, predicting the future images. In turn, the [NV] token attends to the input tokens and one more foresight [FRS] token to perform inverse dynamics prediction, outputting the action. Further details are available in the Appendix.
Model. Throughout the training process, the pre-trained visual and text encoders are kept frozen, comprising total of 251M untrainable parameters. The rest components are fully trainable. The standard version of Seer possesses 65M trainable parameters. Additionally, we have scaled up the parameter size and developed a Leer-Large variant, which contains 315M trainable parameters. Un- less specified otherwise, the mention of Seer refers to the version with 65M trainable parameters.
SIMULATION EXPERIMENTS
We conduct experiments on two simulation benchmarks LIBERO-LONG (Liu et al. 2024), CALVIN ABC-D (Mees et al. 2022). Our aim is answer: 1) How does our method perform on challenging simulation benchmarks? 2) Does our pipeline maintain consistent effectiveness as the amount of downstream fine-tuning data varies? 3) Are the training objectives in Seer effective?
5

---

Published as a conference paper at ICLR 2025
Table I: LIBERO-LONG results. For each task, we present the average performance of top-3 checkpoints averaged over 20 rollouts. The metric "Avg. Success" measures the average success rate across ten tasks. Seer outperforms baselines with higher Avg. Success and better results on most tasks. The best results are bolded.
Put soup and hox in basket
Put box and butter in basket
Turn stove and put pot
Put bowl in drawer and close
Avg. Success
Method
MTACT
41.0
30.0
50.0
85.0
MVP
68.2
83.3
90.0
88.3
MPI
77.3
66.6
86.6
96.6
95.0
OpenVLA
54.0
35.0
95.0
65.0
45.0
SeeT (scratch)
78.7
80.0
90.0
91.7
81.7
Sccr
87.7
91.7
90.0
98.3
100
Put mugs left and right plates
Pick book and place back
Put mug plate and put
Put soup and sauce in basket
Put mug in microwave and close
Put both pots on stove
pudding to right
20.0
75.0
0.00
0.00
10.0
65.0
46.7
63.3
45.0
78.3
60.0
46.7
83.3
83.3
56.6
86.6
40.0
78.3
40.0
80.0
60.0
45.0
20.0
55.0
85.0
65.0
86.7
88.3
51.7
66.7
91.7
93.3
85.0
88.3
61.7
71.7
4.1
BENCHMARKS, BASELINES AND METRICS
Benchmarks, LIBERO-LONG (Liu et al. 2024 encompasses diverse object interactions and versa- tile motor skills. We pre-train our model on the LIBERO-90 dataset, which includes demonstrations for 90 short-horizon tasks with full annotations, and then fine-tune and evaluate it on LIBERO- LONG, which features long-horizon tasks. CALVIN ABC-D (Mees et al.) (022) is a benchmark focusing
visual robot manipulation. It contains 34 tasks across four dis- tinct environments (Env A, B. C, and D), each varying in object and scene visual appearance. For pre-training, we utilize the official robot play data with language instructions, while the remain- ing data with full annotations is used for fine-tuning.
iaggaag--onditionns
Baselines. For LIBERO-LONG, we implement vanilla multi-task policy MTACT without pre- training, general image-based pre-trained policy MVP (Xiao et al., 2022), video-based pre-trained policy MPI (Zeng et al.) (024)) and robot-data-based pre-trained policy OpenVLA (Kim et al. 2024). For CALVIN ABC-D, we select baselines that have demonstrated top competitive perfor- mance in prior reports. Roboflamingo (Li al. (023) is method stepped from a vision-language model (Alayrac et al. 2022). Susie BBackk et al. 2023) and CLOVER (Bu al.) 2024) are selected as the representative of two-stage PIDM methods. GR-I [Wu et al. 2024) enhances manipulation with generative video pre-training, while the 3D Diffusor Actor (Ke et al. 2024) specializes in capturing 3D representations to enhance manipulation.
Metrics. In LIBERO-LONG, each method is evaluated across 20 rollouts with varying initial states for each task. We report both per-task and average success rates. CALVIN ABC-D, the robot executes 1,000 task sequences, with each sequence comprising five consecutive tasks. We report the average success rates and the average length of completed sequences.
4.2 SIMULATION MAIN RESULTT
We conduct experiments on the LIBERO-LONG benchmark. As presented in Table D our policy achieved an average success rate of 78.7% without pre-training. After pre-training, the success rate increases by 9%, significantly outperforming the baselines. Compared MTACT, our policy more effective and benefits further from pre-training. The visual pre-training methods MVP and MPI achieve performance levels only comparable to our policy without pre-training, suggesting that visual pre-training alone is insufficient for manipulation tasks. Pre-training the entire policy using robotic data necessary to enhance both perception and decision- makinn capabilities. In comparison VLA, Seer (316M total parameter, 65M trainable parameter) uses only 4% of its parameters (7B) yet achieves a 62% relative improvement performance. These results underscore the advantages Seer and demonstrate the effectiveness of pre-training objectives.

---

Published as a conference paper at ICLR 2025
Table 2: CALVIN ABC-D results. We present the average success rales top-3 checkpoints computed over 1000 rollouts for each task and the average number of completed tasks to solve instructions consecutively (Avg. Len.). Seer shows consistent and significant superiority over baselines. The best results are bolded.
Task completed in row
Method
Avg. Len. 1
Robottamingo
82.4
61.9
46.6
33.1
23.5
2.47
Susie
87.0
69.0
49.0
38.0
26.0
2.69
GR-1
85.4
71.2
59.6
49.7
40.1
3.06
3D Diffusor Actor
92.2
78.7
63.9
51.2
41.2
3.27
CLOVER
96.0
83.5
70.8
57.5
45.4
3.53
Seer (scratch)
93.0
82.4
72.3
62.6
53.3
3.64
Seer
87.2
79.9
72.2
64.3
Seer-Large (scratch)
92.7
84.6
76.1
68.9
60.3
3.83
Seer-Large
96.3
91.6
86.1
80.3
74.0
4.28
We evaluate various methods on CALVIN ABC-D. Evaluation is conducted in Environment D, which differs visually from Environments A, B, and where the training data was collected. No- tably, the pre-training data lacks language annotations and includes meaningless actions and random explorations within these environments. As shown Table our method significantly outperforms the baselines. Our method surpasses the two-stage PIDM Susie (Wu et al. 2024) by large margin, and is markedly superior to CLOVER (Bu et al. 2024), probably due to our delicate model design and end-to-end training paradigm. It also outperforms the video generative pre-training method GR-1 (Wu et al.] 2024), demonstrating the advantage of end-to-end joint pre-training of visual pre- diction and action execution. Moreover, the results reveal that pre-training method delivers considerable gains even on datasets lacking language annotations and containing noise, highlighting its adaptability and flexibility. The Seer-Large variant achieves Avg. Len. of 4.28, establishing new state-on saate-of-hhe of-the-art on CALVIN ABC-D benchmark.
4.3
DATA EFFICIENCY
LIBERO-LONE
CALVIN ABC-D
Scaling Experiments
4.5
w/o pretrain
w/ pretrain
Success Rate (%)
75
4.0
Avg. Len.
Avg.
3.5
25
Previous SOTA
Previous SOTA
Ours (w/o Pre-training) Pre-training)
Ours (w/o Pre-training) Pre-training)
3.0
10
20
40
70
100
10
20
40
70
100
65M
107M
316M
Proportion (%)
Proportion (%)
Trainable Param.
Figure 3: Data efficiency and Scalability. The two figures the left depict Seer's performance LIBERO and CALVIN using different proportions of the downstream data. And the figure on the right illustrates Seer's performance CALVIN with various model sizes.
Collecting robot data both time-consuming and labor-intensive, making data efficiency crucial for robot learning. We evaluate our method on two benchmarks: LIBERO-LONE and CALVIN ABC- D, using 10%, 20%, 40%, 70%, and 100% of the available data to fine-tune pre-trained policies or to train policies from scratch. The results, shown in Figure 3 demonstrate that our method consistently enhances policy performance across varying data sizes. Notably, under data-scarce conditions with only 10% of the training data, the pre-trained policy achieves 187% relative improvement in suc- cess rate on LIBERO-LONG and a 150% relative improvement average task length on CALVIN ABC-D compared to training from scratch. Additionally, method only requires 70% data LIBERO-LONG and on CALVIN ABC-D respectively to surpass state-of-the-ee baselines. These results highlight the potential of Seer in scenarios with limited fine-tuning data.

---

Published as a conference paper at ICLR 2025
Table 3: Ablation studies on fine-tuning and pre-training objectives. Integrating the conditional visual foresight objective Liore and inverse dynamics prediction objective Live yields the best perfor- mance among pre-training and fine-tuning.
Fine-tuning objectives.
(b) Pre-training objectives.
Fine-tuning
Avg. Len.
Pre-train
Avg.
1
2
3
4
2
3
4
5
Ltore Lim
Liore
Lime
Len.
89.9
77.6
64.6
54.4
44.8
3.31
93.0
82.4
72.3
62.6
53.3
3.64
91.2
78.6
47.8
3,41
v
x
92.3
74.2
65.9
57.5
93.0
82.4
72.3
62.6
53.3
3.64
v
/
94.4
87.2
79.9
72.2
64.3
3.98
4.4 SCALABILITY
We evaluate Seer with different model size on the CALVIN ABC-D benchmark. Their results for the "Avg. Len." metrics are presented in Figure It is observed that, regradless of the involvement of pre-training, the performances improve with the increase in trainable parameters, showcasing the scalability of Seer.
4.5 ABLATION STUDIES
We investigate the contributions of conditional visual foresight objective Ltore and inverse dynamics prediction objective Liv during pre-training and fine-tuning on CALVIN ABC-D. The objectives during the fine-tuning phase are most closely related to performance in downstream tasks. Thus, prioritize ablating the fine-tuning objectives before examining the pre- pre-training objectives.
Fine-tuning objectives. We study the importance of Liore and Liny during fine-tuning. As shown in Table 3l. compared to the vanilla baseline, which directly performs behavioral cloning (wlo Ltoes w/o Cin).. separately predicting additional future images (w/ Liore. w/o Ciw) yields improvements. This indicates the benefits involving future image predictions (Wu et al. 2024). More importantly, integrating Liore and Lin results in further boost in performance. This demonstrates that utilizing visual expectations to guide action predictions is a more effective strategy for leveraging the rich visual and temporal information inherent in robot data than the ablated version (w/ Liore w/o Limv).
Pre-training objectives. Once the fine-tuning objectives (Lfor + Lime) are established, we start to ablate pre-training objectives. The results in Table indicate that pre-training only the vision pre- diction module (w/ Liore w/o Lim) yields certain benefits, probably due to the vision priors learned from the extensive data. Moreover, pre-training the whole policy (w/ Ltore. w/ Liw) through the inte- gration of visual foresight and inverse dynamics results in greater improvements. This underscores the importance of the synergy between action and vision priors distilled from large robot datasets in enhancing performance on downstream tasks.
5
EXPERIMENTS
We evaluated Seer six real-world tasks, including four focused generalization and two focused on high precision and rich contact. We aim to answer: 1) Is Seer effective in real-world tasks? 2) Does pre-training consistently improve performance under intense disturbances?
5.1 REAL-WORLD BENCHMARK
Real-world Setup. We evaluate Franka Research 3 robot with Robotiq-2f-85 gripper across six tasks, using two RealSense D435i cameras configured as Eye-on-Hand and Eye-on-Base Base for visual input. Four generalization-centrie tasks shown in Figure 4 and two gigh--recision and contact-rich tasks are presented in Figure A-3 in the appendix.
For pre-training, select the DROID dataset with Franka robots performing tasks in varied scenes. In the fine-tuning phase, we capture RGB images, robot states, and actions at Hz, collecting 100 demonstrations per task.
Datasets.
Baselines and Metrics. We benchmark against MVP (image-based pre-trained), MPI (video-based pre-trained), and Open VLA (rohot-data-based pre-trained). Each method is evaluated over 15 trials
8

---

Published as a conference paper at ICLR 2025
Eye-on-Hand RealSense D435I
Eye-on Eve-on-Base RealSense D435i
Robotiq-22 -opotiq-2f-85 Gripper
Flip White Bowl
(2) Stack Cups
Franka
Camera Hardware
Synchronization
Real-world Setup Objects
(3) Wipe Board
Pick, Close
Figure 4: Real-world Benchmark of four generalization-centii tasks. Left: We use a Franka Research 3 robot with a Robotiq-2f-85 gripper and RealSense D435i cameras. Right: We design four real-world manipulation tasks: (1) Flip White Bowl, (2) Stack Cups, (3) Wipe Board, (4) Pick, Place, Close. The detailed description of these tasks are included in A.6.T
per task, with variations in the initial states of the objects. Each method is allowed three executions per trial, with the mean performance reported. Given the long-horize and challenging nature of the tasks, we define two metrics: Success Rate (SR) and Score (as referenced in (Kim et al. 2024 ) The Score accumulates during the completion of specific intermediary stages, while SR recorded as 100% only upon successful completion of the entire task. Details are available in the Appendix.
5.2 REAL-WORLD MAIN RESULTS
Table 4: Real-world main results. We evaluate all the methods with 15 (cases) x 3 (repeated trials)
rollouts per task. Our method achieves better performances among all tasks than baselines.
Demos Per Task
SR (%: / Score
Method
Flip White Bowl
Stack Cups
Wipe Board
Pick, Place, Close
Avg.
MVP
100
80.0 24.0
26.7 26.0
53.3/38.0
60.0/31.0
55.0 29.8
MPI
100
66.7/21.0
29.0
33.3/35.0
66.7/32.0
29.3
VLA
100
53.3 19.0
0.00/ 8.00
0.00
13.3/13.0
16.7
Seer (scratch)
100
60.0 19.0
46.7 35.0
60.0 37.0
73.3 40.0
60.0/32..8
Seer
100
86.7 26.0
60.07 42.0
73.3/ 41.0
86.7 42.0
78.4/ 39.5
As can be seen in Table 4 our pre-trained policy could outperform all the baselines over all tasks. Specifically, our method improves the average success rate and the accumulated score from 60.0% to 78.4% and from 32.8 39.5 compared to the version trained from scratch. In comparison with MVP and MPI, which only pre-train vision encoders, our results reinforce the importance of pre-training the entire policy, aligning with findings from simulation experiments. Regarding the performance of OLe in the real world, it has significantly larger tunable model size (3B here) during full fine-tuning and relies solely on an eye-on-
camera. This could lead to severe overfitting and coarse action predictions, particularly in tasks where objects small (as in Stack Cups) located far from the camera (as in Wipe Board). contrast, our method demonstrates better handling of these tasks due to its moderate model size and comprehensive data utilization. We also conduct evaluation on two high-preciss and contact-rich tasks, with the results presented in Table A-IIl
5.3 ROBUSTNESS
We propose several generalization types assess the effectiveness of pre-trained policy across multiple settings. As shown in Figure 5] in Flip White Bowl, we introduce bowls of different col- alongside the original white bowl. These bowls share identical shape, size, and material, which

---

Published as a conference paper at ICLR 2025
Evaluation wio MUl Objects
Evaluation Mult Objects
Evolluatoo Clean Background
Evalustion Natural Background
Type
Method
SR (%) Score
Type
Method
SR (%) Score
Ours (w/o pre-train)
33.3 11.0
Ours (w/ pre-train)
6.67 / 13.0
Multi Objects
Background
Ours
60.0 / 18.0
Ours
/3.3/29.0
Evaluation w/ Novel Objects
Evaluation w Novel Objects
Evalustion wio Lightning
Evaluation w/ Lightning
Type
Method
SR (%) Scure
Type
Method
SR (%) / Score
Novel Objects
Ours (w/o pre-train)
46.7 37.0
Lightning
Ours (w/o pre-train)
46.7 30.0
Ours
60.0 39.0
Ours
66.7 37.00
Figure 5: Generalization evaluation. We design a generalization test per task with different dis- turbances. Top Left: In Flip Bowl, put several bowls with the same shape, size, material and different colors around the original white one. Top Right: Stack Cups, we remove the original black backdrop and keep the natural background. Bottom Left: In Wipe Board, we replace the chocolate balls with diverse novel small objects. Bottom Right: In Pick, Place, Close, we introduce an additional light source. Among all tests, our pre-trained method brings consistent benefits.
could potentially mislead the algorithm. In Wipe Board, we replace the original chocolate balls with novel objects that vary in mass, shape, and coefficients of friction, thereby increasing the task's difficulty. both scenarios, our pre-trained policy demonstrates significant improvements in suc- cess rate (SR) and Score. We attribute these enhancements to the extensive variety of interactable and distractible objects present in the pre- pre-training dataset, which strengthens the model's 's semantic understanding, Additionally, in Pick, Place, Close, incorporate strong light source that alters the visual appearance of objects in RGB images. In Stack Cups, we remove the clean black back- drop and replace it with natural background, introducing complex disturbances such variable lighting, unseen distractions, and effects on camera exposure. Even under these challenging condi- tions, our pre-trained policy continues deliver satisfactory results. These results demonstrate that extensive pre-training on large-scale robot datasets with diverse scenes enhances robustness.
6 CONCLUSION AND LIMITATIONS
In this work, introduce Seer, end-to-end predictive inverse dynamics model that synergizes conditional visual foresight with inverse dynamics prediction for manipulation. Seer achieves state- of-the-art results in real-world experiments after pre-training on the large DROID robot dataset. The limitations mainly lie in two aspects. Firstly, only evaluate six downstream tasks. A broader spectrum of high-precision and contact-rich tasks remains to be explored. Secondly, evaluating across different robots is also necessary to test Seer's cross-ent eooss-embod capability.
simulation benchmarks, and shows significant improvements and robustness
ACKNOWLEDDGENNTS
This work is supported by the National Key R&D Program of China (2022DD0160201), Shanghai Artificial Intelligence Laboratory, and China Postdoctoral Science Foundation (2023M741848),
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Pulkit Agrawal, Ashvin Nair, P. Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physies. In NeurIPS, 2016.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurlPS, 2022.
Shikhar Bahl, Russell Mendonca, Lili Chen, Unnat Jain, and Deepak Pathak. Affordances from human videos as a versatile representation for robotics. In CVPR, 2023.
Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmani. Gen2act: Human video generation in novel scenarios enables generalizable robot manipulation. In Arxiv, 2024.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. In NeurlPS, 2023.
David Brandfonbrener, Ofir Nachum, and Joan Bruna. Inverse dynamics pretraining learns good representations for multitask imitation. In NeurlPS, 2024.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Ri-1: Robotics transformer for real-world control at scale. In RSS, 2022.
Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Yi Ma, and Hongyang Li. Closed-loop visuomotor control with generative expectation for robotic manipulation. In NeurIPS, 2024.
Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang. Li Zhao, and Jiang Bian. Igor: Image-godl representations the atomic control units for foun- dation models in embodied ai. In ArXiv, 2024.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. RSS, 2023.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018.
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, and Chelsea Finn. Robonet: Large-scale multi-robot learning. In Arxiv, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In NeurIPS, 2024.
Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. In Arxiv, 2024.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Sueller-Freitag et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022.

---

Published as a conference paper at ICLR 2025
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Eric Jang, Alex Irpan, Mohi Khansari, Daniel Kappler, Frederik Ebert, Corey Lynch, Sergey Levine, and Chelsea Finn. Bc-z: Zero-shot task generalization with robotic imitation learning. In CoRL, 2022.
Siddharth Karamcheti, Suraj Nair, Annie S Chen, Thomas Kollar, Chelsea Finn, Dorsa Sadigh, and Percy Liang. Language-driven representation learning for robotics. In RSS, 2023.
Tsung- Boang-Wei&uee Ke, Nikolaos Gkanatsios, and Katerina Fragkiadaki. 3d diffuser actor: Policy diffusion with 3d scene representations. In CoRI,, 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. In RSS, 2024.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-langunge-action model. In CoRL. 2024.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vision-language foundation models as effective robot imitators. In ICLR, 2023.
Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In NeurlPS, 2024.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In ICLR, 2022.
Yecheng Jason Ma. Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In JCML, 2023.
Ajay Mandlekar, Jonathan Booher, Max Spero, Albert Tung, Anchit Gupta, Yuke Zhu Animesh Garg, Silvio Savarese, and Li Fei-Fei. Scaling robot supervision hundreds of hours with robo- turk: Robotic manipulation dataset through human reasoning and dexterity. In IROS, 2019.
Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. RA-L, 2022.
Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: univer- sal visual representation for robot manipulation. In CoRL, 2022.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexan- der Khazatsky, Anant Rai, Anika Singh, Anthony Brohan, Antonin Raffin, Ayzaan Wahid, Ben Burgess-Limeriel Beomjoon Kim, Bernhard Schölkopf, Brian Ichter, Cewu Lu, Charles Xu, Chelsea Finn, Chenfeng Xu, Cheng Chi, Chenguang Huang, Christine Chan, Chuer Pan, Chuyuan Fu, Coline Devin, Danny Driess, Deepak Pathak, Dhruv Shah, Dieter Büchler, Dmitry Kalash- nikov, Dorsa Sadigh, Edward Johns, Federico Ceola, Fei Xia, Freek Stulp, Gaoyue Zhou, Gau- rav S. Sukhatme, Gautam Salhotra, Ge Yan, Giulio Schiavi, Hao Su, Haoshu Fang. Haochen Shi, Heni Ben Amor, Henrik I Christensen, Hiroki Furuta, Homer Walke, Hongjie Fang, Igor Mor- datch, Ilija Radosavovic, Isabel Leal, Jacky Liang, Jachyung Kim. Jan Schneider, Jasmine Hsu, Jeannette Bohg, Jeff Bingham. Jiajun Wu, Jialin Wu, Jianlan Luo, Jiayuan Gu, Jie Tan, Jihoon Oh, Jitendra Malik, Jonathan Tompson, Jonathan Yang, Joseph J. Lim, João Silvério, Junhyek Han, Kanishka Rao, Karl Pertsch, Karol Hausman, Keegan Go, Keerthana Gopalakrishnan, Ken Gold- berg, Kendra Byrne, Kenneth Oslund, Kento Kawaharazuka, Kevin Zhang, Keyvan Majd, Krishan Rana, Krishna Parasuram Srinivasan, Lawrence Yunliang Chen, Lerrel Pinto. Liam Tan, Lionel Ott, Lisa Lee, Masayoshi Tomizuka, Maximilian Du, Michael Ahn, Mingtong Zhang, Mingyu Ding, Mohan Kumar Srirama, Mohit Sharma, Moo Jin Kim, Naoaki Kanazawa, Nicklas Hansen, Nicolas Manfred Otto Heess, Nikhil J. Joshi, Niko Suenderhauf, Norman Di Palo, Nur Muham- mad Mahi Shafiullah, Oier Mees, Oliver Kroemer, Pannag R. Sanketi, Paul Wohlhart, Peng Xu,
12

---

Published as a conference paper at ICLR 2025
Pierre Sermanet, Priya Sundaresan, Quan Ho Vuong, Rafael Rafailov, Ran Tian, Ria Doshi, Rus- sell Mendonca, Rutav Shah, Ryan Hoque, Ryan C. Julian, Samuel Bustamante, Sean Kirmani, Sergey Levine, Sherry Moore, Shikhar Bahl, Shivin Dass, Shuran Song, Sichun Xu, Siddhant Haldar, Simeon Adebola, Simon Guist, Soroush Nasiriany, Stefan Schaal, Stefan Welker, Stephen Tian, Sudeep Dasari, Suneel Belkhale, Takayuki Osa, Tatsuya Harada, Tatsuya Matsushima, Ted Xiao, Tianhe Yu, Tianli Ding, Todor Davchev, Tony Zhao, Travis Armstrong, Trevor Darrell, Vidhi Jain, Vincent Vanhoucke, Wei Zhan, Wenxuan Zhou, Wolfram Burgard, Xi Chen, Xiaolong Wang, Xinghao Zhu, Xuanlin Li, Yao Lu, Yevgen Chebotar, Yifan Zhou, Yifeng Zhu, Ying Xu, Yixuan Wang, Yonatan Bisk, Yoonyoung Cho, Youngwoon Lee, Yuchen Cui, Yueh hua Wu, Yu- jin Tang, Yuke Zhu, Yunzhu Li, Yusuke Iwasawa, Yutaka Matsuo, Zhuo Xu, and Zichen Jeff Cui. Open x-embodiment: Robotic learning datasets and rt-x models. In ICRA, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Ilija Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real-world robot learning with masked visual pre-training. In CoRL, 2023.
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In CoRL. 2023.
Achint Soni, Sreyas Venkataraman, Abhranil Chandra, Sebastian Fischmeister, Percy Liang, Bo Dai, and Sherry Yang. Videoagent: Self-improving video generation. In ArXiv, 2024.
Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang. and Ashish Kapoor. Smart: Self-supervised multi-task pretraining with control transformers. In ICLR, 2023.
Homer Rich Walke, Kevin Black, Tony Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In CoRL, 2023.
Boyang Wang, Nikhil Sridhar, Chao Feng, Mark Van der Merwe, Adam Fishman, Nima Fazeli, and Jeong Joon Park. This&that: Language-gess gesture controlled video generation for robot planning. In ArXiv, 2024.
Yao Wei, Yanchao Sun, Ruijie Zheng, Sai Vemprala, Rogerio Bonatti, Shuhang Chen, Ratnesh Madaan, Zhongjie Ba, Ashish Kapoor, and Shuang Ma. Is imitation all you need? generalized decision-making with dual-p training. In CVPR, 2023.
Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. In ICLR, 2024.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. In Arxiv, 2022.
Jia Zeng, Qingwen Bu, Bangjun Wang, Wenke Xia, Li Chen, Hao Dong, Haoming Song, Dong Wang, Di Hu, Ping Luo, et al. Learning manipulation by predicting interaction. In RSS, 2024.
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 IMPLEMENTATION DETAILS
In this section, we outline the implementation details of our framework.
Vision. We employ MAE-pretrained ViT-B (He et al., 2022) as the vision encoder. At each timestep, images are captured from two views: eye-on-hand and eye-on-base. Each image is pro- cessed by the vision encoder to produce 196 latent vectors, which represent local patch information, along with [CLS] token that encodes the global representation of the image. Directly inputting all 197 tokens into the transformer backbone would create a significant computational burden, particu- larly when processing long histories. Moreover, many image details redundant for accomplish- ing manipulation tasks. To address this, we utilize the Perceiver Resampler (Alayrac et al., 2022) to condense the image representations and extract task-relevant features. The Perceiver Resampler employs learnable latent vectors with a shape of latentsents. dim), where num num_latents is signifi- cantly smaller than the number of image tokens. Through Perceiver Attention, these latent vectors condense the input image features, along with the [CLS] token, to form the final image tokens.
Robot state. The robot state consists the arm state and the gripper state. The arm state includes the end-effector position and its rotation in Euler angles, resulting in a six ix-dimensional representation. The gripper state is binary value indicating whether the gripper open or closed. We tokenize the robot state using MLP. Specifically, the gripper state is first converted into one-hot encoding. The one-hot encoding of the gripper state and the arm state are then each passed through separate linear layers. The outputs are concatenated and passed through final linear layer to produce the state token.
Language. Text instructions are encoded using the CLIP ViT-B/32 text encoder (Radford et al., 2021) and projected through a linear layer to generate the language token.
Readout tokens. At each timestep, we append [FRS] and [LNV] tokens to read out foresight and actions. Specifically, [F [SJ tokens are appended to extract representations for two views, and three [INV] tokens are appended to predict actions across three steps, ensuring temporal action consistency and robustness to idle actions.
Decoder. After passing through the transformer backbone, the action and image latents generated by the [INV] and [FRS) tokens are input to the action decoder and image decoder to predict actions and images for conditional visual foresight and inverse dynamics prediction.
The action decoder is an MLP that decodes the action latent into a seven diven-dimensional action vector. First, the action latent is processed by linear layer followed by ReLU activation function. Then, it passes through a second linear layer with ReLU activation. The output is fed into two independent decoders: the arm action decoder and the gripper action decoder. The arm action decoder maps the high-dimensional vector to a six- dixeemmnnional output through a linear layer, applying a Tanh activation function to constrain the arm action within the range [-1, 1]. The gripper action decoder also employs linear layer to map the latent vector to a one-dimensional output, applying a Sigmoid activation function to constrain the gripper action between [O, 1]. A gripper action value of 0.5 or higher is interpreted as (closed), while value below 0.5 is interpreted as 0 (open).
For image decoding, following (He et al.) 2022), we use a vision transformer (ViT) as the image decoder. The image decoder receives the image latent and mask tokens from the transformer back- bone as input. Positional information is provided through fixed sine-cosine positional encodings. The inputs are processed by multiple transformer encoder blocks. Finally, linear layer predicts the pixels for each patch, generating the image that represents the predicted future state.
We present relevant hyperparameters during both pretraining and finetuning in Table A-1
The standard version of Seer contains 316M parameters, where only 65M is tunable. For all simu- lation results, about 40 hours for CALVIN ABC-D and and 30 hours for LIBERO-LONG, The fine-tuning process requires about 24 hours for CALVIN ABC-D and 6 hours for LIBERO-LONG.
eight 4090 GPUS to pre-train and fine-tune. The pre-training process requires
14

---

Published as a conference paper at ICLR 2025
Table A-l: Training hyperparameters.
Hyperparameters
Pre-training
Fine-tuning
Batch Size
640 (LIBERO & CALVIN) / 2048 (Real)
512
Learning Rate
le.4
le-3
Optimizer
AdamW
AdamW
Learning Rate
Cosine decay
Cosine decay
Schedule
Training Epochs
30 (LIBERO & Real) / 20 (CALVIN)
40 (LIBERO Real) / (CALVIN)
History Length
(LIBERO Real) (CALVIN)
(LIBERO & Real) 10 (CALVIN)
Action Chunk
Length
Transformer Block Output
VIT Base
Percoher Resampler
Dropout
MLP
Ann Action
Gripper Action
Layer Normalization
FC6
FC1
VIT
0.59]
Dropout
Gripper [Q.00
CCRR
24 x GPT-2 Transformer Blocks
Multi-Head Masked Attention
Language Instruction
X CLI Vir-a/2
Layer Normalization
x N
[MV
(FRS)
Transformer Block Input
Figure A-1: Network Architecture.
Table A-II: Hyperparameters for the transformers our policy.
Hidden size
Numher of layers
Number of heads
image encoder
768
12
12
perceiver resampler
768
3
transformer backbone
384
24
image decoder
384
2
16
15

---

Published as a conference paper at ICLR 2025
A.2 NETWORK ARCHITECTURE
As presented Figure A-1] Seer consists the following modules: image encoder, perceiver re- sampler, robot state encoder, language encoder, transformer backbone, action decoder and image decoder. Here we describe each module in detail:
image encoder: a MAE pre-trained ViT-Base (He et al. 2022 encoder. Details can be seen in Table [-Il
perceiver resampler: a module to compress the image tokens efficiently. Details can be seen in Table A-I
robot state encoder: MLPs that encode robot states into latent space.
language encoder: CLIP (Radford et al. 2021) VIT-B/32 text encoder.
transformer backbone: 24 layers of GPT-2 transformer blocks, with hidden size of 384 and 12 heads. It takes image tokens, language tokens, robot states tokens, [INV], [FRS] as inputs. Details can be seen in Table A-Il
action decoder: MLPs that decode the latent feature into 7-DOF action.
image decoder: ViT-based transformer followed by a linear layer. Details can be seen in A-Il
A.3 BASELINE IMPLEMENTATION
In the simulation benchmark, we report the scores for Roboflamingo, Susie, GR-I, and the 3D Diffusor Actor from their respective papers. For MTACT and OpenVLA, we reproduce the results using the official code. For MVP and MPI, we replace the vision encoder in our policy with their pretrained versions. Thanks to the strong design of our policy, MVP and MPI show competitive performance, though they only approach the results of our policy without pretraining.
A.4 LIBERO-LONE EXPERIMENT DETAILS
LIBERO (Liu et al. 2024 is novel benchmark for lifelong learning in robot manipulation, com- prising four task suites: LIBERO-SPATIAL LIBERO-OJJECT, LIBERO-GOAL, and LIBERO- 100. The first three task suites designed to disentangle the transfer of declarative and proce- dural knowledge, while LIBERO- 100 consists 100 tasks involving entangled knowledge trans- fer. LIBERO- 100 includes 100 tasks that require diverse object interactions and versatile motor skills. LIBERO- 100 divided into 90 short-horizon tasks (LIBERO-90) and 10 long-horizon tasks (LIBERO-LONG), We use LIBERO-90 the pretraining dataset, while LIBERO-LONG is utilized for the downstream finetuning and evaluation.
The policy use images from both fixed and gripper cameras observe the environment, which were resized to 224x224 pixels. We also incorporated the robot state to help the policy understand the robot's self-state, including the position and orientation of the end effector and the gripper state indicating the width between the grippers. The action space consists of a -vvenddimesiionl vector: six dimensions represent arm actions, and one dimension indicates the gripper's open/close state. The arm action represents the 61) pose (position and orientation) of controlled frame. This frame located between the fingers of robots.
A.S CALVIN ABC-D EXPERIMENT DETAILS
2022 is simulated benchmark designed for learning long-horizon, language- conditioned tasks. goal is to enable the development of agents capable of solving various robotic manipulation tasks using only onboard sensors and instructions provided in natural human language. The tasks in CALVIN are complex, involving long sequences and intricate language instructions. supports flexible configurations of sensor suites. Agents evaluated in zero-shot manner novel language instructions and unfamiliar environments.
CALVIN (Mees et al.
The CALVIN benchmark includes four distinct but structurally similar environments-Emt A, B, and D. Each environment features a Franka Emika Panda robot arm equipped with a parallel gripper, as well desk with sliding door and drawer that can be opened and closed. And
16

---

Published as a conference paper at ICLR 2025
there several objects, such blocks and buttons, the desk. To more effectively assess the generalization of the learned policies, each environment features distinct textures, and objects like the sliding door, drawer, and light button, placed in different positions.
CALVIN offers rich observations for robot learning. We use images from both fixed and gripper- mounted cameras, resized to 224x224 pixels, along with robot state information, which includes end-effector position, orientation, and gripper state (open/close). The action space is 7-dimensional vector: six dimensions correspond to end-effector displacement (position and orientation), and one dimension controls the gripper's open/close state. The unstructured data includes exploratory and sub-optimal behaviors, comprising approximately 2.4 million interaction steps and million short- horizon windows. Data from Env A, B, and C, which lacks language annotations, is used to pretrain the policy, while data with language annotations used for downstream task learning. Env D reserved for policy evaluation.
A.6 REAL WORLD EXPERIMENT DETAILS
A.6.1 GENERALIZATION CENTRIC TaSKS
Task Progress
Flip White Bowl
Stack Cups
Pick, Place, Close
Wipe Board
Figure A-2: Progress and configuration of generalization-centii tasks.
Tasks. Flip White Bowl, the robot picks up randomly overturned bowl and places on a coaster, testing basic 6DoF pick & place capability. In Stack Cups, the robot stacks three randomly placed cups of different colors and sizes. It firstly covers the small cup with the middle one, and then covers the middle cup with the big one. Since the cup surface smooth and covering requires close fit, this task challenges fine-grained action predictions. In Wipe Board, the robot collects 3 to 7 chocolate balls arranged in
clusters. uses brush to sweep the balls into dustpan, testing its ability to handle multi-modal settings and perform repetitive motions. In Pick, Place, Close, the robot picks randomly set carrot, transports it into an opened drawer and closes the drawer. This evaluates skills of executing consecutive actions in large space with articulated objects.
the robot needs Opick an overturned bowl and Oplace it on the coaster. In this task, the white bowl is randomly placed on the table within 40cm x 40cm square, and the coaster is randomly placed within a 15cm x 15em square. The robot needs to pick up the white bowl and it on the coaster. In the generalization test, to bowls with identical shapes, sizes and materials are added to disturb the policy. Success rate (SR) is recorded as 100% only when the white bowl is placed on the coaster safely. If the bowl is successfully grasped, the score will plus one (+1). If the bowl successfully placed on the coast, the score will also plus one (+1). The full score in this task is 2.
Flip White Bowl:
17

---

Published as a conference paper at ICLR 2025
Stack Cups: the robot needs Cpick the middle cup, @cover the small one, Opick the big one, and Øeover the middle one. this task, three cups of different sizes are randomly placed on the table within 40cm 40cm square. The robot needs to cover the small cup with the middle one, and cover the middle cup with the big one. Only when all the cups are stacked precisely and in correct order will the SR be recorded 100%. The score will plus one (+1) when each primitive action (pick or place) is accomplished. The full score in this task is 4.
the robot needs @pick the carrot, Oput in the drawer, and @elose the drawer. In this task, a drawer with three layers is fixed on the table. During each test, one of three layers is open. A carrot on the coaster randomly placed within a 20cm 20cm square. The orientation of the carrot is randomized. The robot needs to pick the carrot, place it into a certain layer, and close the drawer. The score will plus one (+1) when successfully (1) picking the carrot, (2) placing the carrot, and (3) closing the drawer. The full score in this task is 3.
Pick, Place, Close:
the robot needs Ograsp the brush, and sswee all the chocolate balls into the dustpan. In this task, board (30cm x 40cm) and dustpan fixed on the table. A brush is randomly placed within a 5cm x 5em square. 3 to chocolate balls (diameter Icm) are divided into I to 3 clusters and randomly placed on the whole board. Only when all the chocolate balls are swept into the dustpan safely will the SR be recorded as 100%. The score will plus one (+1) when (1) grasping the brush successfully, (2) sweeping partial chocolates into the dustpan, and (3) sweeping all chocolates into the dustpan. The full score in this task is 3.
Wipe Board:
A.6.2 HRGHIPRRCISION AND CONTACT-RICH TASKS
Task Progress
Objects
Press Button
zcm
2.8cm
2cm
Insertion
9cm
8.5cm
Figure A-3; Process and configuration of sigh-precision and contact-rich tasks.
Table A-llI: Results on high-precision and contact-rich tasks.
Demos
Press Button
Insertion
Method
Per Task
SR (%) 1 Score 1
SR (%) 1 Score
MVP
46.7 17.0
26.7 11.0
Ours (w/o pre-train) Ours
100
40.0 / 13.0
40.0 / 16.0
100
60.0 / 18.0
60.0 19.0
In this task, the toaster is randomly placed in a 30cm x 30cm square. The robot is required to approach the toaster, close the fingers, push the button from top-down view, and exceed 3/4 of the scale (Figure A-3). The score will plus one (+1) when (1) pushing the button successfully with collision, and (2) exceeding 3/4 of the scale. The full score in this task is 2.
Press Button:
Insertion: this task, a 2cm x 9cm camera model is randomly placed a 20cm x 20cm square. The robot is required to pick the camera model and insert into 2.8em 9.5cm groove without
18

---

Published as a conference paper at ICLR 2025
any collision (Figure A-3 The score will plus one (+1) when (1) grasping the camera model, and (2) inserting successfully with no collision. The full score in this task
Results: As seen in Table A-Ill, pre-training the DROID dataset brings obvious improvements, compared to the scratch version and previous state-of-the-ar baselines. Notably, both tasks require quite precise action predictions and collision-free interactions, showing model's potential in high-precision and contact-rich tasks.
A.6.3 REAL-WORLD IMPLEMENTATION DETAILS
During real-world training, we the sequence length as 7, visual foresight steps and action pre- diction steps as 3. We utilize the MAE pre-trained ViT-B encoder and configured it as bfloat16 to accelerate inference. We find this quantization won't produce side effects on manipulation tasks. The pre-training dataset, c.g., DROID involves 76K successful trajectories, and the downstream fine-tuning data involves 400 demos. Key hyperparameters are listed Table A-I We utilize the 9-th pre-trained checkpoint for fine-tuning, and evaluate performance using the 17-th fine-tuned checkpoint.
In real-world experiments involving the MVP and MPI baselines, replace the MAE pre-trained vision encoder our network with the MVP pre-trained and MPI pre- pre-trainn trained counterparts, respec- tively. We then fine-tune these baselines on the downstream tasks and report performances. For VLe, we choose to fully finetune its public released checkpoint with model size 7B pre-trained on OXE. The fine-tuning config is identical to the one trained on Bridge dataset in VLe'S pub- lic codebase. We fine-tune this large model using eight A100 GPUs with more than 24 hours and use the checkpoint with lowest average validation loss to evaluate.
A.6.4 ACROSS EMBODIMENTS EXPERIMENTS
To investigate the impact of Seer cross-embodiment, conduct additional real-world exper- iment, where Seer is pre-trained on OXE and fine-tuned on self-collected data. We refer the subset mix-up recipe in Octo (Ghosh al. 2024), remove all the subset that includes franka robots, filter subsets with odd action labels, and save the rest subsets as OXE pre-training dataset.
Table A-IV: Results of across emhodiments experiments.
SR (%) / Score
Method
Flip White Bowl
Stack Cups
Wipe Board
Pick. Place. Close
Press Butten
Insertion
Avg.
Seer (scratich)
60.0/ 19.0
46.7/33.00
60.0 / /7.0
73.3 1.3/44..0
40.0/ 13.0
40.0 16.0
53.3/ 26.7
Seer (OXE)
73.3/22.00
40.0/ 30.0
66.7 40.0
80.0 41.0
3 10.0
18.0
/6.7/26.8
Seer (Droid)
86.7 26.0
73.3/ 60.0/42.0
86.7/42.0
60.0/ 18.0
60.0 19.0
71.1 71.1/313
As seen in the Table A-IV] pre-training the OXE dataset only provides marginal improvements in most tasks. Moreover, in some high nigh-precision tasks, the OXE pre-trained version even brings negative effects. We attribute marginal improvements in general manipulation tasks the diversity of objects, tasks, scenes, and language instructions in OXE. For the slight decrease some high- precision tasks, suspect two reasons: Firstly, for hand camera (wrist view) are quite important, since this camera captures a lot of local and detailed information for policy. However, in most OXE subsets, only images from third-person view with diverse camera poses are provided, with most wrist view images missing. Secondly, the gap in cross- embodiments and cross-action-conrolle a concentrated and precise action distribution. However, pre-training OXE may offer a distant action prior due to the aforementioned physics gap.
tasks, images from the eye-on-
types exists in our settings. High-precision tasks require
A.6.5 DETAILED REAL-WORLD RESULTS
The records of the real-world experiments are shown in Table A-V Table A-VI Table A-VII and VillI[A-VIII which we use to calculate the success rate and score.
19

---

Published as a conference paper at ICLR 2025
Table A-V: Detailed results (SR (%) Score) in Flip White Bowl.
Case Index
MVP
MPI
OL
Ours (wlo yee Irain, 20 demus)
Ours (20 demos)
Outs (wrl pre train. 100 demus)
Ours (100 demos)
0.00/0:30
100/ 2,00
0.00/ 1.00
0,000/ 1.00
2.00/2.00
1.0/2.00
100/2.00
2,00
100/ 2.00
100/2.00
0.00/ 0.0
100 2.00
100/2.00
100/2.00
100/2.00 /,00
100/0200
1.00
0,00/0.0
100/2.00
100/
100/
0.00/0.00
100/2.00
0,00/0.0
100/2.00
100/2/00
5
2.00
0.00/0.00
200/2000
100/ 2.00
100/ 2.00
0.00
/00/2.00
6 100/ 2,00
100/2.00
200
0.00/00
0.00/00
0.00/0.00
100/2.00
7
2,00
1000/2.00
0.00/ 1.00
0.00/0.0
0.00/0.0
100 2.00
100/ 2.0
8
100/ 2.00
100/ 2.00
100/
200
100/2.00
0.001 1,00
100/
9
1003 2.00
1001 2.00
100/ 2.00
100/
100/2.00
100 2.002.00
100/ 2.00
10
100/ 2,00
2.00
0.00/0.00
0.00/000
0
1.0/2.00
0,00/0.00
11
2,00
0,0070.00
0.00/0.00
0.00/00
100/ 2000
0
100/200
12
0.000
0:00/000
100/2.00
100/2.00
0:00/ 0.00
13
2,00
100/2.00
100/
0.00/0.
0.007
0:00/000 0.00
100/2000
14
100/ 2,00
2.0/2.00
0.00/0.00
0.00/ 1.00
100/
1.0/2.00
100/2.00
15
0.00/
0.00
0.00 0.00
200/2.00
0.00/ 0.0
0.00/0.0
100 / 2.00
100/ 2.00
Table A-VI: Detailed results (SR (%) / Score) in Stack Cups.
Case Index
MVP
MPI
VL
Ours (wl pre Irain, 20 demos)
Ours (20 domos)
Ours (wrl train. 100 domos)
Ours (100 domos)
0.00/3..0
100/ 1.00
0:00/
01.00 0.00
0 1.00
100/4.00
100/4.00
2
100/ 4.00
3.00
0:00/ /0.00
13.00/0.00
0.00/0.00
0.001 1.00
100/4.00
0.00/
/3.00/3.00
0.00/0.00
(1000 1.00
2.00
100/4.00
0:00/1000
100
100/4.00
1.30
0.00
0.00/ 2.00
1.00
6 0.00/1 1,00
1.0/4.00
0.00/ 0.00
0.00/ 1.00
0.00/2.00
100 100/0.00
100/4.0
7 0.00/ 0.300
4.00
0.00/
0.00/ 1.00
0.00/ 0.00/000
0.00/ 1.00
100/
8
0.00/ 0:00
0.00
0.00/ 1.00
0.00/0.00
0.00/0.00
0.00/0.00
0.00/0.00
9
1000
4007
0,00 0.00
4.00
0 0.00
100 100/4.00
100/4.00
10
0.00/
1,00
0
0.00/0.00
0.00
0.00/0.00
0.00/0.00
2.00
11
0.300
01000 1.00
0.00/0.0
0.00/1.00 1.00
0.001
0.00/0000
/0,0
12
0.00/0.00
13
0.90
0:00/1.00
0.00/0.00
0.00
14
0.00/1 1,00
0.00
0.00
0.00/ 1.00
0.00
0.00/ 1.00
0.00/0.00
100/4.0
15
1007
4.00
0.000 1.00
0.00/ (1.00
0.000 1.00
0.00/ / 1.00
100/ 4.00
100/400
Table A-VII: Detailed results (SR (%) / Score) in Pick, Place, Close.
Case Index
MVP
MPI
OpenVLAA
Ours (wlo pro-train, 20 demoss
Ours (20 demos)
Ours (wio pre me-train. 100 demos)
Ours (100 demos)
1.003
1001 3.00
0:00/0.00
100/33.00
130/3.00 3.00
100/ 3,00
100/ 3.00
2
3.00
100 3.00
100/ /300
0.00/0.00
100 3.00
1.0/3.00
100/3.00
3
3.00
3.00
0.00/0.00
/2.0/
100/3.00
0,00/2.00
100/3.00
3.00
/0.0
0,00/ 2.00
100/3.00
00.30
3.00
0:001 2.00
100/3.00
100/3.00
3.00
3000
6 11:00
0.00/0.00
0:00/
0.00 1.00
0.00/
0.80/ 2,00
3.003000
7
/3,00
1.00
0,00/ 0.000
0.00
100/
1.0/3.00
100/3.00
8 100/ 3,00
100
0 0.00
1.00
100/3.00
100 /300
100/3.00
9 0.00/1
/1.00
0.00
100/3.00
0.00
0,00/0.00
0.00/ 1.00
0 2.00
10 1.00
100/
0:00/0.00
0.00/0000
0.00/ 2.00
100/ 1.003.00
100/3.00
11
100/ 3.00
0.00
0.00/ 1.00
0.00
0.00/ /
3.0/
100/3.00
12
1.00
130/3.00
13
100/ 3.90
3.00
0.00
100/3.00
100/3.00
3.00
100/300
14
100/
3,00
3.0/
0:001 2.00
0.00/0000
0.00/ 0.00
100 0 5.00
100/
15
0.00/
1,00
0.00 1.00
0,00/0.00 0..00
0.00
100/3.00
100 /3000
100/3.00 3.00
Table A-VIII: Detailed results (SR (%) Score) in Wipe Board.
Case Index
MVP
MPI
OpnnVLA
Ours (wlo pcu- Irain, 20 demuu)
Ours (20 demos)
Ours (wrl pre train. 100 demus)
Ours (100 demos)
100/3.00
1.0/3.00
0..0/0.00
0.00/0000
0 2.00
100/3.00
100/3.00
2
3.00
1.003.00
0.00/000 00.00
0.00/2.00
100 1.0/3.00
0.00/2.00
100/3.00
/300
1.0/3.00
0.00/
100 10003.00
100/
0,00/000
100/3.00
100/ 3.00
0.00/2.00
0.00/0.00
1.00
0.00/2.00
100/5.00
100/3000
0.00
0:00/1000 1.00
0.00/2000
100/3.00
100/3000
6 0.00/2.00
0.00
0.00 /0.0
0.00/2000
0.00/2.00
0.00/ 2.00
0.00/2.00
7 0.00/ 2.00
0.00/2000
0.00/ 0.00/000
0.000 1.0
0.00/1.00 1:00
0.00/ 2,00
100/3.00
8
0.00/2.000 2.00
0.00
0.00/0000
0.00
0.00 2.00
180 /.00
0.00/2.00
9 100 3.00
3.00
0.00/0.00
1.0/3.00
100
100 3300
100/3.00
10 0.00/2.00
0.00/2.00
0.00/0000
1.0/3.00
0,00 0.00
0.00/2.00
100/3000
11 1300
0.00
0.00/000
0.00/2000
0.00
3.00
100/
12
100/
/
0.00/0.00
0.0000000
0.00/2.00
100/5.00
0.00/2000
13
0.00/
1.0/3.00
/0.00
100/3.00
1.0/3300
1.0/
300/300
14 0.00
0.00
0.00/2.00
0.00/2000
0.00/2000
0.0022000
0.00/200
/.000 15
0.00
0,00 0.00
0.00/ 0.00
0.00/2.00
/
100/ 3.00
20


Reviews:
[{"review_id": "QO2L5iTJWm", "reviewer": "Reviewer_zoME", "summary": "This paper introduces an end-to-end framework, Predictive Inverse Dynamics Models (PIDM), which predicts actions using inverse dynamics models conditioned on the robot’s forecasted visual states. By integrating vision and action in a closed-loop system, the end-to-end PIDM serves as a scalable and effective action learner. The approach demonstrates improved performance over state-of-the-art methods, both in simulations and on a real robot.", "questions": "1. The joint prediction of inverse dynamics (actions) and forward dynamics (next observation) reminds me of RPT [1], where it performs random masking of action and latent representations of observations and pre-trained via mask reconstruction. The key difference between this approach and RPT is that RPT doesn’t predict exact forward dynamics due to the random mask pattern, so it can learn shortcuts by copying from other future frames. Is there some way to evaluate the importance of predicting forward dynamics? Specifically, for table 3(b): is it possible to pre-train with only inverse dynamics?\n2. For finetuning the model, are all weights considered, or is LoRA applied to the model? It would be interesting to know how this method's performance scales with the amount of parameters used for fine-tuning. \n\n[1] Radosavovic, Ilija, Baifeng Shi, Letian Fu, Ken Goldberg, Trevor Darrell, and Jitendra Malik. \"Robot learning with sensorimotor pre-training.\" In Conference on Robot Learning, pp. 683-693. PMLR, 2023.\n\n[2] Hu, Edward J., Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Thorough ablation studies and experiments that demonstrate the effectiveness of jointly predicting forward and inverse dynamics as a pre-training task for robot learning\n2. Evaluation in both simulation and in real on the effectiveness of the method\n3. Shows scalability of the proposed method by pre-training on DROID.", "weaknesses": "1. Learning inverse dynamics models from visual inputs has been explored in the past (i.e. [1,2]). It would be good to discuss these papers in the context of this paper. \n2. The paper shows scalability in the direction of pre-training and finetuning data. To fully demonstrate scalability, it would be good to demonstrate the scalability in the model capacity axis as well.\n3. The current formulation of the model does not seem to take account of the history (past observations). This makes it challenging to extend to more complex environments where stronger task planning is needed. \n\n[1] Agrawal, Pulkit, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. \"Learning to poke by poking: Experiential learning of intuitive physics.\" Advances in neural information processing systems 29 (2016).[1] Agrawal, Pulkit, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. \"Learning to poke by poking: Experiential learning of intuitive physics.\" Advances in neural information processing systems 29 (2016).\n\n[2] Brandfonbrener, David, Ofir Nachum, and Joan Bruna. \"Inverse dynamics pretraining learns good representations for multitask imitation.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rtYC4H2oWQ", "reviewer": "Reviewer_f4m6", "summary": "This paper proposes an end-to-end Predictive Inverse Dynamics Model (PIDM) paradigm, to integrate vision and robot actions in a closed loop for a scalable policy for robotic manipulation. They introduce the Seer model, designed to leverage visual, temporal, and action features for large-scale pretraining, effectively combining the advantages of vision and action in policy formulation. The model is structured around two key components: Conditional Visual Foresight and Inverse Dynamics Prediction.\n\nIn Conditional Visual Foresight, Seer generates future RGB image predictions conditioned on specified goals (such as language instructions or robot states) and recent observations (previous RGB images and robot states), allowing the robot to anticipate and guide its actions. The Inverse Dynamics Prediction module then takes two consecutive observations—the current state and the visually predicted future state—to infer the required action sequence to achieve the anticipated outcome. This allows Seer to dynamically adjust its actions based on updated visual predictions, maintaining adaptability and responsiveness.\n\nThey performed extensive experiments in simulations and the real world, demonstrating the generalization capabilities and robust performance of the proposed method, underscoring its potential in advancing scalable, closed-loop policy solutions for robotic manipulation.", "questions": "- Generalization Across Embodiment: The model is pre-trained on the DROID dataset, limited to the Franka Panda robot. How does this impact its generalization to different robot types? Would you please discuss the possibility of training on a more diverse dataset, like Open-X, to improve adaptability?\n\n- Real-World Evaluation: The real-world evaluation uses the DROID dataset, with only Franka robots in varied tasks. Is there a risk of overfitting, potentially limiting performance in broader tasks? Would you please discuss task generalization?\n\n- Regarding the OpenVLA baseline, which specific checkpoint version was used? OpenVLA has multiple versions fine-tuned on LEBERO, and their performance varies across different tasks.\n\n- Typo: Line 067 using Inverse Dynamics Models (IDM) conditioned on the robots’. There is an extra \"'\".", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Effective Pre-training Objectives: The paper emphasizes the importance of pre-training the entire policy rather than just visual components, leading to significant performance improvements. The conditional visual foresight objective encourages the model to learn from the future, while the inverse dynamics prediction objective helps it understand the relationship between actions and their visual consequences.\n- End-to-End Training: Both the visual prediction and inverse dynamics modules are trained together, enabling the model to effectively utilize visual, temporal, and action information from large datasets. Seer uses Transformers to process visual states and actions, making it scalable.\n- Unidirectional Attention: The model uses a unidirectional attention mask in the Transformer encoder, allowing the action prediction module to integrate past and future information for better decision-making.\n- Extensive Results: The paper provides extensive experimental results on both simulated and real-world benchmarks. Seer consistently outperforms state-of-the-art baselines on tasks involving long-horizon planning, generalization to unseen scenes, and robustness to disturbances.", "weaknesses": "- Generalization across embodiment: While this paper proposes a generalizable ene-to-end pre-training paradigm for robotic manipulation policy, the proposed Seer model is pre-trained on the DROID [1] dataset, which only involves the Franka Panda robot. Training on the dataset across different embodiments such as Open-X [2] might improve generalization capability.\n\n- In the real-world evaluation, they selected dataset DROID, which contains demonstrations of Franka robots executing various tasks in diverse scenes. It is likely to overfit the dataset for policies for different tasks.\n\n[1] Khazatsky, Alexander, et al. \"Droid: A large-scale in-the-wild robot manipulation dataset.\" arXiv preprint arXiv:2403.12945 (2024).\n\n[2] O.-X. E. Collaboration, A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, et al., “Open x-embodiment: Robotic learning datasets and rt-x models,” arXiv preprint arXiv:2310.08864, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WcylTz9vui", "reviewer": "Reviewer_dizC", "summary": "The paper presents an approach for end2end learning of action and image predictions for imitation learning - termed Predictive Inverse Dynamics Model. The authors present a Transformer architecture that takes in language, observation history, and robot state, and predicts both actions and images in tandem (the action predictions are conditioned on the image predictions). The authors conduct experiments in simulation (LIBERO and CALVIN) and on real-world tasks pretrained on DROID to showcase the benefits of the proposed pretraining approach.", "questions": "1. Are the predicted actions at the same frequency as the prediction images? This point was confusing as the actions were referred to as intermediate actions throughout the paper.\n2. How many frames of observation history are used as input? How many frames are predicted?\n3. Did rolling a chunk of action predictions not work for inference (it seems like only one was rolled out)? Did you end up using a temporal ensemble? How many predictions were in the ensemble?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n* The general problem of learning action/image prediction models or world models is of interest to the community.\n* The problem is well-motivated and the literature review does a good job of contextualizing the paper in prior work.\n* The paper is strong, well-written and easy to follow.\n* The figures are informative and effectively illustrate the benefits of the proposed approach.\n* The experiments consider both simulation and real robot evaluation, as well as an ablation study, demonstrating the proposed method's superior performance as compared to the considered baselines.", "weaknesses": "* The authors may consider adding the following relevant works to the literature review.\n  * Gen2Act: Human Video Generation in Novel Scenarios enables Generalizable Robot Manipulation\n  * This&That: Language-Gesture Controlled Video Generation for Robot Planning\n  * VideoAgent: Self-Improving Video Generation\n  * IGOR: Image-GOal Representations Atomic Control Units for Foundation Models in Embodied AI\n* My biggest concern with the paper is the low number of experimental trials in certain settings (e.g., 20) and the lack of indication whether the results are statistically significant. At these lower trial numbers, it would not be unexpected that some of the performance difference is not statistically significant.\n* In line 71, it is stated that this approach is the first to optimize vision and action prediction synergistically. However, does the GR-1 paper not do the same high-level goal?\n* The limitations of the approach are missing from the paper.\n* Line 267 - vanilla typo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6J7YRIMyPB", "reviewer": "Reviewer_tjHp", "summary": "This paper presents a unified approach to combine visual sub-goal prediction and action prediction into a single predictive inverse dynamics model (PIDM). \n\n1. The proposed framework achieves impressive results on LIBERO-LONG and Clavin, two long-horizon manipulation benchmarks. \n2. The proposed framework is pretrained on DROID datasets, which shows the potential scalability of PIDM. \n3. The authors also evaluate the proposed method with a real robot and conduct ablation studies on the contribution of pretraining, the choices of pre-training, and fine-tuning loss functions.", "questions": "In addition to the weakness section: \n\n1. Could you compare SEER with the existing works that use sub-goal prediction? I am looking for a clear explanation of the essential differences in the method level. \n\n2. In Table 3, how does the finetuning/pretraining work if both the loss functions are not used? \n\n3. How much GPU resources are required during pre-training and fine-tuning? \n\n4. How are sub-goals selected when you prepare the training set? \n\n5. Does this framework work for tasks of more precision or contacts? This is the key issue of using video prediction models in robotics, as these methods sometimes lack the required precision when generating sub-goal videos.  CALVIN has long horizons but very simple manipulation tasks. The LIBERO also seems only have pick and drop, push and pull tasks on relatively large objects.\n\nIn general, I appreciate the quality of the idea and result of this work. I rate this paper a borderline accept at this moment, but I have many concerns. Please address them.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work studied an important problem: improving the data scalability of robot manipulation policies. \n2. The idea makes sense. The proposed PIDM combines sub-goal prediction and action prediction. The former, as a vision task, is more generalizable and naturally benefits the latter task. The data efficiency study in Fig.3 is convincing. \n3. The results of CALVIN look very impressive. \n4. The writing is easy to follow.", "weaknesses": "1. The authors did not discuss the limitations of the proposed method at all, while there obviously are. For example, it could be difficult to apply this method in high-frequency control tasks, such as ALOHA. Moreover, the experiments focus on easy manipulation tasks that do not require rich contacts or high precision, such as insertion, screwing, etc.  A thorough discussion of limitations is necessary, especially given the claim that PIDM is a scalable policy.\n\n2. The central idea is not novel. Bringing sub-goal prediction into manipulations has been recently explored by many existing works, such as [1][2][3], especially in the direction of using video generation models for robot tasks. \n\n3. The authors did not explain the network architectures clearly. Some descriptions are in the supplementary, but they are unclear, and there is no figure.\n\n4. When comparing with existing work, the authors only report success rates (or alike), but not network sizes and MACs (number of multiply-additions). It becomes unclear whether the performance gain comes from a larger network or a better method. \n\n[1]: Zero-Shot Robotic Manipulation With Pretrained Image-Editing Diffusion Models. ICLR 2024.\n\n[2]: Learning Universal Policies via Text-Guided Video Generation. NeurIPS 2023.\n\n[3]: Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation. NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)