GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation
Authors: Hongyin Zhang
Pengxiang Ding
Shangke Lyu
Ying Peng
Donglin Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Robot Manipulation; Vision Language Action Model
Primary area: applications to robotics, autonomy, planning
TL;DR: We propose a novel closed-loop VLA method GEVRM that integrates the internal model control principle to enhance the robustness of robot visual manipulation.

Abstract: With the rapid development of embodied artificial intelligence, significant progress has been made in vision-language-action (VLA) models for general robot decision-making. However, the majority of existing VLAs fail to account for the inevitable external perturbations encountered during deployment. These perturbations introduce unforeseen state information to the VLA, resulting in inaccurate actions and consequently, a significant decline in generalization performance. The classic internal model control (IMC) principle demonstrates that a closed-loop system with an internal model that includes external input signals can accurately track the reference input and effectively offset the disturbance. We propose a novel closed-loop VLA method GEVRM that integrates the IMC principle to enhance the robustness of robot visual manipulation. The text-guided video generation model in GEVRM can generate highly expressive future visual planning goals. Simultaneously, we evaluate perturbations by simulating responses, which are called internal embeddings and optimized through prototype contrastive learning. This allows the model to implicitly infer and distinguish perturbations from the external environment. The proposed GEVRM achieves state-of-the-art performance on both standard and perturbed CALVIN benchmarks and shows significant improvements in realistic robot tasks.

Published as a conference paper at ICLR 2025
GEVRM: GOAL-EXPRESSIVE VIDEO GENERATION MODEL FOR ROBUST VISUAL MANIPULATION
Hongyin Zhang
Pengxiang Ding!.?
Shangke Lyu² Ying Peng?
Donglin Wang?*
Zhejiang University. Westlake University.
wangdongl n& wewt laeetaaaeeeeee. .n
ABSTRACT
With the rapid development of embodied artificial intelligence, significant progress has been made in vision- saaion-lnngageacction (VLA) models for general robot decision-making. However, the majority of existing VLAs fail to account for the inevitable external perturbations encountered during deployment. These pertur- bations introduce unforeseen state information to the VLA, resulting in inaccu- rate actions and consequently, significant decline in generalization performance. The classic internal model control (IMC) principle demonstrates that closed- loop system with internal model that includes external input signals accu- rately track the reference input and effectively offset the disturbance. We propose novel closed-loop VLA method GEVRM that integrates the IMC principle enhance the robustness of robot visual manipulation. The text-guided video gen- eration model in GEVRM can generate highly expressive future visual planning goals. Simultaneously, we evaluate perturbations by simulating responses, which are called internal embeddings and optimized through prototype contrastive learn- ing. This allows the model to implicitly infer and distinguish perturbations from the external environment. The proposed GEVRM achieves state-of-the-ar perfor- mance on both standard and perturbed CALVIN benchmarks and shows significant improvements in realistic robot tasks.
INTRODUCTION
The pursuit of robust and adaptable robotic systems the cornerstone of embodied general intelli- gence. Recently, with the successful advancement of large-scale robot data collection (Vuong al., 2023), universal state representation learning (Li et al. 2003 Du et al.) 2024). and expressive pol- icy learning (Brohan al.) 2022 Chi et al., 2023). the research on vision-language-actiin (VLA) models for robots has made significant progress. The above strategies have been shown to be ef- fective estimating the robot's s state and generating robust actions a variety of environments, from physical simulators (Mu et al. 2021 Ding al.) 2023) to carefully designed real-world envi- ronments. However, these carefully designed environments do not take into account the inevitable external perturbations during deployment, such as fluctuating lighting conditions video stream noise due to signal transmission problems. When VLA models are deployed in these non-ideal environments, external perturbations will bring unpredictable state information to the robot. This makes VIA produce fragile and unstable actions inaccurate environmental states, resulting in a significant decrease in its generalization performance. Therefore, enhancing the robustness of VLA models to cope with the inevitable external perturbations when deployed is an ongoing challenge.
In the fields of computer vision (Smardd al.) [0003 Ciresan et al.) 2011 Ciregan al. 2012 Chen et al.] (2000) and reinforcement learning (Laskin et al.) 2020 Hansen et al. 202 Zheng et al. 2023). image augmentation is a common technique to alleviate the problem of model over-fitting, resist input image perturbations, and enhance model robustness. The idea is to apply task label- invariant transformations to the model's input images. For example, for object recognition tasks, image flipping and rotation do not change the semantic labels. Therefore, this technique has also
"oorresoondigg author.

---

Published as a conference paper at ICLR 2025
(a) Classic Internal Model Control
Feedback
Signal
Controller
System
Reference Input
Model
Disturbances
Response
(b) Internal Model Visuomotor Control
Feedbuck
State Encoder
Goal-Guided Policy
Action
Alignment
Generaled
Referenec Inputs
Response
Env: ut Disturbancer
Goal Encoder
Text-guided
Generation Model
Instruction
Figure 1: We are inspired by the classical internal model control (a) in automation systems. The principle illustrates that closed-loop system equipped with internal model that accounts for ternal input signals can precisely follow the reference input and effectively neutralize the perturba- tions. In this work, an internal model visuomotor control framework (b) is motivated and designed. We leverages a text-guided video model for generating highly expressive visual goal states as refer- ence input, goal-state and current-state internal encoders for modeling responses, and goal-guided policy for robust action generation.
been applied to robot visual language manipulation tasks. Some previous work has utilized vision as general medium to develop specific agents that can plan various tasks through imagination and execution (Black et al., 2023 Yang et al. 2023a] Du et al.) al.]| 2024). These methods involve generative models for predicting future videos or target images, followed by goal-conditioned policies that transform visual plans into actual actions. Image augmentation technology is utilized when training coal-conditioned policies, which to some extent alleviates the policy's over-fitting of specific tasks. However, these models are limited by their generative capabilities, the future goal image (or video) states they generate are not expressive enough, and image augmentation only allows the model to generalize within a narrow task distribution. It lacks strong resilience to environmental perturbations and struggles produce actions that are consistently effective across diverse task scenarios.
We are inspired by the principle of classical internal model control (IMC) shown in Fig (a). The core idea of this principle (Rivera al., 1986) is that in closed-loop control system, by build- ing a model inside the controller that can simulate external perturbations and reference inputs, the desired output can be accurately tracked and the perturbations can be effectively offset. That is, leverages an internal model to replicate the system's 's behavior and subsequently assess the system's 's perturbations, thereby augmenting the closed-loop stability. It is widely believed that intelligent mammals also rely on internal models to generate their actions (Nguyen-Tuoo & Peters 2011) and such mechanism is also revealed and supported by behavioral, neurophysiological and imaging data (Kawato 1999). More importantly, the integration of the internal model into the robot control system (Emken & Reinkensmeyer 2005) has been verified to enhance the robustness the robot motion control. However, the results are limited specific scenarios and hard extend more complex and general tasks, such as visual-language manipulation. How to instantiate the internal model the VLA framework to improve the robustness of decision actions has not been explored.
To this end, we propose GEVRM, Goal-Expressive Video Generation Model for Robust Vi- sual Manipulation. As shown in Fig [ (b), effectively implement the classic IMC principle in the VLA model, some components of our method are adjusted accordingly. 1) Goal generation. Taking video frames as universal interface to describe the robot state, we introduce an advanced text-guided video diffusion generation model as robot behavior planner to generate future goal frames as reference input. To improve the expressiveness of future goal states, we train the visual planner through efficient video spatiotemporal compression and random mask strategies to priori- tize the understanding of physical world laws (Kang et al. 2024). 2) State alignment. We estimate system perturbations by leveraging the simulated responses of the robot. These responses are called internal embeddings and are extracted from the robot state. Since the responses are inherently em- bedded in the robot's historical observations, the internal embeddings prototypical contrastive learning (Caron et al., 2020 Yarats et al.) 2021; Deng et al.) (022) to align the robot's future expressive goal states with its current state. This enables the model to implic-
be optimized through
2

---

Published as a conference paper at ICLR 2025
itly infer and distinguish perturbations from the external environment. 3) Goal-guided policy, We propose a diffusion policy conditioned on the generated highly expressive goals to better model the multi-modal task distribution of robot manipulation (Chi et al. 2023). This policy and the aforemen- tioned internal embedding are jointly optimized through inverse dynamics and contrastive learning objectives track highly expressive goals well even in the presence of perturbations. In summary, our contributions are threefold:
We introduce GEVRM, novel robust VLA model that incorporates the IMC principle to enhance robot visual manipulation.
We study how to obtain highly expressive goals with a text-guided video generation model and align state representations through prototypical contrastive learning to resist external perturbations at deployment.
Extensive experiments verify the effecttivnness and advancement of the proposed GEVRM. It significantly outperforms the previous state-of-the-art on the CALVIN benchmark with standard and external perturbations. The expressiveness of the goal states generated real visual manipulation is significantly improved compared to previous baseline methods.
RELATED WORK
Vision-Language-Acttin models. With the rise of extensive multi-task robotic datasets (Vuong
al., 2023). the robotics community is increasingly focusing on mast execution capabilities. The Langua-L Action models (Brohan et al. [022 Yue et al. (024) have gained traction for
their ability to use language for goal commands, enabling robots to make informed decisions based
on visual perceptions. Early studies (Brohan et al., 2022 Wu et al.] 2023) utilized cross-modal
attention between language and vision, but limited model performance hindered effectiveness. Re- cently, attention has shifted to large foundational models (Alayrac et al. 2022 Li et al. 2023 Kim
cet al. 2024), for improved versatility. However, text descriptions often lack detail about environ- mental states, complicating morphooyy tasks. As result, some researches (Du et al. 2024
Ko et al. 2023] Zhou et al. 2024 Black et al., 2023) Ajay et al.) 2024) Yang et al.) 2022b) now leverage vision as a universal medium, employing generative models forecast future actions, fol- lowed by goal-conditioned policies for execution. UniPi (Du et al. (024) was one of the first to leverage internet-scale data to train a cox-cconditioned video generator. using an inverse dynamics model to estimate actions. Similarly, SuSIE (Black et al. (023) uses an image-editing model to plan high-level sub-goals for low-level controllers, while ADVC (Ko al., 2023) infers actions from predicted video content with dense correspondences. These efforts aim for universal state repre- sentation but fall short for two reasons. First, existing visual plans experience temporal and spatial inconsistencies due to poor dynamics modeling. We propose robust video generation model that addresses this issue and enhances action execution. Second, prior work focuses on controlled envi- ronments, overlooking the robot's responses to external interference. Our GEVRM method employs contrastive learning for state alignment, effectively simulating responses and resisting disturbances. Together, these elements define our expressive goal representation.
Internal Model Control framework. The IMC framework a widely recognized control strategy that leverages internal model of the system predict future behavior and adjust control actions accordingly, making it highly robust against disturbances and model inaccuracies. First introduced by Garcia and Morari (1982), IMC has been applied extensively in both linear and nonlinear process control, offering significant benefits in terms of stability and adaptability (Garcia & Morari 1982] Rivera et al. 1986; Morari 1989). Its feedback mechanism allows for real-time adjustments, partic- ularly valuable in dynamic environments such as robotics, where precision is critical. IMC's design has been further explored and refined for multivariable and complex systems, proving its versatility and robustness in various control applications (Skogestad & Postlethwaite (005). However, most previous research works are limited to specific control scenarios and are difficult to extend to gen- eral visual language manipulation tasks. More recently. inspired by classical closed-loop control systems, a closed-loop visuomotor control framework (Bu et al. 2024) has been proposed that in- corporates feedback mechanisms to improve adaptive robot control. Different from these works, study how to effectively instantiate internal models in the VLA framework to improve the robustness of decision actions.
3

---

Published as a conference paper at ICLR 2025
3 PROBLEM FORMULATION
In this work, we investigate how to generate highly expressive goal states and induce robust ac- tions to be resilient to external disturbances. Formally, we study robot trajectory and action gen- eration in a non-Markov decision process framework, which is specified by the following tuple: M := (X, 9: T, Po), where X' and A denote the image state and action spaces, G represents the language text goal space, T(x++1 |x a, g) is the transition dynamics, and (o(x) is the initial image state distribution. We aim generate expressive future image goal states and current actions to be performed given abstract language instructions and historical image sequence states (i.e.. videos) To: in visual manipulation tasks: p(a,, Te.T g. To:t). The problem decomposed into two hierarchical levels: 1) Robot behavior planning - given language instructions and historical video states To:t, infer image goal states Ie:T; 2) Robot action prediction given historical and inferred expressive future image goal states To:T, predict the current action ae to be performed. The decoupled process can be expressed as:
pe (a,, Te.T g, To:t) Po (a, To:e) (o:4)))((
(1)
This decoupling process greatly reduces the model training's s dependence on language, image se- quence, and robot action pairs. Specifically, the training of the behavior trajectory planning model Ps only requires text-video pairs Dr,, {(T' 9') without robot action labels, which can be derived from large-scale video clips with language labels and robot sequence decision data with text annotations on the Internet. The training of Pe only requires small amount of play data Dr,a ((r), a²) } without language labels for specific downstream tasks. In the test phase, given a natural language description gteat and an initial image state test of new task, We need to evaluate not only the expressiveness of the future goal states inferred by the model but also the success rate of completing the task under external perturbations.
METHODOLOGY
Our goal is build a robust VLA model that incorporates IMC concepts into robotic visuomotor control, as shown in Fig. 2 To set highly expressive goals before execution, we introduce powerful video generation model as a visual planner (Section 4. In Section 4.2 we detail how to align the goal state to evaluate perturbations and show how to induce the generation of robust decision actions.
Finally, in Section 4.3
we implement the overall test-time execution pipeline of GEVRM.
4.1 ROBOT BEHAVIOR PLANNER
Inspired by the recent success of video generation models (Brooks et al. 2024 Esser et al. 2024), we seek to build text-guided video diffusion transformer model as a robot behavior planner Po(TeTlg, To:)) for robotic goal states generation. The planner can faithfully synthesize future goal image frames based on historical video observations and abstract textual task descriptions. Plan- ning via video generation requires model that can both generate constrained videos starting from the given video and complete the downstream task. Specifically, to obtain highly expressive future goal states, three core aspects need to be considered when designing a robot behavior planner: 1) video spatio-temporal compression to reduce computing resources, 2) a random mask mechanism that highlights the understanding of physical laws and object consistency, and 3) strong model backbone and an efficient training paradigm.
Video spatio-temporal compression. Diffusion transformers (DiT) require massive computational resources to perform complex operations on robot image state sequence data in native pixel space. To alleviate this problem, we first compress the original pixel space with 2D VAE, and then further compress it with 3D VAE to obtain an information-ric low-dimensional dense space. The advantage fact, after
that the high computational cost of 3D VAE in the original pixel space is avoided. the spatial compression of 2D VAE, there is still considerable temporal correlation between adjacent features. Specifically,, during the image state sequences encoding phase, we initially affect spatial dimension reduction by a factor of 8 x 8 through the application of 2D VAE (Rombach et al. 2022). and subsequently condense the temporal dimension by a factor of 4x via 3D VAE (Yu et al. 2023 In the image state sequences decoding phase, the temporal dimension is restored prior to the spatial dimension. The 3D VAE incorporates Causal 3D convolutional layers in place of 3D CNNs, ensuring that each frame's output is contingent solely its antecedent frames Yuu et al. 2023
4

---

Published as a conference paper at ICLR 2025
State
20 VA Preodee
30 VALE Encocks
Latent Patches
State Encoder
psource
Random
3D VAE Decodkr
2D VAR Decoder
Prototypos
Prediction
Goal
State Sequences
DiT Blocks
Goal Encoder
2
ptarget
Training
State lignment
Encoder
2,
7D RototActton
"Put mushrsso and pickle in pot"
Goal-Guided Policy
N Position
Language Instruction
Infermnce
Rotation
Grip
Gal G Genraaion
Action Prediction
The proposed GEVRM model. First, the TS (Raflel et al. 2020) model is utilized to encode language instructions, and 2D and 3D VAE are utilized compress and restore the original pixel space of the robot image state sequence, followed by the DiT module and random mask mech- anism to generate the goal image state. Then, through prototypical contrast learning, the current and goal states are aligned to simulate responses and evaluate perturbations. Finally, the goal-guided policy predicts the 7-dimensional robot decision action.
Figure 2:
Random mask mechanism. To achieve efficient goal image synthesis, we implement a random mask mechanism (Tay et al., 20222.. Precisely, the training process involves the random unmasking of frames, encompassing scenarios such as revealing the initial frame, the first / frames, the final frame, the last h frames, a combination of the initial and last frames, and arbitrary frames. Dur- ing the testing phase, have access to the historical image state but devoid of future image state. Consequently, in the model's is training regimen, the unmasking of the first h frames is as- signed the greatest weight, specifically 75%. The remaining unmasking strategies supplementary objectives, collectively constituting the remaining 25%. Though the masking mech- anism is conceptually straightforward. it enables the robot behavior planner to anticipate subsequent frames based on various temporal snapshots, significantly enhancing the model's 's comprehension and perception of object dynamics and temporal sequential correlations. For the specific parameter configuration of the random mask, please refer to Appendix Tab. g
categorized as
Model backbone and training. Our DiT module is derived from a pre-trained text-guided video generation model (Zheng et al. (024) and integrates a frozen 75 encoder (Raffel et al. 2020) to pro- cess language instructions (Raffel et al., 2020). Drawing inspiration from the recent advancements in Stable Diffusion 3 (Esser al.. 22024), we fine-tune robot behavior planner with Rectified Flow (Liu (ct al. 2022), transcending the conventional DDPM (Ho et al. 2020). The Rectified Flow facilitates the learning of mapping from noise to real image distribution by solving ordinary differential equation along straight paths between samples (Liu et al.) (022). This approach has been shown to be more efficient training paradigm, resulting in significant reduction in video sampling steps, which in turn significantly increases the model training speed and reduces its inference time.
4.2 ROBOT ACTION PREDICTION
The expressive goal state generated by the robot behavior planner is utilized to guide the prediction of decision actions. From the visual goal state Tgoo and the current visual state x to the final action output, goal-guided (.p æ) be divided into the following two parts: 1) State alignment to simulate responses. Extract informative features from the visual goal state and the current visual state and utilize prototypical contrast learning to align state representations, simulate robot responses, and evaluate disturbances. 2) Goal-guided action prediction. Decode the goal and current internal compact encoded signals into actions that the robot robustly perform.
State alignment to simulate responses. Within the domain of classical control systems, the IMC framework necessitates the integration of system's internal model within its controller. This in-
5

---

Published as a conference paper at ICLR 2025
ternal model is capable of offsetting external disturbances and reference inputs, thereby ensuring precise system behavior and reliability. To implement the IMC principle in learning-bai work, we initially deploy residual networks ResNer 34 (He al. 2016) to serve as visual encoders for both the goal and current states. This conversion transforms raw pixel data into an enriched visual representation [x) and fer (Egoatt) For the current visual state representation (1)) the key is how to optimize it to simulate the robotic response for assessing external perturbations, with this response being inherently encoded within the visual goal state. Adhering to the IMC princi- ple, we advocate for the modeling of this process within the latent space 2 and optimize it through contrastive learning to achieve alignment with the visual goal state.
frame-
In the play data Dr,a, if pair of Zgoat and originates from the same trajectory, they are a pos- itive pair, otherwise they are negative pair. These pairs are optimized by swapping the assigned tasks (Caron et al., 2020). Specifically, given sequence of image observations sampled from the play data 2, we can derive the future goal image x goooll from the transition as the target vector, and the current image observation as the source vector. The source and target vectors are fed to the source and target encoders, respectively, to obtain latent features, which are mapped onto the unit sphere in high-dimensional space and L2-normalized:
2 fa(
and 2 = fur (I (z))22
(2)
IIfor (Zgoat)))
To predict cluster assignment probabilities prouree and p Aarget from latent features, we first apply Le normalization to the prototypes to obtain a trainable normalized matrix E = {e, and then take the soft maximum of the dot product of the source or target vectors of all prototypes:
e 20m
e Gen
P source
En ze and p parget
En ed
Here is the temperature parameter. p prouroe and p pranget are the predicted probability that the cur- rent and goal image observations x and Fggoat map to individual cluster with index n. To obtain the predicted probabilities {q) [gooure]] 1.N [n=] and {g [ffaget].. targets while avoiding trivial solutions, the Sinkhorn-Knon Knoppal algorithm (Cuturi 2013) is applied. Now that we have cluster assignment predictions and targets, the state alignment objective is to maximize the prediction accuracy:
Ts BB gooat Pa,, souree In target + ggaagget In p couree.
(4)
It is worth noting that learning representations to distinguish different instruction and visual repre- sentation is a long-standing scientific issue (Pathak et al.] 2017). while few studies have explored their ability simulate robot responses. This capability is not directly accessible in pre-trained vi- sual encoders or policy models learned based only on the current observation (i.e., behavior cloning).
Goal-guided action prediction. To keep the model concise, general, and generalizable, we leverage a goal-guided diffusion policy to decode the action output from the state encoding of the simulated response. We only utilize the third-view RGB images from the static camera as input and the action labels as training labels. The robot proprioceptive observation and gripper view images are not applied. The action space of 7-DoF robot is considered, consisting of the position of the end- effector Ogg E R6 and the gripper state Ogripper E 1), The goal-guided diffusion policy is a latent variable model using Markov noise de-noising process, which can be utilized to model the parameterized behavior distribution The (a|z, 2) = = - (ap ") for the latent variable (ax J The forward noise process follows a fixed variance schedule { k which follows the
distribution q(<//a+1) = (VVI- W B,Z). Following DDPM (Ho et al. 2020), our practical
implementation involves directly parameterizing the score network TT [2: 2, 2, k) to recover the behavior cloning objective:
(5) We utilize this objective to train a goal-guided policy and provide with internal embeddings of the goal and current state. In each policy training iteration, the state encoding is optimized by the state alignment objective, which enables the policy to implicitly infer and distinguish perturbations from the external environment. Therefore, the final optimization objective of the state encoding and goal-guided diffusion policy is:
(1pp::lllllppppppp + V1- 2, 2,
I == Jo AJ
6

---

Published as a conference paper at ICLR 2025
where À is temperature parameter. To sample from T. (apll 2 2), reverse diffusion process where ak ~ I) and e ~~ N(O,I) is utilized, resampling at each step:
1 Var
B:
ag-11
(az
(o +
for for =
V1
akk
4.3 TEST-TIME EXECUTION PIPELINE OF GEVRM
Once both the robot behavior planner Po and the goal-guided policy Fe are trained, they can be utilized to solve new manipulation tasks. Given new scenario Fo,test and a new language command Stest, GEVRM attempts to solve the task by iteratively generating highly expressive goal states and inducing goal-guided policies achieve these sub-goals. Initially, m
sample set of goals PP(--zo,teet Itest), where M represents the number of goal state generation. We pass goal state Ft.goal and current state It through the state encoders over Leest time steps to obtain the internal embedding and derive the goal-guided policy Tg. where Leest is the fixed interval number. After Ltest time steps, we refresh the goal states by sampling from the behavior planner again and repeat the process. The test execution of the algorithm is shown in Algo.
{Im.goal MM
Algorithm GEVRM: Test-time Execution
1: Robot behavior planner Po, state encoder fv, goal state encoder fwr goal-guided policy Tip time limit T, goal sampling interval Ltest goal generation number M, initial state 2o,tests language instruction Stest.
2: 6-0
3: while t < T do
4: Sample goals (am. I ~ Pol-2as,eat (teat)
Behavior planner generates goals.
5: for / = 1 to Liest do
6:
[u(lLttst) 2t (ee,
State encoding and La normalization.
7:
2, - (M
D Goal state encoding and L2 normalization.
8:
Sample action ag ~ (-((122,
> Goal-guided action prediction.
9:
11+1,test Env.Step(a.)
10:
end for
12:
end while
EXPERIMENTAL EVALUATION
In this section, we evaluate the state generation and visual manipulation capabilities of GEVRM. To this end, experiments aim to investigate the following questions: 1) Can GEVRM have strong generalization ability to generate expressive goal in various environments? 2) Does GEVRM exhibit higher success rate in executing robot tasks compared to the baseline in various environments? 3) How important are the core components of the GEVRM for achieving robust decision action?
5.1 EVALUATION ON GOAL GENERATION
Setup. We utilized two types of datasets (realistic Bridge (Wakke et al. 2023) and simulated CALVIN (Mees et al. 2022b) to evaluate the generalization of goal generation. We train the model on predefined training set and evaluate the robot goal generation performance a test set with and without external perturbations. The hyperparameters are shown Appendix Tab, X and Tab. g
Baselines. To make fair comparison, we have chosen open-source video generative models: 1) AVDC (Ko al. 2023), a typical diffusion-style generation model for robotics. 2) GR-1 (Wul al. (202). which is autoregressive-stylle generation model that takes language instructions, and state sequences as inputs, and predicts robot actions and future images an end-to-end manner. the image editing diffusion model high-level planner and proposes intermediate sub-goals that can be achieved by the low-level controller.
3) SuSIE (Black et al., 2023).
Metrics. The evaluation metrics employed the Frechet Inception Distance (FID) (Sitzer 2020) and the Frechet Video Distance (FVD) (Skorokhodov et al., 2021 Yu et al. 2022). both widely recognized in the domains of image and video generation. We also evaluate the quality of videos

---

Published as a conference paper at ICLR 2025
Table 1: Goal generation quality comparison. Our method greatly surpasses the baseline across all metrics. The best results for each task are bolded.
Benchmark
Algorithms
FID (L)
FVD (L)
LPIPS (+)
SSIM (t)
PSNR (1)
BridgeData
AVDC
246,4551:39.8
22.89::4.99
0.23
0.73
18.22:12.53
BridgeData
SuSIE
114.79-t21.38
0.22 +000
0.71
16.39+
BridgeData
GEVRM (Ours)
35.70=t10.77
1.165:1.35
0.06
0.89
22.36-:2.75
CALVIN
GR-1
236..5.1:8.887
12.83-12.60
0,20-1o.0z
0.65
18,59-00.5
CALVIN
SuSIE
414.14:545:45
0.15
0.75=t0.05
18. 12:22.299
CALVIN
GEVRM (Ours)
94,47=22.554
3.80=112
0.09:t0.04
0.80
21.10-t3.29
AVDC
AVDC
No
Ours
Imuge Ovclusions
pertorbation
AYDC
AVDC
Color
Image
Ours
siiji
Figure 3: Comparison of goal generation on task "put blueberry in pot or pan on stove".
generated by different models on other standard metrics (Bu et (SSIM), Peak Signal-to-Noise Ratio (PSNR), Learned Perceptual Image Patch Similarity (LPIPS).
2024): Structural Similarity Index
Goal generation comparison. We evaluate the generalization of the goal generation in the unseen environment (Tab. [). The results show that the performance of our GEVRM model significantly improved compared to the baseline. The results indicate that GEVRM exhibits enhanced expressive capabilities, effectively modeling the intricate textures and temporal coherence of robotic image se- quences. Then, we compare the robustness of goal generation in the perturbed environment (Fig, 3). The baselines struggle with environmental variations, generating severe hallucinations that distort objects and may even completely ruin the scene. In contrast, our method produces fewer halluci- nations and can generate expressive goal states following language instructions. This confirms that GEVRM is indeed better able to understand the laws of the physical world and maintain the 3D consistency objects. More goal generation results are in Appendix Fig. L5
5.2 EVALUATION ON ACTION EXECUTION
Setup. We conduct experiments CALVIN, benchmark for language-conditioned manipulation to evaluate the GEVRM's capabilities in closed-loop action execution. CALVIN consists of four simulated environments (A, B, C, and D), each with dataset of human-collected play trajectories. We study zero-shot multi-environment training on A, B, and C, and testing on D, varying in table texture, furniture positioning, and color patches. We also test GEVRM's robustness to perturbations (Fig. Details of environmental disturbances given in Appendix Section A.I The policy
training hyperparameters are shown Appendix Tab. 10
Baselines. We select the representative baselines to verify the generalization performance standard unseen environments: 1) UniPi (Du et al.] (0024): Recasts decision decision-making into text- conditioned video generation firstly, enabling the production of predictive video sequences and sub- sequent extraction of control actions. 2) HiP (Ajay al. (2044) This model improves upon UniPi by incorporating hierarchical inference to extend long-term planning capabilities. 3) GR-1 (Wu al.
8

---

Published as a conference paper at ICLR 2025
(a) Training
(c) Test 2 Env. D Original Image
Env.
Image Shift
Image Rotation
Env.
(b) Test Env.
Color Jitter
Image Occlusions
Noise Interference
Figure The model trained only on data collected in environments A, B, and C (a), and tested on environment D (b). Besides, we apply five perturbations to the image observations of environment further test the generalization of the model in more challenging scenarios (c).
2023): This model leverages a pre-trained video model to enhance autoregressive action generation. 4) RoboFlamingo (Li al. 2023), uses pre- traitreined VLM for single-step visual language under- standing and models sequential history information with an explicit policy head. In addition, in the test environment with external perturbations,
choose the representative baseline SuSIE (Black) et al. 2023), because it adopts common data augmentation strategies to cope with perturbations and achieves state-of-the-ar results in previous works. We consider third-view RGB images from static cameras as observations, which makes the robot execution more challenging. More comparison with language-conditione methods are in Appendix Sec. A.2 and Tab.
Action execution comparison. We show the success rate completing each language instruction in the chain Tab. 2 The model is trained on en- vironments A, B, and C (Fig. (a)), and test D (Fig- 4 (b)). Compared with the bascline, the GEVRM has a significant performance improve- ment. This shows that method based on the IMC principle has bet- ter goal generation ability when fac- ing new environments and induces the robot predict more general de- cision actions.
Table 2:
Generalization on unseen environments in CALVIN (train A, B, C - test D). reproduced version training on third-view images.
No. of Instructions Chained
Algorithms
2
3
4
5
HiP
0.08
0.04
0.00
0.00
0.00
UniPi
0.56
0.16
0.08
0.08
0.04
GR-1*
0.75
0.45
0.2
0.15
0.10
SuSIE
0.87
0.69
0.49
0.38
0.26
GEVRM (Ours)
0.92
0.70
0.54
0.41
0.26
Table 3: Generalization on perturbed environments in CALVIN (train A, B, - perturbed test D).
No. of Instructions Chained
Five Perturbed Tasks
Algorithms
Avg. Length (t)
1
2
3
5
SuSIE
0.56
0.26
0.13
0.10
0.06
1.11
RoboFlamingo
0.63
0.35
0.18
0.09
0.05
1.31
Average
GR-1
0.67
0.38
0.22
0.11
0.06
1.44
GEVRM (Ours)
0.70
0.47
0.26
0.11
0.07
1.62
Action execution comparison under external perturbations. To thoroughly evaluate the perfor- mance of our proposed GEVRM against the baseline SuSIE, we tested both models across five more challenging scenarios (Fig. (c)). The average performance on five perturbed tasks in Tab. 3 and the specific results are in Appendix Tab. These scenarios were crafted to challenge the models' perception environmental stimuli and comprehension of physical laws. The results show that GEVRM can well simulate robot response and guide the policy to generate robust decision actions to resist external perturbations. More action execution comparison results are in Appendix Tab, Real-world deployment of GEVRM is in Appendix Sec. A.3
5.3 ABLATION STUDY
In this section, we perform an ablation study to assess the contributions of VAE and state align- ment to method. We evaluate the effects of VAE fine-tuning and state alignment application

---

Published as a conference paper at ICLR 2025
2
fnrenge Achreon Tlok Lenggh
Success Fate (W
Acheever Sucoess Raro
100-2
Bon-2
100-4
As an
Mak
Ap.
(a) Different training paradigms.
(b) SA hyperparameters.
Figure 5: Ablation study on the CALVIN ABC - D. (a) We compare different training paradigms. (b) We examine the impact of different values of the state alignment (SA) hyperparameter X.
(a) State Embed. (w/o SA)
(b) State Embed.
Creaa satien-a
the light
use the swich to
the tlck (iaag de
oo clece tha drawer
and
The plock nottl to right
The pre Slock
con oe TCTTo
ppan bod git..
Bush the hancie thg 0000 Drowe?"
Figure 6: Comparison of state representations. The representations with state alignment (SA) show enhanced cluster centers, class boundaries, and temporal consistency.
model performance across CALVIN environments A, B, and C, focusing on robot behavior plan- ning and goal-guided policy training. Results in Fig, (a) indicate that omitting VAE fine-tuning or state alignment integration significantly degrades model performance on CALVIN Env. D, due to the VAE's pre-training on diverse video data enhancing spatio-temporal consistency and subse- quent fine-tuning on robot data aiding in decision-making generalization. State alignment bolsters the policy's visual state representation for better task generalization. Moreover, the hyperparameter X, crucial for balancing expert imitation and state alignment in policy training, was tested across five values (Fig. 5 (b)). Performance metrics varied minimally, showing robustness to À adjustments, with A = 1 optimal for our method. To illustrate the impact of state alignment on goal-guided representation, we conducted a visual comparison experiment. We utilize 7-SNE (Van der Maaten & Hinton 2008) to analyze the latent space representations of current and future image states with and without state alignment in the CALVIN ABC + D "Noise Interference" task, and the results are shown in Fig. 6nd Appendix Fig. g Results indicated that state alignment improves clustering and classification by enhancing intra-category cohesion and cateroca separation. Addition- ally, state alignment ensures temporal consistency image state sequences, thereby bolstering the policy's 's environmental and task recognition, and facilitating generalization to novel scenarios. The ablation experiments the execution efficiency of the goal generation and the goal-guided diffusion policy are in Appendix Tab. and 7 respectively.
CONCLUSION
The innovation of our method lies in its ability to internalize the classical internal model control principle into the modern VLA framework, thereby enhancing the robot's ability to handle environ- mental perturbations and maintain performance integrity. In the proposed robust GEVRM model, we leverage video generation models to obtain highly expressive target states. Meanwhile, we effec- tively align state representations based on prototype contrastive learning to simulate robot responses and evaluate external perturbations. As demonstrated by the GEVRM's sratc-of-the-s performance in simulated and realistic visual manipulation tasks, it effectively enhances the expressiveness of the goal state and exhibits strong resilience to external perturbations. Therefore, our work greatly expands the reliability and robustness of robotic systems in deployment scenarios and is impor- tant step forward in the field of embodied general intelligence. A promising work is to consider incorporating more general high-quality video generation models into the VLA framework to cope with complex and diverse manipulation tasks of real-world robots.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Science and Technology Innovation 2030 Major Project (Grant No. 20222D0208800), and NSFC General Program (Grant No. 62176215).
REFERENCES
Anurag Ajay. Seungwook Han, Yilun Du, Shuang Li, Abhi Gupta, Tommi Jaakkola, Josh Tenen- baum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. Advances in Neural Information Processing Systems, 36, 2024.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Kevin Black, Mitsuhiko Nakamoto, Pranav Atreya, Homer Walke, Chelsea Finn, Aviral Kumar, and Sergey Levine. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint arXiv:2310. 10639, 2023.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06811 2022.
Tim Brooks, Bill Peebles. Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models world simulators. 2024. URL petttp:://ppnne com/ rer/eareeer..
video- geri0-ggneraaae Cerneeratin www.nn.aaiie........ee.... modelssoooeesssse as wor 1d simulators
Qingwen Bu, Jia Zeng, Li Chen, Yanchao Yang, Guyue Zhou, Junchi Yan, Ping Luo, Heming Cui, Yi Ma, and Hongyang Li. Closed-loop visuomotor control with generative expectation for robotic manipulation. arXiv preprint arXiv:2409.00016 2024.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning. pp. 1597-1607. PMLR, 2020.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shu- ran Song. Diffusion policy: Visuomotor policy learning via action diffusion. arXiv preprint 0431: 2023.
Dan Ciregan, Ucli Meier, and Jürgen Schmidhuber. Multi-column deep neural networks for image classification. In 20/2 IEEE conference computer vision and pattern recognition, pp. 3642- 3649. IEEE, 2012.
Dan C Cireşan, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen Schmidhuber. High- performance neural networks for visual object classification. arXiv preprint arXiv:1102.0183 2011.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. Advances in neural information processing systems, 26, 2013.
Fei Deng, Ingook Jang, and Sungjin Ahn. Dreamerpro: Reconstruction-ffer model-based reinforce- ment learning with prototypical representations. In International conference on machine learning, pp. 4956-4975. PMLR, 2022.
Pengxiang Ding, Han Zhao, Zhitao Wang, Zhenyu Wei, Shangke Lyu, and Donglin Wang. Quar-vla: Vision-language-actio action model for quadruped robots. arXiv preprint arXiv:2312. 14457, 2023.
11

---

Published as a conference paper at ICLR 2025
Yilun Du, Sherry Yang, Bo Dai, Hanjum Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Jeremy Emken and David Reinkensmeyer. Robot-enhanced motor learning: accelerating internal model formation during locomotion by transient dynamic amplification. IEEE Transaclions Neural Systems and Rehabilitation Engineering, 13(1)333-39, 2005.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Carlos E Garcia and Manfred Morari. Internal model control. unifying review and some new results. Industrial Engineering Chemistry Process Design and Development, 21(2):308-323, 1982.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision transformers under data augmentation. Advances in neural information processing systems, 34: 3680-3693, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and paitern recognition, pp- 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffuusion probabilistic models. CoRR, abs/2006. 1239, 2020.
11239
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Car- ion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1780-1790. 2021.
Bingyi Kang, Yang Yue, Rui Lu, Zhijie Lin, Yang Zhao, Kaixin Wang, Gao Huang, and Jiashi Feng. How far is video generation from world model: A physical law perspective. arXiv preprint DxXiv:4411002380 2024.
Mitsuo Kawato. Internal models for motor control and trajectory planning. Current opinion in neurobiology, 96):718-727, 1999.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source model. arXiv preprint arXiv:2406.0922 2024.
labion-langugg--ace
Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B Tenenbaum. Learning to act from actionless videos through dense correspondences. arXiv preprint arXiv:2310.08576, 2023.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances neural information processing systems, 33: 19884-198955, 2020.
Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, et al. Vaniuaelanguage foundation models as effective robot imitators. arXiv preprint arXiv:2311.01377 2023.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 2022.
Corey Lynch and Pierre Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint ArXiv:2005.076A4 2020.
Oier Mees, Lukas Hermann, and Wolfram Burgard. What matters in language conditioned robotic imitation learning over unstructured data. IEEE Robotics and Automation Letters, 7(4): 1205- 11212, 2022a.
12

---

Published as a conference paper at ICLR 2025
Oier Mees, Lukas Hermann, Erick Rosete- cosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-condition conditioned policy learning for Long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters. 7(3):7327-73344 2022b.
M Morari. Robust process control. Prentice-Hall google schola, 2:654-658, 1989.
Tongzhou Mu, Zhan Ling, Fanbo Xiang, Derek Yang. Xuanlin Li, Stone Tao, Zhiao Huang, Zhi- wei Jia, and Hao Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2 /4483, 2021.
Duy Nguyen- Tuong and Jan Peters. Model learning for robot control: survey, Cognitive process- ing, 12:319-340, 2011.
Alexander Pashevich, Robin Strudel, Igor Kalevatykh, Ivan Laptev, and Cordelia Schmid. Learn- ing to augment synthetic images for sim2real policy transfer. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2651-2653 IEEE, 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787.
PMLR, 2017.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1667, 2020. URL http:
7/ wttp://pppa/////////00.0 074 html
Daniel E Rivera, Manfred Morari, and Sigurd Skogestad. Internal model control: Pid controller
design. Industrial & engineering chemistry process design and development, 25(1):252-265, 1986.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Maximilian Seitzer. pytorch-fid fid: FID Score for Py PyTorch. httppa///gitaub..com/ees pytorch-fid] August 2020. Version 0.3.0.
Patrice Y Simard, David Steinkraus, John C Platt, et al. Best practices for convolutional neural networks applied to visual document analysis. In Icdar, volume 3. Edinburgh, 2003.
Sigurd Skogestad and Ian Postlethwaite. Multivariable feedback control: analysis and design. john Wiley & sons, 2005.
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2021.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung. Siamak Shakeri. Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint arXiv:2205.0513! 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Quan Vuong, Sergey Levine, Homer Rich Walke, Karl Pertsch, Anikait Singh, Ria Doshi, Charles Xu, Jianlan Luo, Liam Tan, Dhruv Shah, al. Open x-embodiment: Robotic learning datasets and rt-x models. In Towards Generalist Robots: Learning Paradigms for Scalable Skill Acquisition
CoRL2023, 2023.
Homer Rich Walke, Kevin Black, Tony 7. Zhao, Quan Vuong, Chongyi Zheng, Philippe Hansen- Estruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning scale. In Conference Robot Learning, pp. 1723-1736. PMLR, 2023.
Hongtao Wu, Ya Jing, Chi-Hou Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. ArXiv, abs/2312.13139, 2023.
13

---

Published as a conference paper at ICLR 2025
Mengjiao Yang. Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter Abbeel. Learning interactive real-world simulators. ArXiv, abs/2310.061144 2023a. URL
https: / /ttp://ppp.... www.seeeeacncsoo......... wwtp/oorgssaaoooo.ee. 26:33338838
Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, and Pieter
Abbeel. Learning interactive real-world simulators. arXiv preprini arXiv:2310.06/ 14, 2023b.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. 11920-11931. PMLR, 2021.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion- tokenizer key to visual generation. arXiv preprint arXiv:2310.05737, 2023.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversariaal networks. In International Conference on Learning Representations, 2022, UKL/attps///gennreeeeee.cc forumm?
SLEC2sdv-S4-w9
Yang Yue, Yulin Wang, Bingyi Kang, Yizeng Han, Shenzhi Wang, Shiji Song, Jiashi Feng, and Gao Huang. Decr-vla: Dynamic inference of multimodal large language models for efficient robot execution. NeurlPS, 2024.
Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdi- nov, and Sergey Levine. Stabilizing contrastive rl: Techniques for robotic goal reaching from offline data, arXiv preprint arXiv:2306.03344 033466 2023.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URLhttps
wwt.au/.cca/pppaaaeee//ppppaepee Sora
Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit- Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination. arXiv preprint arXiv:2404. 12377, 2024.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ENVIRONMENT PERTURBATIONS.
CALVIN Datasets. We conduct experiments on CALVIN (Mees et al. 2022b), benchmark for long-horizon, language-conditioned manipulation to evaluate the GEVRM's capabilities in closed- loop action execution. CALVIN consists of four simulated environments (A, B, C, and D), each with a dataset of human-collected play trajectories. Each environment consists of a Franka Emika Panda robot arm positioned next to table with various manipulable objects, including drawers, sliding cabinets, light switches, and colored blocks. Environments are distinguished by their tabletop texture, the positions of furniture objects, and the configuration of colored blocks. We study zero- shot multi-environment training on A, B, and C, and testing on D, varying in table texture, furniture positioning, and color patches.
Environment perturbations Details. We list the details of perturbations in Fig 4 1) The image state randomly translated to the upper left, with maximum translation ratio of 0.1 relative to the image size. 2) The image state is randomly rotated counterclockwise, with a maximum rotation angle of 30 degrees. 3) The image state saturation, brightness, contrast, and sharpness are randomly jittered with maximum random factor of 3. 4) The image state is randomly occluded with a random number of occlusion blocks ranging from to 3 and a maximum length of 60. 5) The image state perturbed with random noise blocks. We follow the evaluation protocol of Mees et al. (Mees et al.) 2022b). During the evaluation, the policy is required to complete five chains of language instructions for 360 time steps. Notably, only consider RGB images from the static camera as observations, which makes CALVIN much more challenging.
A.2 BASELINE METHOD INTRODUCTIONS.
For a fair comparison, we here select the video generation canseation-based baselines to verify the zero- shot generalization performance standard unseen environments: These bascline methods include language-conditiong policies that leverage pre-trained visual-language models in various ways: 1) HULCC (Mees al. 2022a), model that employs multi-modal transformer encoder for language- conditioned robotic manipulation, combines self-supervised contrastive learning to align video and language representations and uses hierarchical robotic control learning tackle complex tasks. 2) MCIL Lynch& & Sermanet, 2020). multi-context imitation learning framework, is capable of han- dling large-scale unlabelled robot demonstration data, MCIL trains a single goal-conditioned policy by mapping various contexts, such as target images, task IDs, and natural language, into shared latent goal space. 3) MdetrL.C (Kamath et al., 202T), which integrates visual and textual informa- tion to perform object detection and multi-modal understanding. Mdetrl.C uses text query mod- ulation to detect objects within images and demonstrates strong performance in tasks like visual question answering and phrase localization. 4) AugLC (Pashevich et al. 2019)), which optimizes image augmentation strategies to enable Sim2Real policy transfer from simulated environments to real-world scenarios, applies random transformation sequences to enhance synthetic depth images and uses auxiliary task learning to reduce the domain gap between synthetic and real images. 5) LCBC (Walke al. 2023), which employs ResNer-34 as the image encoder combined with MUSE language embeddings for robotic decision-making uses FiLM conditioning to embed language in- formation into the visual encoding, which in turn generates robot actions. 6) UniPi (Du al. 2024), which transforms decision decision-making problems into text-conditioned video generation tasks, produces future video sequences of the target task and extracts control actions from the generated videos. 7) HiP (Ajay et al.] 2024), an extension of the UniPi method, enhances the model's ability to handle long-horizon tasks by introducing hierarchical inference and planning. This approach decomposes tasks into high-level planning and low-level action generation, improving task execution in com- plex scenarios. 8) SuSIE (Black et al. 2023). which leverages a pretrained image-editing diffusion model to generate sub-goal images, guides the robot through complex manipulation tasks via lan- guage instructions. SuSIE integrates a large-scale internet visual corpus during sub-goal generation and achieves these generated sub-goat through a low-level gral-oriented policy.
15

---

Published as a conference paper at ICLR 2025
Instruction: Pick up the red cup. SR:0.8
Instruction: Put the smaller blue bowl into the red bowl. SR:0.8
Instruction: Take the figer oul of the red bowl, SR:0.6
Figure 7:
Real-world task results. Our method GEVRM can be effectively deployed in real-world scenarios, such as the picking and placing tasks of cups, bowls and tiger plush toys.
A.3 REAL- WORL- TASKS.
Protocol. To examine the effectiveness of the proposed GEVRM on real-world robotic manipulation tasks, we propose real-machine deployment protocol. We evaluate GEVRM on robotic arm URS for the pick-and-place tasks of a cup, a bowl, and a tiger plush toy. Specifically, we use camera capture third-pen hiird-person images as the observation space (image width 640, height 480), and relative poses and binarized gripper states as the action space (7 dimensions). The total number of collected real-world teleoperation expert trajectories is over 400, with trajectory lengths ranging from 20 to 120 steps and control frequency of 5Hz.
Experiments. We train and evaluate GEVRM under real-world protocols. The VAE and DiT the behavior planner trained for 30,000 and 12,000 iterations, respectively, while the goal-guided policy trained for 100,000 iterations. Other hyperparameters remain the same as in the experi- ments in CALVIN (and Bridge). Fig. sssows the policy execution process of proposed GEVRM on three types of real-world tasks, indicating that our method can be effectively deployed on real machines. In terms of task success rate (SR), evaluated each type of task 10 times. The exper- imental results show that compared with the grasping and placing of cups (or bowls) with regular shapes (success rate of about 0.8), the grasping of tiger plush toys with soft materials and irregular shapes is more challenging (success rate of about 0.6). Further improving GEVRM's perception of real-world scenes and task execution accuracy an important future work.
16

---

Published as a conference paper at ICLR 2025
(a) Goal State Embed. (w/o SA)
(b) Goal State Embed.
Buah the handle handle 3ose disere diawer
the switch burn on the light butb
turn oo led light
26
pink bock and rotaie 1tep the right
26
16
block
use switchltorg
turn on the light bulb
Esu pink black
Putte it the right
pul the handles open Bhe.draee
push the handle
4
bress the button to light
to clss the
-40
-10
Figure 8: Visual comparison the latent space representation of the goal image state with and without state alignment (SA). The gaol state representation sequence with SA also has better cluster centers, category boundaries, and temporal consistency.
Language Instruction: push the switch upwards
Conditioned Frume
Synthesized Frames
Oracle
GR-1
Ours
Figure 9: Comparison of goal generation. We visually compare Oracle, GR-1, and proposed algorithm on the "push the switch upwards" task in CALVIN Env. D. Compared with the baseline GR-1, the goal video we generate more realistic and better restores details.
17

---

Published as a conference paper at ICLR 2025
Language Instruction: go lowards the blue block the drawer and lift Synthesized Frames
Conditioned Frame
Oracle
GR-I
Ours
Figure 10: Comparison of goal generation. We visually compare Oracle, GR-1, and our proposed algorithm the "go towards the blue block in the drawer and lift it" task in CALVIN Env. D. Compared with the baseline GR-1, the goal video we generate is more realistic and better restores details.
Language Instruction: grasp the red block lying in the slider Conditioned Frame
Synthesized Frames
Oracle
GR-1
Ours
Comparison of goal generation. We visually compare Oracle, GR-1, and proposed algorithm on the "grasp the red block lying in the slider" task in CALVIN Env. D. Compared with the baseline GR-1, the goal video we generate more realistic and better restores details.
Figure 11:
18

---

Published as a conference paper at ICLR 2025
Language Instruction: go push the red block the right Frame
Synthesized Frames
Oracle
GR-1
Ours
Figure 12: Comparison of goal generation. We visually compare Oracle, GR-1, and our proposed algorithm on the "go push the red block the right" task CALVIN Env. D. Compared with the baseline GR-1, the goal video we generate is more realistic and better restores details.
Language Instruction; put stove and stuffedduck in pan. Conditioned
Synthesized Frames
Oracle
AVDC
Ours
Figure 13: Comparison of goal generation. We visually compare Oracle, AVDC, and our proposed algorithm the "put pan stove and put stuf edduck in pan" task in Bridge Data. Compared with the baseline AVDC, the goal video we generate is more realistic and better restores details.
19

---

Published as a conference paper at ICLR 2025
Language Instruction: put yellowpepper on plate and cookiebox pot or pan Synthesized Frames
Conditioned Frume
Oracle
AVDC
Ours
Figure 14: Comparison of goal generation. We visually compare Oracle, AVDC, and our proposed
algorithm on the "put yellowpepper on plate and cookiebox in pot or pan on stove" task Bridge Data. Compared with the baseline AVDC, the goal video we generate is more realistic and better restores details.
Language Instruction; put pot or pan on stove and put egg in pot or pan Conditioned Frame
Synthesized Frames
Oracle
AVDC
Ours
Comparison of goal generation. We visually compare Oracle, AVDC, and our proposed algorithm on the "put pot or pan on stove and put egg in pot or pan" task in Bridge Data. Compared with the baseline AVDC, the goal video we generate more realistie and better restores details.
Figure 15:
20

---

Published as a conference paper at ICLR 2025
Table 4: Generalization perturbed environments in CALVIN (train A, B, - perturbed test D).
No. of Instructions Chained
Perturbed Tasks
Algorithms
Avg. Length (1)
5
SuSIE
0.56
0.28
0.08
0.04
0.00
0.96
Image Shift
RohoFlamingo
0.48
0.32
0.12
0.00
0.00
0.92
GR-1
0.43
0.33
0.20
0.10
0.00
1.00
GEVRM (Ours)
0.52
0.40
0.08
0.00
0.00
1.00
SuSIE
0.48
0.16
0.08
0.00
0.00
0.72
Image Rotation
RoboFlamingo
0.42
0.24
0.11
0.02
0.02
0.82
GR-1
0.46
0.32
0.14
0.10
0.03
1.07
GEVRM (Ours)
0.32
0.12
0.08
0.04
1.16
SuSIE
0.72
0.36
0.16
0.12
0.08
1.44
Color Jitter
RoboPlamingo
0.52
0.08
0.04
0.94
GR-1
0.6
0.35
0.21
0.12
0.07
1.35
GEVRM (Ours)
0.64
0.48
0.32
0.12
0.08
1.64
SuSIE
0.72
0.48
0.32
0.32
0.24
2.08
Image Occlusions
RoboFlamingo
0.43
0.30
0.13
0.06
0.03
0.96
GR-
0.78
0.60
0.46
0.32
0.23
2.39
GEVRM (Ours)
0.92
0.68
0.48
0,24
0.20
2.52
SuSIE
0.32
0.04
0.00
0.00
0.00
0.36
Noise Interference
RoboFlamingo
0.49
0.23
0.03
0.01
0.01
0.80
GR-1
0.67
0.42
0.26
0.14
0.08
GEVRM (Ours)
0.80
0.48
0.32
0.12
0.04
1.76
SuSIE
0.56
0.26
0.13
0.10
0.06
1.11
Average
RoboFlamingo
0.63
0.35
0.18
0.09
0.05
1.31
GR-1
0.67
0.38
0.22
0.11
0.06
1,44
GEVRM (Ours)
0.70
0.47
0.26
0.11
0,07
1.62
Table 5: Zero-shot Generalization. The experiment is set up to train on data from environments A. B, and C (Fig. (a)), and D (Fig. (b)). reproduced version on static camera. Static camera: camera of fixed third-person view. Our proposed method can chain more instructions together with a higher success rate than all previous baseline methods. Baseline results are from previous work
Black et al. (023). The best results for each task are bolded.
No. of Instructions Chained
Algorithms
Source
1
3
5
HULC
Mees et al. 2022a
static camera 0.43
0.14
0.04
0.01
0.00
MCIL
Lynch Sermanet
2020
static camera
0.20
0.00
0.00
0.00
0.00
MdetrLC
(Kamath et al.
2021
static camera 0.69
0.38
0.20
0.07
0.04
AugLC Pashevich al.
static camera 0.69
0.43
0.22
0.09
0.05
LCBC Walke al. 2023
static camera 0.67
0.31
0.17
0.10
0.06
HiP
jay et al. 2024
statie camera 0.08
0.04
0.00
0.00
0.00
UniPi Du al. 2024
static camera 0.56
0.16
0.08
0.08
0.04
SuSIE
Black ei al. 2023
static camera 0.87
0.69
0.49
0.38
0,26
GR-1*
Black et al. 2023
static camera 0.75
0.45
0.2
0.15
0.1
GEVRM (Ours)
static camera
0.92
0.70
0.54
0.41
0.26
Sampling steps
Infer: time [s]
2
3
4
5
Avg. Length
50
0.598
0.80
0.48
0.32
0.12
0.04
1.76
40
0.501
0.73
0.53
0.20
0.13
0.06
1.67
30
0.379
0.73
0.40
0.23
0.20
0.06
1.63
20
0.260
0.71
0.46
0.22
0.11
0.08
1.60
10
0.135
0.77
0.47
0.17
0.15
0.10
1.67
Table 6: The comparative analysis of the computational efficiency and task success rate of the behavior planner (Noise Interference task). Due to the good properties of the adopted Rectified Flow, when the video sampling steps are reduced, the model inference time is greatly reduced, while the success rate significantly reduced.
21

---

Published as a conference paper at ICLR 2025
Algo
Control steps
Infer: time [s]
2
4
Avg. Length
0.077
0.80
0.48
0.32
0.12
0.04
1.76
2
0.044
0.85
0.50
0.20
0.15
0.05
1.75
DP
3
0.027
0.82
0.50
0.22
0.10
0.07
1.72
0.020
0.68
0.48
0.24
0.16
0.08
1.64
MLP
0.019
0.73
0.40
0.13
0.06
0.06
1.40
Table 7: Comparison of goal-guided diffusion policies (DP) with different open-loop control steps (Noise Interference task). The results show that the state-aligned policy has hetter action robustness, and increasing the number of open-loop control steps significantly reduce the inference time while having little effect on the task success rate. Therefore, the control frequency of our goal- guided diffusion policy can be maintained at the order of tens of Hz, which is sufficient for most robot manipulation tasks in reality. Moreover, when the number of open-loop control steps is 4, the diffusion policy has higher performance, while the inference speed is very close that of MLP.
Table 8: Behavior planner training hyperparameters.
Component
Parameter
Value
num_frame_toie
51
Dataset settings
cransform_nam
resize_crop
image size
(256, 256)
num_workers
wwm.bbucke..biild.wwrk...
16
Acceleration settings
dtype plugin
bfl6
zero2
type qk.norm enable flash atin enable layernorm kernel freeze.y.embedder
STDIT3-XL/2
True
DIT Model settings
True
True
True
micro frame size
17
VAE settings
micro batch size
type
T5
Text encoder settings
model saadellmax_eeee length
300
shardformer
True
type
rflow
Scheduler settings
www.timestee.transfiee
True
sample method
logit-normal
random
0.025
intepolate
0.025
guarter.random random
0.025
quarter.hear head
0.75
quarter_tar tail
0.025
Random Mask settings
quarter_hee head tail
0.05
image_random image_head
0.0
0.025
image_tail
0,025
image head tail
0.05
batch_size grad_clip learning rate ema decay adam_ep warmup_steps
1.0
le-4
Optimization settings
0.99
le-15
1000
22

---

Published as a conference paper at ICLR 2025
Table 9: GEVRM test hyperparameters.
Component
Parameter
Value
fixed inteenneelllni number
20
condition length
5
wel_geeerationnnummbe
51
micro_frame firrofframe_siee size
17
womm_saplling_teee
50
General settings
resolution aspect ratio image_size
256
1:1
(256, 256)
fps
30 for CALVIN; 5 for Bridge
frame
dtype
bf16
Table 10: Goal-guided policy training hyperparameters.
Component
Parameter
Value
scheuuee
Cosine
diffusion stepss
20
action_sample
General settings
lastst_
0
learning rate
3c-4
warmup steps
2000
actor_decay_seee steps
2e6
time_dim
32
num blocks
3
Score network settings
dropout rate rate
0.1
hidden dim
256
use_layer_nomm
True
hidden dim
512
SA Encoder settings
num prototype temperature
3000
0.1
23


Reviews:
[{"review_id": "bQuwsUK19U", "reviewer": "Reviewer_TJEW", "summary": "This paper introduces GEVRM, a method for robust visual manipulation through video generation. It trains a video generator for goal generation, and a goal-conditioned behavioral cloning policy. The video generator takes a sequence of observations and a language instruction, and predicts the subsequent image observations as the goal. The policy head is a goal-conditioned diffusion policy. GEVRM uses an auxiliary state alignment loss to improve learned representations. Results show that GEVRM outperforms previous methods on Bridge and CALVIN data, and is more robust to perturbations and generalizable than baselines in the simulated environment CALVIN.", "questions": "- Baseline (video generation): is AVDC/GR-1/GEVRM trained with data augmentation?\n- Baseline (CALVIN rollout): how does GEVRM compare to HiP/UniPi/GR-1/behavioral cloning with data augmentation?\n- Ablation: how does setting the state alignment loss $\\lambda = 0$ affect task success rate? Quantitative results in addition to the t-SNE plot would be convincing.\n- Ablation/Clarification: video generation for control and goal-conditioned behavioral cloning are relatively well-studied problems. Could you clarify the key contribution of GEVRM? (How is GEVRM different from prior work in its architecture, and what design choices contributes to the improved performance?) A more comprehensive ablation studying the importance of each component in GEVRM would be convincing to see.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Video generation for goal-conditioned behavioral cloning is well-motivated.\n- GEVRM outperforms previous methods on Bridge and CALVIN for video generation.\n- GEVRM outperforms HiP, UniPi, and GR-1 on generalization in CALVIN, and outperforms SuSIE on generalization with visual perturbations.", "weaknesses": "- Limited/inconsistent evaluation. GEVRM is compared with AVDC on Bridge and with GR-1 on CALVIN for video generation, with HiP/UniPi/GR-1 on unperturbed generalization, and with SuSIE on perturbed generalization. It would be good to see GEVRM compared with a consistent set of baselines for video generation and policy rollouts in a few more environments.\n- If the core problem setting is robust manipulation, it would be convincing to see GEVRM compared with HiP, UniPi, GR-1, and pure behavioral cloning, each with data augmentation.\n- It is unclear what the key contributions of GEVRM are and why it outperforms prior work. A more comprehensive ablation with downstream task success rate could justify each component in the design of GEVRM and clarify its improvement over prior state of the art.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "thwiYtWWPe", "reviewer": "Reviewer_siH9", "summary": "This paper presents a goal-guided policy conditioned on future predictions generated by a video diffusion model. The authors try to integrate the classic principle of internal model control (IMC) into visuomotor control, thereby enhancing the robot's capability to resist environmental perturbations.", "questions": "1. The integration of a video diffusion model might introduce considerable computational overhead. However, the paper lacks a comprehensive analysis of the computational requirements. How about the inference efficiency in Calvin simulation environment?\n2. On the goal-conditioned policy, the authors propose a diffusion policy conditioned on the generated goals. In such goal-conditioned policy framework, does the diffusion policy exhibit significant superiority compared to a simple MLP?\n3. In Table 3, the generalization experiments are conducted in perturbed environments. How do the other baseline methods, apart from Susie, perform in this setting?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method appears to be reasonably designed. It utilizes a video diffusion model as a visual planner to generate future predictions that guide the policy in action output. In terms of detailed design, the proposed method enhances the generation consistency through efficient video spatiotemporal compression and random mask strategies. Furthermore, the authors utilize prototypical contrastive learning to align the goal states with its current state, thereby enabling the model to implicitly infer and distinguish perturbations from the external environment.\n2. The authors propose generalization experiments on perturbed environments in CALVIN (train A, B, C → perturbed test D), which can assist the research community in examining the the robustness of policies against external environmental disturbances.", "weaknesses": "1. This paper did not undertake real-world robotic experiments. Instead, it focused exclusively on analyzing video prediction quality using the Bridge dataset and conducting action execution tests within the simulated environment of the Calvin benchmark. However, the effectiveness of GEVRM on real-world robots was not validated.\n2. In Table 2, the authors compare their proposed GEVRM method with other baseline approaches within the CALVIN ABC-D environment, utilizing the settings of RGB input only. However, they did not involve powerful baseline methods for comparison. While GR-1 is a comparatively strong baseline, its reproduced performance is notably low. Could the authors provide further details regarding the reproduction of GR-1? Furthermore, would it be possible for the authors to incorporate additional powerful methods for comparison by adjusting their input modalities to RGB only? Additionally, I would like to mention that Susie's results on Calvin ABC-D should be included in Table 2.\n3. The authors should give more credits to the paper \"Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation\" (https://arxiv.org/abs/2409.09016). The design philosophy behind GEVRM and the presentation of the article, such as the layout in Figure 1, seem to be inspired by this paper; however, it is not mentioned in the introduction or related works sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2vFLF11W9", "reviewer": "Reviewer_9iWM", "summary": "This paper considers the problem of vision-language-action model for robotics. The proposed approach leverages a text-video generation model to generate the goal images conditioned the current observation and task description. Then the action policy leverages the generated goal and the current state to predict the robot action. To aggress the problem of disturbance encountered during deployment, the paper leverages the idea of IMC from classic control by using internal embeddings (optimized through prototype contrastive learning) as the policy model input.", "questions": "Please see my questions above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper considers an important topic in VLM for robotics. The paper is well-written, and the logic is clear. The proposed approach outperforms the baseline in terms of goal generation when the input image is perturbated and in terms of closed-loop eval in simulations.", "weaknesses": "The connection to IMC. The IMC in control uses “diff” between the predicted output from the process model and the actual output from the actual robot, induced by the same action, to generate the feedback signal to compensate the controller. But the proposed approach seems to use the current robot state and the desired output to generate the control signal. I think the proposed structure is just a goal conditioned policy, and I don’t see the strong connection to the IMC.\n\n\nImage encoder. The image encoder is trained through the discriminating if a goal image and the current state image are from the same trajectory. The experiment is missing ablation studies on the encoder. Would be interested in how an off-the shelf encoder like DinoV2 performs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zkyHlHHqnL", "reviewer": "Reviewer_Ehog", "summary": "This paper introduces GEVRM, a novel close-loop VLA model that adopts the Internal Model Control principle to a learning-based framework and improves robustness to perturbations from external environments. This method leverages text-conditioned video generation to infer image goal states, aligns the current and goal states through prototypical contrast learning, and performs goal-guided action prediction using a diffusion policy. Evaluation results demonstrate the strong generalization capability of image goal generation across simulated and real-world datasets, as well as the robustness through high task success rates under external perturbations in the visual manipulation task.", "questions": "1. When GEVRM uses prototypical contrastive learning for state alignment, the positive pairs become the current and goal states from the same trajectory, instead of different augmented views from the same image. I wonder if this contrastive learning process will learn to focus on only trivial consistent features in the background, which are likely to be similar across all timesteps, and ignore some differences (e.g. the position change of objects or the robot arm) between the current and goal states that are essential to help the model make correct decisions.\n2. Would the input states be perturbed by any transformations while training the state alignment encoders? If not, are these encoders directly used to process the perturbed images during inference?\n3. During the testing time in CALVIN evaluation, apart from the visual discrepancy of the environments, are the instructions for each (sub)task seen during training, or are novel instructions tested to demonstrate task generalization capabilities?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-motivated, clearly written, and easy to follow.\n2. The paper properly discussed the related work that leverages text-conditioned video models to generate visual plans for decision-making.\n3. GEVRM innovatively uses several techniques to improve the quality of generated visual goals and aligns state representations to enhance the robustness under perturbations\n4. The authors provide extensive evaluation results on both behavior planning quality and task completion in perturbed environments, showing the effectiveness of the proposed method.", "weaknesses": "1. The underlying framework – composing a text-conditioned video model and an action predictor has been widely explored by previous work, in which most works adopt a relatively lightweight method to extract inverse dynamics. This work employs a diffusion policy for action prediction and interacts with the environment in a close-loop manner, which may help learn multi-modal action distribution but can be time-consuming and often not desirable in real-time deployment.\n2. The thoroughness of behavior planner evaluation can be further improved, for example, the effectiveness of techniques like random masking compared to simply masking out the last few frames for decision-making scenarios, should be supported by ablations.\n3. It would be great if the authors could thoroughly discuss the limitations of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)