Empowering World Models with Reflection for Embodied Video Prediction
Authors: Xiaowei Chi
Chun-Kai Fan
Hengyuan Zhang
Xingqun Qi
Rongyu Zhang
Anthony Chen
Chi-Min Chan
Wei Xue
Qifeng Liu
Shanghang Zhang
Yike Guo
Venue: ICML 2025 poster (2025)
Keywords: World Model
Generation Model
Robotics
Primary area: applications->robotics

Abstract: Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios. However, existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios.  To address this challenge, we propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction.  It leverages the complementary strengths of pre-trained vision-language and video generation models, enabling them to function as a world model in embodied scenarios. To support RoG, we introduce Embodied Video Anticipation Benchmark(EVA-Bench), a comprehensive benchmark that evaluates embodied world models across diverse tasks and scenarios, utilizing both in-domain and OOD datasets. Building on this foundation, we devise a world model, Embodied Video Anticipator (EVA), that follows a multistage training paradigm to generate high-fidelity video frames and apply an autoregressive strategy to enable adaptive generalization for longer video sequences. Extensive experiments demonstrate the efficacy of EVA in various downstream tasks like video generation and robotics, thereby paving the way for large-scale pre-trained models in real-world video prediction applications. The video demos are available at https://sites.google.com/view/icml-eva.

Empowering World Models with Reflection for Embodied Video Prediction
Xiaowei Chi
Chun-Kai Fan
Hengyuan Zhang *2
Xingqun Qi' Rongyu Zhang? Anthony Chen
Chi-Min Chan
Wei Xue
Qifeng Liu'
Shanghang Zhang
832
Yike Guo 23
Abstract
Observation Encode Module
CLIP
Isput: Obsorval
Output
Understand
Video generation models have made significant progress in simulating future states, showcas- ing their potential as world simulators em- bodied scenarios. However, existing models of- ten lack robust understanding, limiting their abil- ity to perform multi-step predictions handle Out-of-Distributiinn (OOD) scenarios. To ad- dress this challenge, we propose the Reflection of Generation (RoG), a set of intermediate reason- ing strategies designed to enhance video predic- tion. It leverages the complementary strengths of pre-trained vision-language and video gen- eration models, enabling them to function world model in embodied scenarios. To support RoG, we introduce Embodied Video Anticipa- tion (enAhmaar((VA--Beneh)) a comprehensive benchmark that evaluates embodied world mod- els across diverse tasks and scenarios, utilizing both in-domain and OOD datasets. Building on this foundation, we devise world model, Em- bodied Video Anticipator (EVA), that follows a multistage training paradigm to generate high- fidelity video frames and apply an autoregres- sive strategy to enable adaptive generalization for longer video sequences. Extensive experi- ments demonstrate the efficacy of EVA in vari- ous downstream tasks like video generation and robotics, thereby paving the way for large-scale pre-trained models in real-world video prediction applications. The video demos are available at https://sites..goon
prediction
What's Next?
Finish?
Olservation
State
Observation Prediction Module
Raflaction
Video Gencration Model
Action
Generation
Prediction
Action Monel
Pose Medel
How to
Text Tokenizer
Figure I. The illustration of the Reflection of Generation RoG). Giving instruction and observation as input, the world model gives proper output with the combination of understanding, pre- diction, and generation.
1. Introduction
The rapid development of video generation technologies has brought predictive models to the forefront of building embodied world simulators (Agarwal et al., 2025), where the ability to anticipate future status is critical in embodied scenarios (Wang et al., 2024b). These generated predic- tive videos can serve as interactive guidelines, mixed-reality product manuals, driving instructors (Gao et al., 2024; Wang et al., 2023b), gaming assistants (Bruce et al., 2024), even a robot's 's planning imagination (Du et al., 2023a; Yang et al., 2023). By imagining how the world evolves as an agent behaves, this prediction capability significantly enhances decision-making in embodied scenarios, providing foun- dation for robust world models (Ha & Schmidhuber, 2018).
However, existing video generation models for robotics (Zhou et al., 2024; Yang et al., 2023) pri- marily focus on conditional simulation, neglecting the critical role of understanding both current and predicted states. Therefore, these models are limited to generating fixed-frame sequences and single-round predictions. Moreover, they lack the ability to detect flawed outputs, relying heavily human intervention. These limitations severely degrade their performance in (OOD) scenarios and undermine the fundamental goal of an embodied world simulator.
'qqual contribution. The research was supported by Theme- based Research Scheme (T45- 205/21-N) from Hong Kong RGC, NSFC (No. 62206234), the National Natural Science Foundation of China (6247601 1), and Generative Al Research and Develop- ment Centre from InnolIK. 'Hong Kong University of Science and Technology "State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University. Cor- respondence to: Shanghang Zhang <shanghang& @ www..du..n.... Yike Guo <yikeguo@ ust.hk>..
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
To address these challenges, we propose Reflection-of- Generation (RoG), inspired by recent advances in reasoning

---

Observation
Embodied Video Anticipator
Adopter
Video Prediction
Rabot
Video Generator
Llama 2 7B
Real
Adopter
How (action) What next? APTV
Query Embed
Text Decoder Follow this [action)
Encoder
Meta- Action-Description
Meta-Task 2: Finish-Thinking
D
doing?
shuntly dostibe the
frished fobling
sweatalin??
should follnwing video You
You are pulting oninge a white bowl
Add back pepper the cooked egu
Has firished reaching the left ewestchin?
Meta- Task 3: How-To
diawer?
need follow
Aoi place
du like vidho.
this valen.
side tuble
Meta-Task Next-Step
What
chips top
picked yellow wipe packec
You noad to the sable. Junt this guide.
guide
Figure 2. Meta-tasks of the embodied-vides prediction. We present four meta-tasks, including Action- Oetion-Descriptions Pinish-Thinken How-To, and Next-Step, for embodied vidco anticipation and build the related dataset, benchmark, and model.
and alignment (Wei et al., 2022; Liu et al., 2024b; Kawa- harazuka et al., 2024). As shown in Figure 1, RoG inte- grates intermediate reasoning steps into the video genera- tion process, enabling iterative self-correction and deeper understanding. It separates the understanding and genera- tion tasks, allowing for modular and interpretable design. Therefore, RoG can combine any Visual Language Model (VLM) and Video Generation Model (VDM) into uni- fied world model. By introducing reflection as a critical step, RoG empowers the world model iteratively refine its predictions, and generate longer video sequences.
Building on this benchmark, we propose EVA (Embodied Video Anticipator), a unified understanding and generation model tailored to RoG. EVA employs chunk-wise autoregressive paradigm, integrating reflection conditions adaptively extend the length of generated video sequences. By supervising the generation process with VLMs. EVA achieves consistent and self-correcting video generation, enhancing the interaction between humans and machines in embodied scenarios.
In summary, our contributions are as follows:
We propose RoG, novel strategy that integrates intermediate reasoning steps into video generation, enabling self-correction and deeper understanding in embodied scenarios.
To facilitate the training and evaluation of RoG, intro- duce EVA-Bench, comprehensive benchmark designed for embodied video anticipation. EVA-Bench decompose the RoG within embodied scenes into four meta-tasks: Action- Description, How-To, Finish-Thinkin Next-Step, as illus- trated Figure 2. These meta-tasks simplify data collection and evaluation by focusing both understanding and gen- eration aspects.
-VA-Benchmark: We create EVA-Bende benchmark that evaluates world models across diverse embodied tasks with in-domain and OOD scenarios, using both under- standing and generation metrics.
includes both in-domain and out-of-distribution videos, leveraging diverse datasets from egocentric and robotics scenarios. It evaluates models us- ing combination of Video AeastioonAAswerriin (VQA) metrics and pixel-level video prediction metrics.
Embodied Video Aiticipator: We design EVA, a unified understanding and generation world model that gener- ates longer, more consistent videos through chunk wise autoregressive reflection.
2

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Extensive experiments on EVA-Bench highlight the strong performance of RoG in both in-domain and OOI) tasks. Fur- thermore, to validate the applicability of EVA in robot plan- ning, we evaluate the model using a robot simulator (Mees et al., 2022; Brohan et al., 2022), demonstrating that EVA and RoG effectively support real-world task execution.
frame-level prediction, task-level prediction, and serial task-level prediction.
hierarchical levels of complexity:
Frame-Level Prediction:
The objective at this level is to
generate subsequent video frames Oz(t 2 1), conditioned on the initial observation Og- This formulates a next-frame prediction problem, where the model approximates the dis- tribution: P(O,|Oo. I), = 1.2.....
2. Related Work
Task-Level Prediction: Task-level prediction focuses on achieving high-level instruction 1, such as "pick up the book". The goal is to model the sequence of predictions {Or} required to complete the task: P({O,)-=1 I) Here, T represents the time horizon necessary to complete the task. This level abstracts away low-level frame transi- tions, instead emphasizing coherent task execution.
Video Generation With the advent of diffusion-basee vi- sual generation models, there has been significant progress in extending the capabilities of video generation. For in- stance, models like VideoCrafter (Chen et al., 2023; 2024) and VideoPoet (Kondratyuk et al., 2023) have demonstrated impressive abilities in generating high-quality video. More- over, video generation models with image conditions like Dynamicrafter (Xing al., 2023), Stable Video Diffu- sion (Blattmann et al., 2023) and Animatediff (Guo et al., 2023) meet impressive generation quality and have already been used in many areas. Such weakness also happens in some long video generation methods (Wang et al., 2023a; Yin et al., 2023). These video-generatio soning abilities and still struggle with consistency and under- standing, recent works also combine generation models with LLMs (He et al., 2024).
Serial Task-Level Prediction: The more complex level in- volves sequentially completing multiple sub-tasks to fulfill broader goal. For instance, instruct "clean the table", the model must decompose it into sub-tasks (e.g.. "remove ob- jects", "wipe the surface") and execute them sequentially.
models lack rea-
Task-Level Prediction via a Unified World Model: This paper focuses on task-level prediction as the core of the video prediction task, bridging frame-level generation and task reasoning. The central objective of task-level prediction is to simulate the state of the world, which is the role of the world model, denoted by W. Formally:
Embodied World Model World models aim to provide future predictions based on current observations. This con- cept has been explored in various domains, including Ge- nie (Bruce et al., 2024), which shows interesting ability in gaming simulation, Vista (Gao et al., 2024; Wang et al., 2023b), etc., in autonomous driving. Video prediction is special world-model-like task, Seer (Gu et al., 2024), AID (Xing et al., 2024) adapting image-to-video generation model to predict the motion of future frames. Additionally, world models such as RoboDreamer (Zhou et al., 2024), Enerverse (Huang et al., 2025) and AVDC (Ko et al., 2023) have been utilized video generation as robot simulators. Unisim (Yang al., 2023), for instance, combining pre- trained web-scale data with embodied videos expands world models' applications. VLP (Du et al., 2023b) integrated language and video generation models for robot planning but stayed at the concept level. A task- video predictor still needed.
W(Oo. 1) R P((0.)T=1100,1
The world model processes Og and I, dynamically com- bining frame generation and task reasoning to generate the sequence {O,}. The predictions are refined iteratively until the task requirements are met. This hierarchical approach en- sures that task-level prediction incorporates both low-level frame synthesis and high-level task satisfaction, addressing the demands of complex video prediction tasks.
3.2. Reflection of Generation World Model
The RoG world model introduces a dynamic feedback mech- anism that integrates an understanding module and a gener- ation module to iteratively refine task-level predictions, as shown in Figure The model defined as follows:
3. Reflection-of-Geeeraee
WROG (Op, I) Reflect(H) P({O.) 1, H) (2)
In this section, we introduce the RoG by first defining hierar- chical levels of visual prediction and then the basic process of the RoG world model.
Where {O.]T_ is a sequence of predicted observations, I is the encoded hidden state representing the prediction sta- tus, Reflect() is the understanding module, which provides three types of responses: extend, regenerate, or output. The whole inference process is as shown in Algorithm 1. The algorithm iteratively refines a prediction sequence {Ot} us- ing a reflection mechanism. It begins by encoding the initial
3.1. Hierarchy of Video Prediction
The video prediction task, begin with an initial observation On at time step 0 and a task instruction 7, is categorized into
3

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Input
Output
Reflection
Obs Frame
Moving the chips near the up.
Instruction: Haw ...
Task Completion Judgement No
Embodied Video Anticipator
Task Completion
Moving the chips near the up.
Self-Ask Prompt: Is Task Finish?
Prediction Frame
Judgement: Yes
Heweeen Figme
Figure 3. RoG inference pipeline of EVA with chunk wise frame extension. Given the visual observation and human questions input, EVA would first generate fixed frames of vidcos and related text answers. Then, the model prompts itself to check the task completion status; if the predicted vidco is not finished, EVA keeps generating the extended frames until the task completion judgment is true.
Algorithm 1 Reflection of Generation World Model (WRoG)
Action-Description:: Evaluate the model's understanding ability by generating concise textual descriptions (subject + verb + location + destination). Metrics include text similarity, GPT-4 (OpenAI, 2024) keyword matching, and CLIP (Radford et al., 2021) score for description quality.
Input: Initial observation Og, task instruction
Output: Sequence of predictions {0t)e=
Initialize: Prediction sequence {O,} t- 0
repeat
+ Enncod(Oo.
# Understanding Module
Finish-Thinking This task assesses whether the frame- level prediction should be extended complete the task. Metries include the accuracy of binary outputs ("Yes"/"No")) We introduce the Goal Completion Estimation (GCE) to compare the generated frame with the ground truth, Fréchet Video Distance (Unterthiner et al., 2018) multiple video gen- eration quality metrics from VBench (Huang et al., 2024).
encodes input states
f/ Reflect(H H) Generate reflection output based
H
if V Extend (O+)) then
Extend the prediction sequence {O,}
else if V = Regenerate( {O+}) then
Regenerate the prediction sequence {Ot}
else if V = Output( (O,)) then
This task transforms task instructions into visual outputs and text responses, evaluated by video metrics and text metrics. The combined score of language and video forms the EVA-Score for How-To tasks.
How-To:
Finalize and output the prediction sequence {Ot}
break
end if until Prediction sequence converges breaks
Predict the next action in video sequences and language instructions. The metrics are the same as for How- To, including the accuracy of the QA the task level and the correlation between the intermodality between the predicted frames and the ground truth.
Next-Step:
observation On, task instruction I, and current predictions into a high-dimensional representation H.
Based on H, the reflection mechanism determines whether extend the sequence by generating new frames, regenerate flawed predictions, finalize the sequence if it is complete. This process continues until the predictions converge or meet the task's 'ss stopping criteria, thereby ensuring adaptive and accurate video generation in embodied scenarios.
4. Embodied Video Anticipator
As we formulate the problem as a video prediction task as equation we propose the world model EVA. EVA includes two main pre-trained models for multimodal prediction, multistage training strategy, and uses Ensemble method for domain-specine LoRA (Hu et al., 2021) for VDM and interaction tokens to achieve this complex visual prediction task. We describe these key elements in detail in the following section.
3.3. RoG World Model Task Decomposition
The RoG world model is evaluated through four sub-tasks with corresponding metrics, decomposing video prediction into actionable components for a comprehensive assessment. The tasks are follows:

---

Empowering World Models with Reflection for Embodied Vidco Prediction
4.1. Model Structure
round of generation, producing an extended video ((++1), This process iterates, concatenating successive predictions (v(0), o(.,...., ((k) until the VLM determines that the task is complete. By iteratively refining predictions and focus- ing only on keyframes for extension, this method ensures coherent and task-aligned video generation.
EVA integrates a 7B VLM backbone and a 1.5B VDM to enable high-quality video generation for diverse tasks. The VLM backbone combines CLIP (ViT-L/14) visual encoder with the Vicuna-v1.5 .5 language model, using a parameter- free token clustering algorithm to reduce video token over- head. Special interaction tokens, such as < image > and < IMGp >, bridge visual and textual inputs, aligning with task-specific processing. The VDM generates 16- frame videos in 2 seconds by conditioning on image, FPS, and language embeddings, leveraging temporal and spatial transformers pre-trained on large-scale scale video datasets. To enhance adaptability, the Ensemble-LoRe method trains low-rank adaptation modules for each domain (e.g., human- egocentric, real-robot). It combines them using a task token- controlled gating system, ensuring efficient multi-dome generalization. Additionally, cross-attention generation adapter aligns VLM hidden features with VDM text embed- dings via linear transformation, optimized using diffusion denoising loss for superior video generation quality. De- tailed module structure is described in Appendix A.
5. Embodied Video Anticipation Benchmark
Under the RoG setting, we divide the complex interleaved multimodal generation task into four sub-tasks. This section describes how we separate the tasks and introduces our pro- posed dataset and benchmark. The fine-graines description of each metric and benchmark composition provided in Appendix C.
5.1. Dataset Construction
We construct four datasets for embodied tasks. The How- To and Action-Description datasets are built by extending prompts using GPT-4o (OpenAI, 2024) and standardizing annotations into a subject-verb-objeee format, based on the structure. For the Finish-Think dataset, we use the first 25% of videos to identify unfinished tasks and convert relevant Robo VQA (Sermanet et al., 2024) questions into this format. The Next-Step dataset is created using key step annotations from the Ego-Exo4D (Song et al., 2024) dataset and se- quential task annotations from Robo VQA (Sermanet et al., 2024), focusing on ordered steps for next-step predictions. The complete EVA instruction tuning dataset comprises 500K QA pairs. Detailed information, including prompt structures, dataset ratios, and data examples, is provided in Appendix and anonymous supplementary pages.
4.2. Multi-stage Training Strategy
To train EVA, we employ a three-stage process. In the first stage, we use COCO (Chen et al., 2015) and a curate subset of CC3M-595K (Sharma et al., 2018) to warm up the visual encoder adapter in VLM. The second stage involves aligning the VLM with an instruction tuning dataset, transforming it into embodied QA bot using multimodal instruction data from sources open-domain data e.g. VideoChatGPT (Maaz et al., 2024), along with our EVA instruction tuning dataset. In the final stage, we adapt the entire pipeline and train the Siremble-LoRA, for each
generation adapter and VDM's task. This stage addresses the diverse visual distribution among UVAA-Instuction datasets by tuning the Ensemble- LoRA and adapter to maximize generation quality. We introduce detailed information on training stages is provided in the Appendix A.
5.2. Benchmark Construction
To ensure comprehensive evaluation, EVA-Bench includes 125 meticulously curated high-quality samples from the datasets introduced above, spanning diverse domains such as real-world robots, simulated robots, and egocentric human daily activities. These samples cover a wide range of sce- narios, including pick-and-plaa tasks, cooking, bike repair, COVID testing, and indoor organization, providing a bal- anced distribution of tasks. For Finish-Thinkin frames, we manually annotate the task completion frames. For out-of- distribution (OOD) evaluation in robotics, experts carefully annotate the prompts. Additionally, we rigorously annotate and refine all 125 samples and their corresponding prompts to ensure accuracy and consistency, making EVA-Bench robust benchmark for embodied video anticipation.
4.3. Autoregressive Frame Extention
The chunk-wise autoregressive frame extension method en- ables iterative video generation based on task completion verification. Given an input frame on E 18 (B)) the model first processes it through the VLM to obtain a hid- den language state, which is passed to the generation adapter and used as a condition for the VDM to generate an initial video (U) E (BXXXXCXXXW) After each generation step, the model employs RoG by prefixed prompts as shown in Figure 3 to verify whether the generated video satisfies the task's completion requirement. If the task is incom- (k)
6. Experiments
plete, the model extracts the last 1 and t frames ("1-1,7-1) from the current output and uses them as input for the next
In this section, we give a comprehensive experiment to evaluate the multimodal understanding and generation abil-
5

---

Empowering World Models with Reflection for Embodied Vidco Prediction
on: How to move brown chip bag near wite can
Move brown chip bag near wite can.
Reflection Prompt: Task
Completion Judgement
RoG
Reflection Prompt: Finish
Task Completion Judgement Yes
Instruction: This video depicts how to [Move the chip to wite can]. The current goal is [Move Chips to up can]. What is likely to happen next?
Move Chips near the 7 up can.
Reflection Prompt: Is Task Finish
Task Completion ludgement
RoG
This video depicts how hands], What likely happen
Wipe hand.
Reflection Prompt: Is Task Finish ?
Task Completion Judgement Yes
RoG
Figure Visualization results of the How-To, Next-Step, and Finsh-Tiie moish-Thinking. Starting from random statue, EVA can generate robot motion and human-ego motion according the instructions. The first two continuous show the EVA; the last example, EVA can generate video based on its reasoning results. We include more example results on the demo page and the Appendix B.
generation ability of
ity of EVA four meta-tasks. First, VQA reasoning abilities the oetion-Descriplie task in Section 6.1, highlighting the knowledge gaps in existing VLMs regarding embodied scenes. Next, we provide the comparison of generation performance in Section 6.2. Sections 6.3 we compare EVA against four baselines us- ing EVA-Score to represent the capabilities when facing How-To and Next-Step tasks. Last, demonstrate how the world model performs in robot tasks in 6.4. Both qualita- tive and quantitative results are presented throughout, with additional qualitative analyses available in Appendix and page https::/sittes. www.ge..oo//viiee......
EVA's
videos visual input. Data sources summarized and more details included in Appendix A.7.
Datasets for Evaluation. EVA-Bench includes a curated collection of 125 high-quality samples from our EVA- Instruct dataset. These samples encompass real-world robots, simulated robots, and egocentric human daily activ- ities. The benchmark is categorized based meta-tasks. details are included in Appendix C.
Implementation details. We set up two kinds of models, EVA-Generator and EVA. EVA-Generator uses Dynami- crafter as the backbone and fully fine-tunes it on the EVA- Instruct. EVA constructs an end-to-end pipeline with fine- tuned ChatUniVi as our VLM backbone and WVA-eenerrtor generation adapter to align the feature embedding among two back- bones and only train the adapter with the EVA-Instruct. We also implemented discrect version of EVA EVA-2stage by using the VL.M and VDM EVA without adapters.
EVA Instruction Tuning Dataset(EVA-Instruee encom- passes four tasks featuring videos of humans, real-world robots, and simulated robots. The complete EVA instruction- tuning dataset consists of 500K QA pairs sourced from Open-X X-ie-md-mmoddettt (Padalkar et al., 2023), Ego4d (Grau- man et al., 2022), Ego-Exo4d (Grauman et al., 2024), and CALVIN (Mees et al., 2022). The EVA-Instruct is presented conversational format, paired with single images and
VDM backbone, in the middle,
6

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Table AAtionn-eseeeee DesiripDesciiiioon results in comparison of VLM. We compare the open-source VLM models. The QA prompts for each model are included in the appendix. In this table, blue means the best- untrained model.
Model
In-Domain
BLUE-It
METEORT
R-L7
CIDEr
Spicet
CLIP
GPT-4of
ChatUniVi (Jin et al., 2024)
x
0.0969
0.0640
0.1497
0,0427
0.0636
27.49
9.03
LLAVA-NI (Li et al,, 2024b)
0.0725
0.0741
0,1174
0.0843
0.0982
29.63
26.94
LLAVA-NV (Zhang el al.. 2024c)
0.0717
0,0642
0,1062
0.1267
0.0961
30.36
25.56
LLAVA-O (Li et al., 2024a)
0.0874
0.0591
0.1118
0.2172
0.1043
27.97
22.35
Miniepmv2 (Hu al., 2024)
0.0672
0.0572
0.0913
0.0404
0.0456
28.88
17.63
Qwen2 (Yang et al., 2024)
0,2484
0.1434
0.3255
0.8914
0.28399
28.98
29.58
GPT-4o (OpenAI, 2024)
X
0,2651
0.1671
0,2902
0.7355
0,3015
22.96
33.19
ChatUniVi-llRA
0.3007
0.1054
0.3268
0.8245
0.2213
24.89
31.94
ChatUniVi-FP
0.4105
0.1809
0,4416
1,9012
0,3414
25.36
38.46
EVA
0.5735
0.3095
0.5873
4.0139
0.5506
24.98
62.63
Table 2. Finish-Thinking Video Generation Quality Compar-
6.1. Video Question Answering
ison. Subject Consistency(SC). Background Consistency(BC) Motion (M))thn Goal Completion Estimation(GCE.
E EVA-Language Metrics. First, we present the QA task
results from our EVA-Bench. Comparing the BLEU (Pa-
Fréchet Video Distance(FVD). Blue means second-best best results.
pineni et al., 2002). METEOR (Banerjee & Lavie, 2005), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016). and CLIP (Radford et al., 2021) scores as multimodal measures. Furthermore, to have a better word analysis, we use GPT-4o (OpenAI, 2024) as an automatic evaluator to obtain a GPT-4o score. The detailed metric description is included in Appendix A.7.
Model
Inpul
SC
BC
MS f
GCE1
FVDJ
DC (Xing al., 2023)
I-T
87.25
91.91
96.72
80.96
362.56
DC-Tume
1+T
83.49
89.70
97.87
80.72
235.52
Evn
I-T
95.74
95.11
99.09
86.83
177.28
ChatUniVi+DDC
V
87.10
90.82
96.97
80.80
314.11
Qwen2-DC
V
87.13
91.39
96.29
81,96
307.33
ChatUniVi+EVA.coo
V
96.54
95.26
99.19
189.61
Twen2-EVA-Gea
V
96.13
95.48
99.15
85.87
193,89
LLAVA-O+EVA-Grr
V
96.64
95.54
99.17
88.74
192.83
EVA 2Stage
V
96.68
95.82
99.17
90.19
185,89
Main Results. In Table we make the model fine-tuning on EVA-instruct as in-momain and compare a serial of zero-shot VLMs. Qwen2 (Yang et al., 2024) and GPT- 4o excel in zero-shot inference. Under GPT-4o's evalua- tion, while Qwen2 remains the best-performing open-source model with a score of 29.58, the gap between the LLaVA- NeXT (Liu et al., 2024a) series and Qwen2 has narrowed.
EVA
V 97.11
96.01
99.31
89.09
184.81
Table 3. How-To and Next-Step Task generation evaluation result the EVA- -YA-Bench.
Task
Model
EVAS-L
EVAS-V 1 EVA-Score
LLAVA-O+EVA-C AeGn
33.81
67.41
50.61
Qwen2+EVA
41.54
69.34
55.44
HOW-TO
EVA-2Stuge
85.51
73.32
79.42
Fine-tuned models are trained on a 50K subset of our EVA instruction dataset by LoRA, full- (ull-paramater(FP)).. Notably, our EVA model achieves the best performance with a score of 62.63 under GPT-4o's evaluation. demonstrating that the diverse data and multi-stage training strategy used in EVA surpass fine-tuning on pre-trained weights.
EVA
85,51
77.83
81.67
LLAVA-O+EVA-GE
16.75
54,19
35.47
Wven++EVA-Gen
42.99
60.11
51.55
Next-Step
EVA-2Stage
73.02
64.46
68.74
EVA
73.02
68.68
70.85
model with the understanding module under the RoG setting. we compared series of 2-stage VIMM+VDM models. We give the video the input condition and ask the model to reconstruct. The input video is first converted into text description by the VLM and then recreated using the first frame and the description.
6.2. Longer Video Generation
EVA-Video Metrics. We use Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS) (Huang et al., 2024), Fréchet Video Distance (FVD), and especially Goal Completion EstimationnGCEE) for task completion evaluation. The quantitative experiments are separated into three groups, as shown in Table 2. Finish- Thinking also includes VQA accuracy comparison the output of Yes/No, which is not included in the table.
Main Results For the video generation model with text and image as input, our fine-tuning version EVA-Gen improves significantly in GCE and FVD, with scores of 86.83 and 177.28, respectively, highlighting the effective LoRA design. Among these frameworks, EVA excels in SC(97. 11), BC(96.01), and MS(99.31) and achieves the low- est (VDD(184.1)) score. This experiment demonstrates the quality of EVA and highlights the efficiency of RoG in task completion rate.
Baselines For the model with only the video generation module, which takes image and text as input, we compare the Dynamicrafter (Xing et al., 2023). its fine-tuned version, and EVA's video generation module(EVA-Geen For the
7

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Table Task successful rate on in-domain and OOD tasks on RT-1. We randomly select six tasks from the RTI validation set, spanning Pick Object, Move Object Near Object, Place Object Upright, Knock Object Over, Open/Close, Place Object into Receptucle.
Table 5. Successful rate in CALVIN, including human evaluation of video, and task successful rate in simulation environments. The quality demos are included on the demo page.
eval
model
lightbulb
led
rotate
Open drawer
Domain
Model
Pick
Move to Place Kneek Over Openiclose
Place into
w/o RoG
12/41
43/54
98/157
1844
AVDC
11/16
2/8
29/50
Video
EVA
35/41
45/54
124/157
36/41
Domain wio RoG
13/16
12/12
2/2
4/4
4/8
6/8
41/50
EVA
13/16
12/12
2/2
4/1
4/8
7/8
42/50
w/o RoG +CMLP
7/41
13/54
32/157
4/41
AVIC
2/4
1/2
1/3
7/15
Simulation
EVA+CMLP
9/41
15/54
45/157
9/41
OOD
w/RGG
(4)
0/3
w/o RoG+
10/41
32/54
48/157
11/41
EVA
3/4
4/4
1/1
2/2
1/1
1/3
12/15
EVA+2
32/41
43/54
90/157
25/41
6.3. Interleaved Generation
significantly outperforms the baseline (w/o RoG) in both video and simulation evaluations, demonstrating its superior robotic planning ability. On the video level, EVA achieves much higher task completion rates across all tasks, high- lighting the effectiveness of RoG. In the simulation, applied two versions of the video-to-action head, named CMLP and CMLP2. CMLP&CMLP2 uses video prediction and first action status as conditions, outputting 16 frames of actions. CMLP2 uses more tricks including simple ac- tion chunks, etc. The generated video The combination of EVA with CMLP2 achieves the best performance, sig- nificantly surpassing both the baseline and EVA with stan- dard CMLP. These results confirm that both RoG and the enhanced CMLP2 model contribute to improved task plan- ning and execution, and demonstrate the potential of this approach in robot task planning.
Among the two-stage methods, we demonstrate that EVA- L has a positive correlation with EVA-V. Understanding this ability can directly help improve generation quality and showcase the effectiveness of RoG. Our proposed model, EVA, outperformed other approaches under the RoG setting, achieving an EVAS-V score of 77.83 and an EVA-Score of 81.64. Qualitative results showcasing EVA's performance with different prompts are included in Appendix B.
As shown in Table 3, for the quantitative result of the Next- Step task, EVA had the best performance. As result, it provides better text (semantic) condition to guide EVA- Generator for improved generation outcomes. In contrast, LLAVA-One Vision (Li et al., 2024a) and Qwen2-VL-7B performed worse in this task compared to the How-To sce- nario due to their inability to accurately predict the next-step description. This clearly demonstrates the importance of VLM that is thoroughly trained in embodied scenes for the Embodied World Model. Such deficiency in EVAS- L also affects the generation quality, leading to the lower performance of LLAVA- Vision(51.75) and Qwen2-VL- 7B(57.23) on EVAS-V. This comparison also shows the overall consistency and quality of EVA-S.
6.5. Qualitative analysis
Figures 4 showcase EVA's capabilities in task execution, rethinking, and long-horizon guidance. In one example, EVA generates the motion "Move brown chip bag near the white can," and through the reflection of generation, extends frames until task completion. In an egocentric hu- man QA scenario, EVA responds to Next-Step question with "Wipe hand" and generates the corresponding video. Further results, including action-following in real and simu- lated robots, Video QA outputs, and dynamic adaptability in egocentric videos, are detailed in Appendix B. Besides, more video demonstrations are on the anonymous page at http:///ieesgooglle.comm/viii/ii
6.4. Evaluation on Robot Planning
We evaluated the success rates of hu- man evaluation tasks in RTI (Brohan et al., 2022) using the RoboDreamer (Zhou et al., 2024) framework, compar- ing AVDC, EVA without RoG, and EVA across in-domain and OOD prompts in Table 4. EVA outperformed AVDC with a 28% higher overall success rate, achieving a 100% success rate in the Move Object task. demonstrating strong frompt-following ability. However, the Open/Close task proved particularly challenging due to cases like "open right fridge door," which involve transparent glass doors. In OOD, the EVA also shows an obvious improvement in successful counts, 5 more success cases than AVDC. The RoG con- tributes significantly, 7 cases higher than the model without RoG, demonstrating the importance of it's frame extension.
Evaluation on RTI:
7. Conclusion
In conclusion, our proposed Reflection of Generation (RoG) stretagy and the Embodied Video Anticipator (EVA) model demonstrate significant advancements in video prediction for embodied scenarios. By integrating video generation with reasoning tasks like VQA, and leveraging chunk- auto-regressive strategy for longer sequences, EVA achieves high-fidelity predictions and robust generalization. The introduction of EVA- -Bench further enables comprehensive and fine- ame-grained evaluation, validating the effectiveness of RoG across diverse downstream tasks. These contributions
Evaluation on CALVIN We selected four tasks in CALVIN same as (Luo & Du, 2024). As shown in Table 5, EVA
8

---

Empowering World Models with Reflection for Embodied Vidco Prediction
pave the way for applying large-scale pre-trained models to real-world video prediction and embodied Al applications.
Chefer, H., Gur, S., and Wolf, L. Generic attention-model explainability for interpreting bi-modal and encoder- decoder transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 397- 406, 2021.
Impact Statement
Our EVA and EVA-Bench provide a unified evaluation benchmark to conduct a comprehensive assessment of the World Model's capabilities, and advance the development of World Models in Embodied Intelligence. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q., Wang, X., Weng, C., and Shan, Y. Videocrafterl: Open diffusion models for high-quality video generation, 2023.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang, X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for high video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7310-7320, 2024.
References
Agarwal, N., Ali, A., Bala, M., Balaji, Y., Barker, E., Cai, T., Chattopadhyay, P., Chen, Y., Cui, Y., Ding, Y., et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 2025.
Chen, X., Fang, H., Lin, T.-Y,, Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L.. Microsoft coco captions: Data collection and evaluation server. arXiv preprint vrXiv:/504.00322 2015.
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 382-398. Springer, 2016.
Cheng, S., Guo, Z.., Wu, J., Fang. K., Li, P., Liu, H., and Liu, Y. Egothink: Evaluating first-person perspective thinking capability of Dision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14291-14302, 2024.
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Dong, R., Han, C.,, Peng, Y., Qi, Z., Ge, Z., Yang, J., Zhao, L.,, Sun, J., Zhou, H., Wei, H., Kong, X., Zhang, X., Ma, K., and Yi, L. DreamLLM: Synergistic mul- timodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttttts////peeeeeeeeee http///pppeeeeeeeeeeeeeeeeeeeeee net/ forum? id=y01KGvd@9 KGvd99w..
Blattmann, A., Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.., Voleti, V., Letts, A., al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Du, Y., Yang, M., Florence, P., Xia, F., Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., Kaelbling, L., Zeng, A., and Tompson, Video language planning. arXiv [cs. CVJ, October 2023a.
Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Dabis, J., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A.,
Hsu, J., al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.068) 2022.
Du, Y., Yang, M., Florence, P., Xia, F. Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., al. Video language planning. arXiv preprint arXiv:2310. 2023b.
Bruce, J.. Demnis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., Aytar, Y., Bechtle, S., Behbahani, F., Chan, S., Heess, N., Gonzalez, L., Osindero, S., Ozair, S., Reed, S., Zhang, J.. Zolna, K., Clune, J., de Freitas, N., Singh, S., and Rocktäschel, T. Genie: Generative interactive environments. arXiv [G), February 2024.
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., and Isola, Dreamsim: Learning new dimensions human visual similarity using synthetic data. arXiv preprint arXiv:2306. 09344, 2023.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J.,
Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., and Li, H. Vista: A generalizable driving world model with high fidelity and versatile controllabil- ity. arXiv [cs. CVJ, May 2024.

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Grauman, K., Westbury, A., Byme, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp- 18995-19012, 2022.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
Jin, P., Takanobu, R., Zhang, W., Cao, X., and Yuan, Chat-umivi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700-13710, 2024.
Grauman, K., Westbury, A., Torresani, L., Kitani, K., Ma- lik, J., Afouras, T., Ashutosh, K., Baiyya, V., Bansal, S., Boote, B., et al. Ego-exo4d: Understanding skilled hu- man activity from first-and third-pend perrdoperoonn perspectives. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19480, 2024.
Kawaharazuka, K., Matsushima, T., Gambardella, A., Guo, J., Paxton, C.,, and Zeng, A. Real-world robot applications of foundation models: A review, 2024. URL https: //arxiv. org/ abs 2202.4422000 05741.
Gu, X., Wen, C., Ye, W., Song. J., and Gao, Y. Seer: Lan- guage instructed video prediction with latent diffusion models, 2024. URL https: //apsii//axxv..oo org/ abs/ 2303. 14897.
Kingma, D. P. Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6988. 2014.
Ko, P.-C., Mao, J., Du, Y., Sun, S.-.., and Tenenbaum, J. B. Learning to Act from Actionless Videos through Dense Correspondences. arXiv:2310.08576, 2023.
Guo, Y., Yang. C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472 04725, 2023.
Kondratyuk, D., Yu, L.., Gu, X., Lezama, J., Huang, J., Hornung, R., Adam, H., Akbari, H., Alon, Y., Birodkar, V., et al. Videopoet: A large language model for zero- shot video generation. arXiv preprint (tt::2//2.11225, 2023.
Ha, and Schmidhuber, J. World models. arXiv preprint arXiv: 1eX3.v1803.100 10122, 2018.
He, Y., Liu, Z., Chen, J., Tian, Z., Liu, H., Chi, X., Liu, R., Yuan, R., Xing, Y., Wang, W., al. Llms meet multimodal generation and editing: A survey. arXiv preprint arXiv:2405.19334 2024.
Li, B., Zhang, Y., Chen, L., Wang. J., Pu, F., Yang. J., Li, C., and Liu, Z. Mimic-it: Multi-modal in-context instruc- tion tuming, 2023. URL https: http///axxvvvvo// abs/ 2306. 05425.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024a.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0 09685, 2021.
Li, F., Zhang, R., Zhang, H., Zhang, Y., Li, B., Li, W., Ma, Z., and Li, C. Llava-next-interleawe Tackling multi- image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07899 2024b.
Hu, S., Tu, Y., Han, X., He, C., Cui, G., Long, X., Zheng, Z., Fang, Y., Huang, Y., Zhao, W., et Minicpm: Unveiling the potential of small language models with scalable train- ing strategies. arXiv preprint ar 0rXiv:2404.06335, 2024.
Lin, C.-Y. Rouge: A package for automatic evaluation summaries. In Text summarization branches out, pp. 74-81, 2004.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tuning. In NeurlPS, 2023.
Huang, S., Chen, L., Zhou, P., Chen, S., Jiang, Z., Hu, Y., Gao, P., Li, H., Yao, M., and Ren, G. Enerverse: Envi- sioning embodied future space for robotics manipulation. arXiv preprint arXiv:2501.0
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reasoning, ocr, and world knowledge, 2024a.
2025.
Huang, 7., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q. Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 21807-21818 2024.
Liu, Y., Chen, W., Bai, Li, G., Gao, W., and Lin, L. Align- ing cyber space with physical world: A comprehensive survey on embodied ai. arXiv preprint axXiv:2407068886 2024b.
10

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Luo, Y. and Du, Y. Grounding video models actions through goal conditioned exploration. arXiv preprint arXiv:2411.07223, 2024.
Song, Y., Byrne, E.,, Nagarajan, T., Wang, H., Martin, M., and Torresani, L. Ego4d goal-step: Toward hierarchi- cal understanding of procedural activities. Advances in Neural Information Processing Systems, 36, 2024.
Maaz, M., Rasheed, H., Khan, S., and Khan, S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01711 2018.
Mees, O., Hermann, L., Rosete- Rosete-Beas, E., and Burgard, W. Calvin: A benchmark for language-conditioned policy learning for long-hod oog-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7(3)77327-7334, 2022.
Vedantam, R., Lawrence Zitnick, C., and Parikh, D. Cider: Consensus- -based image description evaluation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 45644566666 4575, 2015.
Wang, F.-Y., Chen, W., Song, G., Ye, H.-J., Liu, Y., and Li, H. Gen-l-video: Multi-text to long video generation via temporal co-denoising. arXiv preprint arXiv:2305. 182.6 2023a.
OpenAL. Gpt-4 technical report, 2024. URL https:// arxiv. orgo abs/////////// /200. 08774.
Padalkar, A., Pooley, A., Jain, A., Bewley, A., Herzog, A., Irpan, A., Khazatsky, A., Rai, A., Singh, A., Brohan, A., et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint 2rXiv:23/0.08800 2023.
Wang, P., Bai, S., Tan, S., Wang. S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y. Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.1211 2024a.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-. Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Wang, X., Zhu, Z., Huang, G., Wang, B., Chen, X., and Lu, WorldDreamer: Towards general world models for video generation via predicting masked tokens. arXiv
Peng, Z.,, Wang, W., Dong, L.,, Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[cs. CVJ, January 2024b.
Wang, Y., Li, K., Li, Y., He, Y., Huang, B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z., et al. Internvideo: General video foundation models via generative and discrimina- tive learning. arXiv preprint arXiv:2212.03191 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
Wang. Y., He, J., Fan, L., Li. H., Chen, Y., and Zhang, Z. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. arXiv CV), November 2023b.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pultern recognition, pp. 10684-10695, 2022.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Sermanet, P., Ding, T., Zhao, J., Xia, F., Dwibedi, D., Gopalakrishnan, K., Chan, C., Dulac-Arood, G., Maddi- neni, S., Joshi, N. J., et al. Robovqa: Multimodal long- horizon reasoning for robotics. In 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 145-652. IEEE, 2024.
Xing. J., Xia, M., Zhang, Y., Chen, H., Wang, X., Wong, T.-T., and Shan, Dynamicrafter: Animating open- domain images with video diffusion priors. arXiv preprint 1:Xiv:2310.12190 2023.
Sharma, P., Ding, N., Goodman, S., and Soricut, R. Con- ceptual captions: cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACI., 2018
Xing, Z., Dai, Q., Weng, Z., Wu, Z., and Jiang, Y.- G. Aid: Adapting image2video diffusion models for instruction- -uidedd video prediction. arXiv preprint arXiv:2406 06465, 2024.

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprint arXiv: 10677 2024.
Yang, M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:2310.061 14, 2023.
Yin, S., Wu, C., Yang. H., Wang, J., Wang. X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et Nuwa-xl: Diffusion diffusion for extremely long video generation. arXiv preprint arXiv:2303. 12346, 2023.
Zhang, R., Cheng, A., Luo, Y., Dai, G., Yang, H., Liu, J., Xu, R., Du, L.,, Du, Y., Jiang, Y., et al. Decompos- ing the neurons: Activation sparsity via mixture of perts for continual test time adaptation. arXiv preprint arXiv:2405. 16486, 2024a.
Zhang, R., Luo, Y., Liu, J., Yang. H., Dong, Z., Gudovskiy, D., Okuno, T., Nakata, Y.. Keutzer, K., Du, Y., et al. Efficient deweahter mixture-of-experts with uncertainty- aware feature-wise linear modulation. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 38, pp. 168 188122168820 16820, 2024b.
Zhang, Y.. Li, B., Liu, h. Lee, Y. j., Gui, L., Fu, D., Feng, J., Liu, Z. and Li, C. Llava-next: A strong zero-shot video understanding model, April 2024c. URL
io/
Zheng, Z_, Peng. X., Yang, Shen, C., Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL https:
//github. ccmm/ppca tech wwweaatteehho/eorerr.t.e Sora.
Zhou, S., Du, Y., Chen, J., Li, Y., Yeung, D.-Y., and Gan, C. RoboDreamer: Learning compositional world models for robot imagination. arXiv ROJ, April 2024.
12

---

Empowering World Models with Reflection for Embodied Video Prediction
A. Appendix: Model Architecture and Training
EVA enables the pre-trained diffusion generator and visual language model to provide an autoregressive world prediction model.
EVA Model Architecture
Multimodal Input
Prediction
Querry Emibedding
Spatial dual-attention Transformer Block
Large Language Model
Char
Cross Coos-Ritentoon Rote-tttttonn
EloRe
LoRA
Visual Projector
Text Tollenteer
Text Detokerizer
Generation Adapter
"Pick
UNet
seed formant of Fooeeee
Supes 2013
LoKK I
Visual Uncoder
How to pick
tpS
Self Hi-ttertin
Aotention
Tanh goting
Image Conditonal Rerempter
Eloik
VAE
LoRA N
Token
Video Caption Alignmant
QA Instruction Tunning
Stage 3: QA Generation Tunning
Visual
Large Language
FLaBAA
Visual Proyector
Large Larguage Model
Visual Projector
Large Language Model
Generation Adaptier
FloRA
Projector
Model
Figure 5. A unified visual understanding and generation framework of EVA. The EVA introduces a visual projector in understanding LLM, image conditional resampler the generation model, trained generation adapter text condition for denoising UNet, and added an Ensemble LoRA system for domain-specific generation. We train the EVA separately, including three stages of alignment and training.
A.1. Vision Language Model
The Modern Video Language Model (VLM) is based on Large Language Model (LLM), It leverages the powerful language capabilities of pre-trained language model to transform the input image video In into latent visual features & and then generates language description output The representation equation of VLM is:
VLM( + 1))
(3)
where Ø EIc)der(II), and sb represents the text embedding. Here, is typically in the language embedding sequences, and the visual encoder projects the visual content 7, into the text embedding domain. By this method, VLM trains the visual information in an autoregressive format, similar to another language model.
Given our limited computational resources. it was essential to represent more video frames using fewer video tokens. ChatUniVi's adaptive parameter-free token clustering method significantly reduces the number of video tokens without introducing additional computational overhead, allowing training to remain within the 2048-token limit. We independently trained the VLM on the Embodied- Videad-Vii-----iee cedteo-Descrittion dataset, as shown in Fig. 5.
We tested several VLM backbones, including ChatUniVi (Jin al., 2024), LLaVA- Vision (Li al., 2024a), LLaVA- NeXT (Liu et al., 2024a), MiniCPM (Hu et al., 2024), and other mdoels. We found that while existing models were capable of generating detailed descriptions, they struggled with tasks involving prediction and planning, primarily due to the lack of relevant data in their training corpora. Additionally, the simplicity of the text prompts used in the diffusion model's 's training data necessitated concise responses from the VLM. These responses needed to be composed short, straightforward sentences that clearly include a subject, verb, object, location, and destination. As a result, we needed to fine-tune the VLMs fully on our dataset.
13

---

Empowering World Models with Reflection for Embodied Vidco Prediction
A.2. Latent Diffusion Model
The Diffusion Model (Ho et al., 2020) type of generative model that iteratively refines a noisy input to generate high-quality data samples. It leverages series of denoising steps to transform initial noise distribution 2o (0, I) into desired data distribution x. The process involves gradually removing noise from the input, guided by learned model eg to produce clear and coherent output. The latent diffusion model(LDM) further uses VAE to scale down the input features and reduce computation costs. Given the initial condition, the representation equation of the LDM (Rombach et al., 2022) is:
XT = VAE(Unet(Zz) + +)
(4)
where 2g is the initial noise, Xr is the final output, Unet is the denoising network, and e represents the text or other control condition embeddings. The Variational Autoencoder (VAE) further decodes the latent feature into video.
In our comparisons with Animatediff (Guo al., 2023), VideoCrafter2 (Chen et al., 2024), and Open-Son (Zheng et al., 2024), we found that Dynamicrafter (Xing et al., 2023), which employs additional image condition injection methods, excels in retaining low-level features and maintaining high consistency for training ffreingg-freeiee longer video extensions. The core components of this VDM include a VAE encoder and decoder, an image condition resampler, and denoising UNet.
Interaction Token Inspired by (Peng et al., 2023; Dong et al., 2024), we use special tokens to fit the task better. For image input, the VLM uses an <image> token as a placeholder within text tokens. Before being processed by LLM, the <image> token replaced by visual feature tokens, obtained through a visual encoder and visual projector layer. For video inputs, the number of <i <image> tokens used corresponds directly the number of frames in the video. During generation, concatenate prefix token <IMG_P> with VLM language input query embeddings extract the feature. As shown in Figure 5, after obtaining the query embeddings, use it as a condition for VDM to substitute for text prompt.
A.3. Ensemble of LoRA
We propose Ensemble-LoRAA method designed adapt to various domains by utilizing distinct LoRA modules. This approach ensures the highest generation quality in multiple robotic and egocentric environments while maintaining the generalization capabilities of the pre-trained Variational Diffusion Model. Let W represent the original weight matrices in the transformer layers. For each domain d, we train a low-rank adaptation:
Wa= W + AW, = W + A.B
(5)
where Aa and Ba are low-rank matrices specific to domain d. We applied LoRA after each transformer block in video diffusion to quickly adapt the video generation model to different tasks, as shown in Figure 5. Moreover, inspired by the Mixture of Experts (Jacobs et al., 1991), we proposed an Ensamble-LoRA for each domain by a Task Token controlled gating system (human-egocentric, real-robot, simulation-robot, etc.):
gd = softmax(f/(aaaaaa Token))
(6)
where f is a function that maps the task token to gating values, and ga is the gating value for domain d. Therefore, the ensemble output for a given task is computed as:
W W + Σ go A
This formulation allows for efficient adaptation across different tasks without discarding previously learned adaptations.
A.4. Cross-Attentiin Generation Adapter
Our generation adapter employs a cross-aties cross-attennion module to align the VLM's ss hidden features with the VDM's 's text embedding features. Specifically, the adapter first applies a linear transformation to the VLM's output to match the dimensionality of
14

---

Empowering World Models with Reflection for Embodied Vidco Prediction
the VDM's feature space. We the diffusion denoise loss to train this adapter achieve the best generation quality. The detailed module information is introduced in Appendix A.
A.5. EVA Model Architecture
We employ Vicuna-based VLM and 3D U-Net architecture VDM parameterize the EVA model. The model follows the ChatUniVi structure for VLM, and standard 3D U-Net structure, with spatial downsampling pass followed by an upsampling pass, utilizing skip connections from the downsampling activations. This process is interleaved with 3D convolution and attention layers. The model and training hyperparameters of EVA are summarized in Table 6 & 7.
Table Model Architecture for EVA.
Name
Type
Parameters
VDM
UNet
1.4B
VAE Encoder
AutoencoderKL.
83.7M
Image adapter
Resampler
48.8M
Text adapter
Resampler
32.3M
VLM
ChatUniVi
7.0B
Query embedding
Linear
262K
Table 7. Hyperparameters for training EVA diffusion model.
Hyperparameter
Value
Base channels
320
Optimizer
Adam (31
0.9, B3 = 0.999)
Channel multipliers
1, 2, 4,4
Learning rate
0.0001
Blocks per resolution
2
Batch size
4
Attention resolutions
4, 2, 1
Num attention heads
64
Conditioning embedding dimension
4096
Conditioning embedding MLP layers
4
Conditioning token length
64
Dropout
0.1
Training hardware
8 Nvidia A800 chips
Training steps
20000
Diffusion noise schedule
cosine
Noise schedule log SNR range
[-20, 20]
Sampling timesteps
50
Sampling log- lor-variance interpolation
= 0.1
Weight decay
0.0
Prediction target
A.6. VLM Training Details
Stage 1: Video-Caption Alignment. the first stage, train VLM and VDM separately to fit them into the embodied prediction domain. During VLM training, aim align the video encoder with language model using image-caption pairs from various datasets, including COCO (Chen et al., 2015) and curated subset of CC3M (Sharma et al., 2018 () (CC3M-595K) screened by LLaVV (Liu et al., 2023). The Visual Language Model (VI.M) is pre-trained for one epoch with a batch size of 128, using the AdamW optimizer (Kingma, 2014) and a cosine learning rate schedule. The learning rate is set to 2e-3, with a warmup rate 0.03.
Stage QA Instruction Tuning In the second stage, we further align the VLM with the instruction tuning dataset,
15

---

Empowering World Models with Reflection for Embodied Vidco Prediction
transforming it into an embodied QA bot. We incorporate open-domain multimodal instruction data from multiple sources: multimodal in-context instruction datasets (MIMIC-IT (Li et al., 2023)), LLaVA visual instruction datasets (Liu et al., 2023), and video instruction data from VideoChatGPT (Maaz et al., 2024). We then add our EVA instruction tuning dataset as described in Section 3. All input images and frames are resized to 336 x 336. This stage is trained for two epochs with a batch size of 128 and a learning rate of 2e-5.
Stage 3: Adapting the Pipeline and Training Ensamble-LoRA In the final stage, we adapt the entire pipeline and train the adapter and VDM Ensamble-Loke specifically for each task, where with the inspiration of the mixture of experts (Jacobs al., 1991) (Zhang et al., 2024a) (Zhang et al., 2024b). This stage utilizes the EVA instruction tuning dataset for LoRA training. Despite the diverse visual distribution among EVA-Instruction datasets, the language annotations similar. Therefore, we propose a special tuning method that tunes the Ensamble-LoRA and adapter to maximize generation quality, In this setup, the generation adapter is trained on the whole dataset, while the Ensamble-LoRe is updated on each domain separately. We compare tuning VDM LORA, CDM full aallmparreeter tuning, and a two-stage inference method in Section 6.
A.7. Training Dataset
To enhance comprehensive understanding, reasoning, planning, and video prediction capabilities in embodied environments, we meticulously curate a comprehensive training dataset including 500K instances, termed EVA-Instruct. This dataset encompasses four tasks, each containing videos of humans, real-world robots, and simulation robots. To enhance the diversity of prompts, we employed ChatGPT-4v (OpenAI, 2024)t to generate question-answer pairs, which were then applied to different tasks. The complete EVA instruction tuning dataset comprises 500K QA pairs collected from Open-X- Embodiment (Padalkar al., 2023), Ego4d (Grauman et al., 2022), Ego-Exo4d (Grauman al., 2024), and CALVIN (Mees et al., 2022). The data sources are shown as Table 8. The text in these datasets can be effectively restructured into components such as subject, verb, object, location, and destination, as illustrated in 11, making it highly suitable for our task requirements. The instructions for each meta-task are as follows.
Dataset
# Examples
Weight
Simulation
CALVIN (Mees et al., 2022)
23k
0.85
RoboVQA (Sermanet al., 2024)
800k
0.1
Real Robot
RT-1 data (Brohan et al., 2022)
70k
0.5
Ego4D (Grauman et al., 2022)
3.5M
0.01
Human activities
Ego-Exo4D Keystep data (Grauman et al., 2024)
21k
0.9
Table 8. Dataset name, number of training examples, and mixture weights used for EVA-Instruct.
Instructions for action description. The list of instructions used to briefly describe the video content is shown in Table 12. They present the same meaning with natural language variance. Given the complexity of scene understanding in embodied environments, we aim to simplify the problem by selectively incorporating guidelines into the prompt with a probability of 50%. These guidelines are generated using GPT-4V shown in Table 13.
Instructions for How-to, Finish-Thinks and Next-Step. The list of instructions used construct the "How-to" format generation is shown in 14. The instructions for the Finish-Thine meta-task are shown in Table 15. Considering dataset differences, we constructed "next-step" prompts using the instructions from Table 16.
Data Construction for EVA Tasks. As shown in Table 8, the data sources for our four tasks are constructed using distinct instructions to create datasets for each meta-task. The datasets for the How-To and cetion-Desrrittion tasks are relatively straightforward. Given that the textual annotations in embodied scene datasets generally follow the structure shown in Table 11, we only needed extend the prompts using GPT-4o and standardize them into the format of subject, verb, object, location, and destination. Due to the ease of constructing the How-To and Descrrittniiittonnn tasks, we built two large datasets: Now-To-200K and Action- Description 200K.. For the Finish-Thie Think dataset, our statistical analysis indicates that taking the first 25% of a video provides good examples of unfinished tasks. Additionally, some datasets within RoboVQA already contain questions regarding task completion, allowing for direct conversion. Based on this approach, we constructed the Finish-Think-S5 Think-50K dataset. Finally, for the next-step dataset, we utilized the key step annotations from the Ego-Exo4D dataset. This dataset marks key steps for each segment of a complete video, making it easier to convert into a next-step prediction task, In the Open-X Xeen-X-Fii X-Embodiment's real robot datasets, such as RoboVQA which contain long-horizon
16

---

Empowering World Models with Reflection for Embodied Vidco Prediction
task annotations involving sequence of multiple steps, converted these sequential steps into next-step tasks by focusing on the ordered steps provided. Using this approach, constructed the Next-Step-50K dataset.
B. Appendix: Extensive experiments
In this section, we provided the results of the extent visualization experiment. Figure provides visualization results on simulation robot data, demonstrating that EVA could drive the robot by text instruction. The result shows that with proper action, the following is quite accurate.
Figure and Figure 8 also show the action following generation ability of EVA in real robots. Figure 7 shows EVA could answer the question by generation the video.
Figure 9 is an ego-centric generation example. With proper training, the model has some planning abilities, generating a continuous motion sequence that first turns right and then gets the sugar.
Lift pink block table
Rotate pink block left
sweep the pink block to the left
put down the pink block
Figure 6. We show the prompt following the ability of the EVA on the simulation robot. Given the same input video, the model can generate different actions according different instructions.
17

---

Empowering World Models with Reflection for Embodied Vidco Prediction
How would you do [pick green can from middle shelf of fridge]?
Could you explain how (pick green can]?
How to [pick green rice chip bag]?
Figure 7. EVA'S action control abilities on the real robot videos.
B.1. Comparison in RT1
We compare the success rates of human evaluation tasks in RTI, following the framework of RoboDreamer(6], Evaluation spans two groups seen seen prompts and unseen prompts and includes a comparison AVDC, EVA without fmishhthinkinn and EVA.
Model
Object
Object Near Object
Place Object Upright
Object
Open/close
Object Receptacle
Summary
AVDC EVA (wl linish thinking)
11/16
912
1/2
4/4
2/8
2/8
29/50 (58%)
13/16
12/12
2/2
4/4
4/8
6/8
41/50 (82%)
EVA
13/16
12/12
2/2
4/4
478
7/8
42/50 (84%)
Table human evaluation results RT-1 (Brohan et al., 2022). perform human evaluation results to judge task completion rate on seen prompts and seen cases of EVA and AVDC.
In seen tasks, randomly selected 50 tasks from the validation of RTI, including tasks, across multiple scenes(Pick Object, Move Object Near Object, Place Object Upright, Knock Object Over, Open/close, Place Object into Receptacle). For the seen tasks in Tab. 9, AVDC has a 58% success rate, while EVA is 28% higher in total success rate. Moreover, EVA performed better in the Move Object task with a 100% success rate, showing good prompt-following ability. The Open/close task especially hard since a few cases like "open right fridge door" include the transparent glass door.
Model/Tacks
Pick Object
Move Object Near Object
Place Object Upright
Knock Object Over
Open/close
Place Object into Receptacle
Summmary
AVDC
2/4
1/4
1/1
1/2
1/1
1/3
7/15
EVA (wlo finish thinking)
2/4
1/4
0/1
0/3
5/15
EVA
3/4
4/4
I/1
1/3
12/15
Table The human evaluation results RT-1 (Brohan et al., 2022). We perform human evaluation results to judge the task completion rate on EVA and AVDC unseen prompts.
For unseen tasks, we start from the existing cases and manually change the subject or object of the prompt. For example, "Place coke can into the bottom drawer" to "Please close bottom drawer".. AVDC performance is better than EVA (wlo finish thinking) in Place, knock, and Open/Close tasks, since the trajectory is longer than EVA (w/o finish thinking) can generate simultaneously. However, EVA could fix this issue and significantly improve the success rate by extending the video.
C. Appendix: EVA- -VA-Bennhmark
To facilitate evaluation, the proposed EVA-Bench curated a collection of 125 high-quality samples from our EVA-Instrut, covering real-world robots, simulated robots, and egocentric human daily activities. These samples encompass diverse scenarios such as pick-and-pll place tasks, cooking, bike repair, COVID testing, and indoor organization. Drawing from existing
18

---

Empowering World Models with Reflection for Embodied Vidco Prediction
knock orange can over
knock redbull can over move brown chip bag near blue chip bag
move green jalapeno chip bag near apple
move sponge near rxbar blueberry
Figure 8. EVA's action control abilities the real robot videos.
datasets, we categorize them into three groups: egocentric human videos, real-world robots, and simulations. The statistical distribution of different scenes in our EVA-Bench is shown in Fig. 11.
C.1. Benchmark Examples
We selected frames from the benchmark in three areas: egocentric human videos, real-world robots, and simulated robots. These frames showcase the richness and diversity of our embodied scenes, as shown in Fig. 10. The first three rows in the figure represent scenes from cooking, COVID testing, and bike repair. The fourth and fifth rows display indoor robotic arms manipulating various objects, while the sixth row shows scenes involving simulated robots.
C.2. EVA Score Language Metrics
BLEU (Bilingual Evaluation Understudy Score): BLEU1 and BLEU2 represent the BLEU scores using 1-gram and 2-gram matches, respectively. BLEU measures the overlap of n-grams between the generated text and reference text. Higher scores indicate greater similarity the reference.
METEOR (Metric for Evaluation of Translation with Explicit ORdering): METEOR considers factors like stemming, synonyms, and word order, making it more flexible than BLEU. It evaluates translation quality based on precision, recall, and penalty for longer sentences. Higher scores indicate better translation quality.
ROUGE-L -Rreatl-Oriented Understudy for Gisting Evaluation Longest Common Subsequence): ROUGE-L measures the quality of text summaries and translations by calculating the longest common subsequence (LCS) between the generated and reference texts. It focuses on recall, with higher scores indicating better coverage of the reference content.
CIDEr (Consensus-based Image Description Evaluation): CIDEr is used primarily for image description tasks. It evaluates the quality of descriptions by calculating the TF-IDF weighted n-gram similarity between the generated and reference descriptions. Higher scores indicate greater consistency with the reference descriptions. While calculating EVAS-Language points, we normalized the CIDEr by:
X norm X/10
19

---

Empowering World Models with Reflection for Embodied Vidco Prediction
How would you do [Get Sugar]?
Get sugar.
Turn right
Get Sugar
Figure 9. EVA can do large Motion. In this example, the generation video first turns left and then gets the sugar with a different hand.
where X is the CIDEr score.
SPICE (Semantic Propositional Image Caption Evaluation): SPICE evaluates image descriptions by parsing the generated and reference descriptions into semantic graphs. It focuses on the semantic content and relationships within the descriptions. Higher scores indicate better semantic alignment with the reference descriptions.
CLIP(Contrastive Sanguage-Image Pre-training) score: CLIPScore is a reference-fiee free evaluation metric for image captioning. Unlike traditional metries that compare generated captions to reference captions, CLIPScore uses a pre-tratne trairri CLIP model to directly measure the similarity between the generated caption and the image itself. This approach leverages the model's ability to understand both images and text, providing a robust evaluation of how well the caption describes the image. Higher CLIPScores indicate better alignment between the image and the generated caption. In EVA Bench, we normalize the CLIP score between 1 by:
X - min(X')
X norm
max X') min(X')
Where x' is the reciprocal of the X, we fix the upper and lower bound by summary the general testing result of different methods.
GPT-4o as a Judge: Unlike traditional similarity-based methods, GPT-4o emphasizes semantic understanding. In our implementation, we format the question, model output, and reference into a prompt, as outlined in Appendix C.5, and input it into the GPT-4o evaluator. The comparison between the generated and reference answers is based on four key criteria: object, action type, location, and attribute. During the model evaluation. we observed that some generated responses, despite being semantically close to the ground truth, received low scores. Conversely, responses omitting key information occasionally received high scores. For example, the ground truth might state, "Cut out the tomato stem with knife on the cutting board" while Qwen2-VL-7h5 (Wang et al., 2024a) predicts, "Chop the tomato with knife on the cutting board". Despite high BLEU scores, the key difference between removing the stem and chopping the tomato remains significant. Therefore, using GPT-4o as a judge to score QA text pairs and model-generated responses is essential. By leveraging GPT-4o's 's advanced analytical and reasoning capabilities, we can more accurately evaluate the similarity between generated and reference texts. The specific prompt is detailed in Appendix C.5.
C.3. EVA Score Video Metrics
Overall Consistency, Motion Smoothness, Background Consistency, and Subject Consistency metrics are inspired by the contributions from the open-source project VBench (Huang et al., 2024).
We also evaluate the overall consistency between video and text using ViCLIP (Wang et al., 2022). which measures how well the generated video aligns with general text prompts in terms of both semantics and style.
Overall Consistency:
While Subject Consistency and Background Consistency focus on the temporal consistency of the appearance, Motion Smoothness evaluates whether the motion in the generated video is smooth and adheres to real-world physical laws. This is assessed using motion priors from a video frame interpolation model.
Motion Smoothness:
Subject Consistency: This metric assesses the alignment between the subject in the input image and the subject in the
20

---

Empowering World Models with Reflection for Embodied Vidco Prediction
generated video, also using DINO (Caron et al., 2021) features and order-statistics schemes.
Background Consisteney: This metric evaluates the coherence between the background scene in the input image and the generated video. It utilizes DINO (Caron et al., 2021) features and carefully designed order-statistics schemes.
Goal Completion Estimation: In our benchmark, we manually labeled a frame as the goal condition for loop tasks like "wash hands" lack clear ending frames. Robot tasks often have more defined endpoints, we use the last frame. We use DreamSim (Fu al., 2023), and use an enhanced version based on DINOv2 (Caron et al., 2021), to compare the generated frames with the ground truth frames. A higher GCE indicates that the generated image is closer the target.
C.4. Last Frame Comparison Feature Selection
To measure the similarity between the generated frames and the ground truth, we compare DreamSim(Fu et al., 2023), CLIP(Radford et al.. 2021), and DINOv2(Caron et al., 2021). We visualize CLIP and DINOv2 features (Fig. 12) to analyze perceptual differences in embodied scenes. Specifically:
CLIP Semantic Heatmap (Chefer et al., 2021), which highlights regions relevant to text prompt, often misfocuses on irrelevant objects and fails to consistently capture task-relevant interactions. This likely occurs due to the absence of embodied data in CLIP's training set. DINOv2, in contrast, excels at capturing fine-grained features, as evidenced by its attention to interactive objects such as robotic grippers. However, the visualization results of DINOv2 features heavily rely on manually adjusted PCA parameters. It often assigns high attention to background objects, and without proper PCA adjustments, the DINOv2 features of the final frames tend exhibit extremely low variance, making it difficult to discern meaningful differences. In diverse embodied scenarios, ranging from egocentric tasks to robotic interactions, the PCA values must be frequently adjusted manually for different cases, which undermines the generalization ability of this metric in goal completion estimation tasks.
Ultimately, we select a version of DreamSim which is finetuned on DINOv2-based metric, for its improved alignment with human perception. It better aligns with human perceptual priorities by emphasizing foreground objects and semantic content while remaining sensitive to color and layout. This makes DreamSim particularly generalizable and efficient for evaluating goal state estimation in embodied tasks.
Moreover, analyze the minimum and maximum DreamSim scores in as our goal completion estimation (GCE) value, where higher GCE score indicates closer alignment to the target.
and use them to normalize DreamSim
C.5. Model Inference Prompts
Since the annotated answers in dataset typically focus on key actions, composed of elements such as subject, verb, object, location, and destination, existing general VLMS struggle to generate responses in the same style. They often produce redundant or irrelevant scene descriptions. To address this, we designed specific prompts to guide the visual language models (VLMs) in generating concise, non-redundant answers. The prompt designs for various VLMs are listed in Table 17. For each meta-task in our EVA-Bench, all models, except for our EVA model, use the same prompts listed in Table 17.
C.6. Evaluation Prompts
To evaluate general Vision-Language Models (VLMs) in a zero-shot setting, results using standard text evaluation metrics are often poor. Through experimental analysis, we found that traditional metrics like BLEU, METEOR, and ROUGE-L do not effectively capture similarities in actions, objects, locations. and other essential factors. Therefore, we follow the approach proposed by Think (Cheng et al., 2024) and employ GPT-40 assess the predictions of these VLMs.
However, directly using Thinksnk's prompts often leads to extreme scores, with many reasonable predictions being rated as O. We believe this discrepancy stems from the increased complexity of tasks in embodied scenes, which demands more nuanced and sophisticated evaluation framework.
For the step description generation task, we guide GPT-4o to assess predictions based on four key criteria: object, action type, location, and attribute. Each criterion is scored as follows: fully correct, 0.5 if partially correct or somewhat aligned, and 0 incorrect. The final score the average of these four criteria.
Object evaluates whether the objects involved in the action are correctly identified. Action type leverages GPT-40's reasoning ability to assess whether the predicted action aligns with the ground truth. For instance, comparing "get the fork
21

---

Empowering World Models with Reflection for Embodied Vidco Prediction
from the table" and "pick up the fork from the table," the use of "get" and "pick up" would result in of 0.5. Location assesses whether the location where the action is performed is accurately described, along with the broader context. If the action involves movement (e.g., moving an object from one place to another), the evaluation considers whether the starting point, destination, or path is correctly specified. Attribute examines whether the attributes of the objects involved (e.g., size, color, state, condition) are described accurately.
Our designed prompts are presented in Table 18.
"C adds the shredded ginger into the small stainless cowl with his right hand."
CC steps to the right while holding her head and swaying her hips."
CC places his right hand around the waist of woman X."
"Tighten the left axle nut with your left hand."
"Place the black chip bag in the tray."
Grasp the red block from the drawer."
"Keep the brown potted plants together."
Table 11. The example answers of EVA-Instruct-Answee Example answers from BVA-Instmut-Ansserr In this context, C refers to the wearer of the egocentric camera, while represents other person involved in interaction.
22

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Ego Video
Real robot
Simulated robot
Figure 10. Random frames from EVA-Bench.
23

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Ego Humon
6.40%
Sim-Rocs Robot
Real- Real-Rober Robot 9.60%
17.60%
Next-S Mext-Step Step
Real-
Real-Robot Robot 7.20%
Finish-Think Think
Finish-
Real Robot,
13. 60.60%
Robot 8. 8.00%
Action-
Desc.
Ego. Ego-Humma Human,
Ego HHmmnn 8.80%
How-
4.80%
Ego Human 8.80%
Real Roboot
7 7.20%
Sin Robot 8.00%
Figure This chart illustrates the distribution of various embodied scenes categories within the EVA-Bench
Ground Truth
Ground Trubh (Last Franc)
Action Success (Lasi Fremc)
(Fine Franc)
(Last Frame)
Origin image
CLIIseeaann Hectmp
CLIP CLIPARRER Antmoo
DRNDs2 WHO
Buckpood
lnckgronnd
DINOV 207
Prompe pink degrees right am
Figure 12. Visualization of CLIP and DINOv2 Embeddings. The second row shows the attention heatmaps (relevancy) between the input text prompt the corresponding image. The third row represents the final attention map from CLIP visual encoder. The fourth and fifth rows illustrate the PCA visualizations of DINOv2 embeddings. showing the top three principal components. with and without background masking, respectively. PCA visualizations, brighter regions represent higher values after the PCA transformation.
24

---

Empowering World Models with Reflection for Embodied Vidco Prediction
"What is happening this egocentric video?"
"Can you describe the interaction in this video?"
hat actions are being performed in this video?"
Please provide a description of the activity in this video."
"What is the person/robot doing in this video?"
"What object is being interacted with in this video?"
"Can you summarize the actions in this video?"
"What task is being carried out in this video?"
"What is the main activity shown in this video?"
"Can you provide brief description of the video content?"
"What interaction is taking place in this video?"
"What is the main focus of this video?"
"What is the subject doing in the video?"
"Can you explain the actions in this video?"
"What specific steps are being taken in this video?"
"What is the sequence of actions in this video?"
"What key activities are being shown in this video?"
"What is the primary task in this video?"
"What is the individual engaged in within this video?"
"What detailed actions are depicted in this video?"
"Can you outline the main steps in this video?"
"What is the purpose of the actions in this video?"
"What are the key interactions in this video?"
"What process demonstrated in this video?"
"What is the sequence of events in this video?"
Whhtt detailed activities are performed in this video?"
"What main actions can be observed in this video?"
"What specific task is being executed in this video?"
"What are the primary actions taking place in this video?"
"Can you detail the key steps shown in this video?"
Table 12. The list of instructions for action descriptions.
25

---

Empowering World Models with Reflection for Embodied Vidco Prediction
"Please identify the primary object in the first-person view and describe the main actions involving
it."
"Determine the main object and outline the key actions associated with it."
"Fistt identify the primary object, then summarize the main actions performed with it."
"Locate the primary object and describe the key actions taken."
"Focus on the interaction between objects and the human."
"Follww these two guidelines: (1) identify the main objects, and (2) describe the key steps."
"Spot the central item and describe the key steps performed."
"Point out the central object and explain the key steps involved."
Foocus on the primary item and narrate the sequence of actions associated with it.""
"Do not confuse the objects hallucinate about them."
"Answer based on what you observe, without over-interpreting, distorting facts, or fabricating information." "
"Think like a human: first identify the interacting objects, then infer the actions being performed." "
"First, infer the overall action, then identify the category of objects, and finally, use the object category determine if the action is correct. Please output the final description of the step."
Table 13. The list of instruction guidelines for action description.
"What is the way to [action]?"
"Can you show me how to [action]?"
"How can I do [action]?"
Whhat is the step to [action]?"
"Could you explain how to [action]?"
"What method should use [action]?"
"How should perform [action]?"
"What is the best way [action]?"
"How would you do [action]?"
"How to [action]?"
Table 14, List of instructions for How-to meta-task.
26

---

Empowering World Models with Reflection for Embodied Vidco Prediction
"The video is describing the step to [action], is this step completed?"
Basedd on the given video, has the task to [action] completed?"
"Has the action to [action] completed?"
"Does the video confirm the completion of the step to [action]?"
"Has the process of [action] been completed based the given video?"
"Has the video shown the completion [action]?"
"Based on the given video, has the action to [action] completed?"
Has the activity of [action] been successfully completed according to the video?"
Table 15. List of instructions for Finish-Thiie meta-task.
"This video depicts how to [action]. The current goal is [goal]. What is likely to happen next?"
"This video depicts how [action]. What is likely to happen next?"
"What is likely to happen next?"
Table 16. List instructions Next-Step meta-task.
Meta-Task
General Prompts
Action-Description
Question: (question)) Please provide a brief description in one sentence. The response should be clear and to the point, containing key action words such as objects, verbs, objects, location, and destination. Avoid unnecessary details or explanations. Please briefly describe the key action in a few words.
Finish-Thie Think
This is Finish-Think" task where you need to predict if step is completed or not. Question: (question). Please answer with either yes" or no".
How-To
This is How-to" task where you need explain how accomplish a specific task. Question: (question). The response should be clear and to the point, containing key action words such objects, verbs, objects, location, and destination. Avoid unnecessary details or explanations. Please briefly provide simple step description in few words.
Next-Step
This Next-Step" task where you need to predict the next step in sequence of steps. Question: (question]. The response should be clear and to the point, containing key action words such objects, verbs, objects, location, and destination. Avoid unnecessary details explanations. Please briefly describe the next step in a few words.
Table 17. Model inference prompts used for four meta-tasks.
27

---

Empowering World Models with Reflection for Embodied Vidco Prediction
Model
Prompts for Evaluation
[Instruction] You are tasked with evaluating the quality of the response provided by an Al assistant. The evaluation should focus on correctness, helpfulness, and relevance. Depending on the task type, you will evaluate specific attributes of step-level generation tasks or score a simple yes/no question.
1. For step-level generation tasks, evaluate the assistant's response based on the following attributes:
Object: Does the assistant mention the same or a closely aligned object as the reference? Minor but relevant differences (c.g., an additional unnecessary object) can receive partial credit, but introducing unrelated missing key objects should lower the score.
Action Type:
Is the action in the assistant's answer precise and in line with the reference? If the intent of the action is similar but less precise, give partial credit. However, if the action significantly changes the task's s context or result, it should be more strictly penalized.
Does the response correctly identify the location or context of the action? If the action involves movement (e.g., moving an object from one place to another), evaluate the destination, starting point, or path are accurately described. Minor location discrepancies can receive partial credit, but if the location changes the context goal of the action, assign lower score.
Location:
Attribute: Are the attributes of the object(s) (such as size, color, state, or condition) correctly described? Missing or incorrect key attributes should lead to a lower score. If attributes are implied but still align with the context, partial credit can be given.
GPT-40
Scoring: If the reference answer does not include information for particular attribute (e.g.. object, action type, location, or attribute), do not score that attribute. For each attribute, assign:
1 if fully correct,
- -0.5 if somewhat correct or partially aligned,
- 0 if incorrect.
After evaluating each attribute, sum the scores and calculate the overall rating by averaging the individual scores. Do not round the final result. The final rating will be non-rounded average score between 0 and 1.
2. For yes/no questions, directly evaluate whether the assistant's response is correct: Assign 1 if the answer is correct, and 0 if it is incorrect.
After providing your analysis, rate the response with the calculated average score, formatted as: "Rating: [[average_score]]" Now proceed with the evaluation based on the provided task:
[Task Type] (task type).
[Question] {question].
[The Start of Reference Answer] (refanswer),
[The End of Reference Answer),
[The Start of Assistant's Answer] (answer).
[The End of Assistant's Answer].
Table 18. Model inference prompts used four meta-tasks.
28


Reviews:
[{"review_id": "MkuuCLIW5j", "reviewer": "Reviewer_3Yfm", "summary": "This paper introduces Reflection of Generation (RoG), an intermediate reasoning strategy that enhances video prediction by combining pre-trained vision-language and video generation models as a world model. We introduce Embodied Video Anticipation Benchmark (EVA-Bench) to evaluate these models across diverse tasks and OOD settings. Based on RoG, we develop Embodied Video Anticipator (EVA), a multistage-trained model that generates high-fidelity frames and adapts to long video sequences. Experiments show EVA’s effectiveness in video generation and robotics, advancing large-scale pre-trained models for real-world applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 3Yfm\n\nWe thank the reviewer for their thoughtful and encouraging feedback. We are glad that the reviewer appreciated the novelty of our Reflection-of-Generation (RoG) mechanism, the well-structured EVA-Bench, and the demonstrated real-world applicability of our EVA model. Below, we address the two main concerns raised point by point:\n\n---\n## 1. Insufficient Ablation on the Reflection-of-Generation (RoG) Mechanism\nWe appreciate the reviewer’s point regarding the need for more fine-grained ablation studies to isolate the contribution of RoG. To maintain clarity in the main paper, we had streamlined ablation findings. In the revision, we will include a dedicated subsection. We will revise the experimental section to include a dedicated subsection for RoG ablations. Below, we provide a summary of the key findings:\n1. **In the Finish-Thinking Video Generation task(Tab. 2)**, we conducted ablations by applying RoG using different vision-language models (VLMs) on the same fine-tuned generation model (EVA-Gen). The GCE scores consistently improve with RoG, even when using non-finetuned VLMs, demonstrating that RoG—by introducing intermediate reasoning—can effectively enhance task completion.\n2. **In robotic task scenarios (Tab. 4 and 5)**, RoG-supervised generation significantly boosts task success rates:\n  - In-domain tasks: While performance of the baseline is already strong, RoG shows benefits in longer-horizon tasks such as \"place into\", increasing the number of successful executions (Tab. 4).\n  - Out-of-distribution (OOD) tasks: RoG leads to over 2x improvement in success cases compared to the baseline version without RoG (L.430). This supports our claim that RoG enables adaptive correction and robustness in unfamiliar scenarios.\n\nThese studies support the necessity and effectiveness of the RoG mechanism. We will explicitly restructure and expand the ablation section in the revision.\n\n---\n## 2. Reproducibility and EVA-Bench Availability\nThank you for pointing this out. We fully agree with the reviewer that reproducibility and accessibility are crucial for community adoption and validation. **We will open-source the code, upload a preview-version during rebuttal.**  We include part of the data and some key code in an anonymous link https://sites.google.com/view/icml-eva#h.w1phh0qv55ho We have taken concrete steps to address these concerns as follows:\n- We will organize the appendix for more clear description, and merge important details into the main paper. More  information in appendix includes: model architecture in Sec. \"A. Appendix: Model Architecture and Training\" L660, VGM and fine-tuning method Ensamble-LoRA in sec A.3(L.740),  training detail of VLM in Sec A.5 (L.815), and detailed model architecture and hyperparameters in L.774 and Tab.7. The dataset construction and most construction prompts are also included in Fig.11, Tab.12~18. As we claimed in L.710, we noticed that the fully fine-tunned VLM on a mixture data performs better than fine-tunned on a pretrained backbone, therefore, we choose Dynamicrafter and ChaUniVi since they have a good opensource code base with comprehensive scripts on multistage training. \n\nRegarding EVA-Bench, we commit to releasing:\n- A subset of curated data examples on our project page\n- Data processing scripts and annotations\n- Download instructions for third-party datasets used (subject to licensing constraints)\nAs noted in L.817, we outlined multiple stages in the experimental pipeline, but we agree that more granularity is needed. We will expand this section to provide clearer information for the research community.\n\n---\nWe again thank the reviewer for the insightful feedback, which has helped strengthen the paper.", "claims_and_evidence": "The paper provides extensive experimental validation for its proposed methods, including comparisons across multiple baselines on diverse benchmarks. The introduction of EVA-Bench strengthens the evaluation by systematically assessing both in-domain and out-of-distribution (OOD) performance. However, while the empirical results demonstrate improvements, some claims about the necessity of Reflection-of-Generation (RoG) for long-horizon video prediction could be better substantiated with ablation studies isolating its impact.", "methods_and_evaluation_criteria": "The proposed Reflection-of-Generation (RoG) approach aligns well with the challenge of making video prediction models more robust and adaptive in embodied scenarios. EVA-Bench provides a well-structured evaluation framework, covering multiple levels of task complexity, including goal completion and adaptive reasoning. While the benchmarks are relevant and diverse, additional real-world deployment results could further validate the applicability of the method beyond simulated environments.", "theoretical_claims": "No theoretical claims are required for review in this work.", "experimental_designs_or_analyses": "The experimental design is generally sound, with clear task decomposition and evaluation metrics, particularly in EVA-Bench, which assesses both understanding and generation. The comparisons against baselines are well-structured, demonstrating EVA’s advantages in long-horizon video prediction, but some evaluations, such as the role of RoG in performance gains, could benefit from stronger ablations.", "relation_to_broader_scientific_literature": "The paper builds on prior work in world models and video prediction, integrating ideas from pretrained vision-language models (VLMs) and autoregressive video generation to improve long-horizon consistency.", "essential_references_not_discussed": "Not found", "other_strengths_and_weaknesses": "Strengths\n\n1. The Reflection-of-Generation mechanism is an innovative approach that introduces intermediate reasoning steps into video generation, allowing for self-correction and adaptive long-horizon forecasting. This is a meaningful departure from traditional video diffusion models, which often struggle with consistency over extended sequences.\n\n2. The introduction of EVA-Bench provides a well-structured evaluation framework that assesses both in-domain and out-of-distribution generalization, addressing a critical gap in embodied AI evaluation. The decomposition of tasks into action description, task completion verification, and next-step prediction enhances the clarity of model assessment.\n\n3. The EVA model demonstrates state-of-the-art performance across multiple video generation and robotics applications, showing robust transferability to real-world scenarios. By evaluating EVA on robot simulation and embodied interaction datasets, the paper strengthens its claims of applicability beyond purely synthetic settings.\n\nWeakness\n\n1. While the paper presents quantitative improvements, it lacks fine-grained ablation studies isolating the contribution of RoG beyond basic comparisons. A more detailed breakdown of RoG’s impact (e.g., performance with vs. without intermediate reasoning steps) would better justify its necessity.\n\n2. Some methodological details, particularly regarding the training setup for EVA and the specific architecture choices, could be better documented for reproducibility. While EVA-Bench is a strong contribution, it would benefit from open-source code or dataset access, enabling broader validation by the research community.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dOU1L3eCho", "reviewer": "Reviewer_wGih", "summary": "This paper introduces Reflection of Generation, a set of reasoning strategies aimed at improving video generation models for multi-step predictions and Out-of-Distribution scenarios. To support RoG, the authors propose the Embodied Video Anticipation Benchmark  to evaluate world models across diverse tasks, and they develop Embodied Video Anticipator , a model that leverages multistage training and autoregressive strategies for high-fidelity video generation. Experimental results demonstrate EVA's effectiveness in video prediction and robotics.\n\nUpdate after rebuttal: Thanks for the detailed rebuttal. I appreciate the authors' efforts in addressing my concerns and helping me better understand the work. Therefore, I will maintain my original rating Weak Accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal to Reviewer wGih\nWe sincerely thank the reviewer for the thoughtful feedback and recognition of our contributions. Below, we address the key concerns raised:\n\n## 1: Accumulated Errors in Long Video Generation\nWe appreciate the concern regarding error accumulation in long-horizon video generation with autoregressive frame extension. To mitigate this, we adopt the following strategies:\n1. Randomized Start Frame Conditioning: During training, we randomly sample starting frames to improve generalization across temporal contexts.\n2. Contextual Inference: During inference, while we typically extend from the last frame, we also support using the last 4–8 frames as context, which enhances temporal consistency.\n3. Hyperparameter setting: Frame Stride (FS): We empirically adjust the FS to limit the final length, reducing the risk of error propagation on very long videos>15s.\n\nRobotic video generation poses greater challenges than generic video synthesis, due to the need for object consistency, goal completion, and temporal reasoning. Our proposed RoG mechanism, by incorporating intermediate reasoning from a VLM, adaptively selects high-quality generations and provides a self-corrective feedback loop. This significantly enhances long-term coherence without requiring frame-level supervision.\n\n## 2: Why Only Fine-Tune ChatUniVi in Table 1 Instead of Qwen2\nThank you for this valuable observation. As shown in Table 3, ChatUniVi+EVA-Gen slightly outperforms Qwen2+EVA-Gen in terms of GCE, particularly in zero-shot settings. This motivated our decision to select ChatUniVi as the default model for fine-tuning in Table 1. Moreover, as discussed in Line 710, we aim to explore the benefits of fully pretraining a VLM before integration.In practice, ChatUniVi offers a more accessible codebase and better support for multi-stage tuning, thus making it more suitable for our current RoG-focused evaluation. This makes it more suitable for our experiments focused on architecture-level validation of RoG.\n\nWe fine-tune Qwen2 directly on the EVA dataset. Without multi-stage alignment and mixture data pertaining, Qwen2-Full Parameter fine-tuning on 300k data(Qwen2-FP) or 50k data (Qwen2-FP-50k) overfits and suboptimal generalization (weak in CLIP and GPT-4o score). Therefore, we still recommend a multi-stage training strategy for stronger VLMs to fully leverage RoG’s potential.\n\n|Model|In-Domain|BLEU-1 ↑|METEOR ↑|R-L ↑|CIDEr ↑|SPICE ↑|CLIP ↓|GPT-4o ↑|\n|----|----|----|----|----|----|----|----|----|\n|ChatUniVi| × |0.0969|0.0640|0.1497|0.0427|0.0636|27.49|9.03|\n|Qwen2| × |0.2484|0.1434|0.3255|0.8914|0.2839|28.98| 29.58|\n|Qwen2-FP-50k| ✓ |0.4189|0.2424|0.5005|2.2336|0.4404|26.73|27.19|\n|Qwen2-FP| ✓ |0.4833|0.2443|0.5129|2.7301|0.5363| 26.39|24.12|\n|ChatUniVi-LoRA| ✓ | 0.3007|0.1054|0.3268|0.8245|0.2213|24.89|31.94|\n|ChatUniVi-FP| ✓ | 0.4105|0.1809|0.4416|1.9012|0.3414|25.36|38.46|\n\n## 3: Baseline Training Status in Table 3\nThank you for pointing this out. We clarify that the baselines in Table 3 were trained to ensure fairness:\n- Qwen2-FP + EVA-Gen, note as EVA(Qwen) was fine-tuned on 300k same downstream data as indicated in the method section.\n- EVA(Qwen)-2stages better than Qwen2: This supports our broader claim that RoG is a model-agnostic reasoning framework: different VLMs can benefit from being embedded into the RoG pipeline for robotic video generation.\n- EVA is still SOTA: multi-stage alignments are important.\n\n|Task|Model|EVAS-L ↑|EVAS-V ↑|EVA-Score ↑|\n|-----|-----|-----|-----|------|\n|**HOW-TO**|EVA(Qwen)-2Stage|88.49|71.72|80.11|\n| |Qwen2+EVA-Gen|41.54|69.34|55.44|\n| |EVA-2Stage|85.5|73.32|79.42|\n|**Next-Step**|EVA(Qwen)-2Stage|75.54|63.83|69.69|\n| |Qwen2+EVA-Gen|42.99|60.11|51.55|\n| |EVA-2Stage|73.02|64.46|68.74|\n\n## Question: Why video generation\nIt enables us to take hypothetical actions without affecting the real environment, facilitating a low trial-and-error cost.[1]\nCompared to simulators, video generation doesn’t need detailed modeling. Unlike real-world testing, it’s more scalable and cost-efficient. Therefore, it can offer a promising foundation for building scalable and generalizable robotic systems in the real world.\n\n[1] Ding J, Zhang Y, Shang Y, et al. Understanding World or Predicting Future? A Comprehensive Survey of World Models[J]. arXiv preprint arXiv:2411.14499, 2024.\n\n## Summary\nQwen2 is a stronger foundation model, and our supplementary results show that EVA based on Qwen2 also performs well. However, directly fine-tuning such large models can lead to catastrophic forgetting without multi-stage alignment.\nThis does not affect our main claim: RoG is a model-agnostic framework that consistently improves temporal coherence and goal completion in robotic video generation. We also highlight the importance of hybrid data and staged training to fully unlock model potential.\nAs more powerful VLMs emerge, we believe EVA combined with RoG will continue to advance the field of video prediction.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Yes. See \"Weaknesses\" for issues.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. The proposed RoG strategy enhances video prediction by integrating intermediate reasoning steps, enabling self-correction and improved generalization in embodied scenarios.\n2. The introduction of EVA-Bench provides a comprehensive and standardized evaluation framework for world models, assessing both understanding and generation capabilities across diverse in-domain and OOD embodied tasks.\n3. Extensive experiments demonstrate the effectiveness of EVA in various downstream tasks.\n\nWeakness:\n1. The paper does not provide a quantitative comparison of long video generation. How does EVA mitigate the potential issue of accumulated errors in long video generation using Autoregressive Frame Extension?\n2. In Table 1, why does the In-Domain experiment only fine-tune ChatUniVi, while the stronger Qwen2 model is not included in the experiment?\n3. In Table 3, have the baseline methods, LLAVA-O+EVA-Gen and Qwen2+EVA-Gen, been trained? If not, the comparison may not be fair.\n\nMinor typos:\n1. Line 340: \"ChatUniVi-loRA\" should be corrected to \"ChatUniVi-LoRA.\"\n2. Table 3: Incorrectly bolded values.", "other_comments_or_suggestions": "See Weaknesses.", "questions_for_authors": "Why is the task of generating future frames considered significant? Research in simulation environments or even on real-world machines may hold greater practical relevance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nbHKVV2n0U", "reviewer": "Reviewer_aEuy", "summary": "In this work, the authors propose a  Reflection of Generation (RoG) solution to enhance video prediction in embodied scenarios. To achieve it, they further introduce an Embodied Video Anticipation Benchmark (EVA-Bench) and an Embodied Video Anticipator (EVA) model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Rebuttal to Reviewer aEuy\nWe sincerely thank the reviewer for the insightful comments and for highlighting the relevance of Pandora. We also appreciate your recognition of the strengths of our work, such as “most claims are supported by clear evidence” and “the methods and evaluation basically make sense for video prediction.” Although both EVA and Pandora share the general goal of integrating vision and language for video generation, we differ substantially in purpose, methodology, and impact. We will explicitly include a comparison with Pandora in the revised manuscript. Below, we provide a point-by-point response to clarify the key differences between EVA and Pandora and why these distinctions are essential.\n\n## 1. Conceptual Difference (RoG Solution Style vs. Pandora’s Approach):\nThe primary contribution of EVA is the Reasoning-over-Generation (RoG) strategy, which introduces a structured thinking paradigm for unified understanding-generation tasks. RoG is analogous to the Chain-of-Thought (CoT) reasoning in LLMs, enabling stepwise reasoning and decision-making in multimodal models. Unlike Pandora, which is primarily a multimodal-to-video generation model, RoG is a generalizable strategic framework that can be applied to any unified understanding-generation model, including Pandora or other LLM+VGM frameworks(Tab. 2 & 3, or we newly add EVA(Qwen) version in https://openreview.net/forum?id=onumui0nHi&noteId=cunTZeC5Vq ). As claimed in our experiments, e.g. Table 2, we demonstrate how RoG enhances reasoning capabilities across different models.\n## 2. Benchmarking Contribution (EVA-Bench vs. Pandora’s Dataset):\nEVA-Bench is specifically designed to evaluate both visual understanding and generation abilities in embodied scenarios, which include multiple text metrics(Tab.1), video metrics like goal completion estimation(Tab.2, L.1104, Fig.12), and multimodal generation and reasoning metrics(Tab.). Pandora’s dataset, although it includes some robot-related videos, only has a visual evaluation. EVA-Bench incorporates manual caption adjustment, case separation, and task-specific design to ensure fine-grained evaluation. Additionally, in generation evaluation, we introduce goal-conditioned estimation metrics, which are crucial for benchmarking embodied AI tasks. This makes EVA-Bench distinct from Pandora’s evaluation setup.\n## 3. Model Difference (EVA Model vs. Pandora’s Model):\n  EVA and Pandora share the general idea of leveraging multimodal foundational models (ChatUniVi+Dynamicrafter), our approach emphasizes reasoning-driven generation, rather than direct multimodal-to-video synthesis. Therefore, our training stages and alignment targets, including loss design are different from Pandora.\n  1. **Input-Output Difference (QA-Text&Video vs. Text-Video):**\nA key distinction between EVA and Pandora lies in input and output. EVA allows Question-Video input and Answer-Video as the response. While Pandora only accepts text instructions as input, and its text output also simply copies the input instruction. This dual-modality output in EVA is crucial for complex reasoning tasks, as it allows the model to explicitly describe, explain, and refine its thought process before generating video sequences. By incorporating textual reasoning alongside video synthesis, EVA significantly enhances its ability to handle complex multi-step tasks, making it more suitable for embodied AI and decision-making applications.\n  2. **Differences in Training Data and Strategies:**\nAs detailed in the Appendix, EVA employs a unique instruction-tuning approach for text-video interleaved generation. We carefully design training data that spans four meta-tasks under the RoG framework, ensuring comprehensive coverage of embodied task reasoning. During training, we first used mixed data to align the model, similar to Pandora. Then we did one more step as QA instruction tuning. The second step differs from Pandora’s data construction, which does not explicitly incorporate interleaved text-video reasoning in the same structured way.\n  3. **Video-to-Action Module**\nPandora is a general video generation world model. While EVA also includes a base video-to-action head that could further transfer video generation to robot action (Table 5), connecting the generation model with the real world. In EVA, we can achieve end-to-end QA-to-Robot manipulation tasks, which are also far different from Pandora. （L. 450)\n\nIn summary, while EVA and Pandora share a broad vision, our work introduces a novel reasoning strategy (RoG), a dedicated embodied AI benchmark (EVA-Bench), and a structured training approach tailored for reasoning-enhanced generation. These distinctions ensure that EVA provides unique contributions beyond Pandora. We include the data and benchmark format in our anonymous pages, for a more detailed and straightforward comparison.\n\nWe appreciate the reviewer’s feedback and will further clarify these differences in the revised manuscript.", "claims_and_evidence": "Most claims are supported by clear evidence.", "methods_and_evaluation_criteria": "The methods and evalution basically make sense for video prediction within the proposed RoG style.", "theoretical_claims": "No theoretical claims are introduced in the paper.", "experimental_designs_or_analyses": "The experiments are basically sufficient to support the design.", "relation_to_broader_scientific_literature": "The paper provides a possiblity for unifying visual understanding and generation tasks.", "essential_references_not_discussed": "The model style is similar to  [Xiang, et al. Pandora: Towards General World Model with Natural Language Actions and Video States]. Please further make the comparison to show the key difference.", "other_strengths_and_weaknesses": "There are three contributions including RoG solution style, EVA-Bench, and EVA model. However, such style and model has been partially investigated in the [Xiang, et al. Pandora: Towards General World Model with Natural Language Actions and Video States].  This would reduce the importance of RoG and EVA contributions in this paper, especially model structure is similar to Pandora. Please clarify the key differences.", "other_comments_or_suggestions": "Please see the weakness.", "questions_for_authors": "Please see the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hai2Fd7U0o", "reviewer": "Reviewer_cisM", "summary": "This paper proposes a world model based on video prediction. The world model is specifically designed for embodied AI (manipulation, more specifically). The authors design a novel strategy, termed Reflection of Generation (RoG), to leverage VLM and video generation models to serve as a world model. Besides, this paper also introduce a benchmark to evaluate embodied world models. Basically this is a good submission, with reasonable methodology and extensive experiments.\n\n**Update after rebuttal**: I appreciate the authors' detailed response. The response has addressed most of my concerns, including the paper organization, video duration and open source. After reading the rebuttal and the comments from other reviewers, I choose to keep my original score, i.e., weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer cisM\nWe thank the reviewer for recognizing the novelty of RoG, the value of our benchmark, and the thoroughness of our experimental results. Your constructive feedback helps us further improve the clarity and rigor of the paper.\n\n---\n## 1. Paper Organization and Benchmark Presentation\nThank you for pointing out the confusion regarding the structure of the paper. As our primary contribution lies in the design of a reasoning framework akin to Chain of Thought (CoT) [1], we initially organized the paper to mirror this inspiration. Specifically, we introduced the system framework (RoG) first, followed by the dataset and benchmark (Sec.4), and then the model (Sec.5), to reflect the flow of reasoning.\nHowever, we agree that the current structure may appear unconventional and potentially confusing. Upon your suggestion, we have revised the organization for better clarity and coherence. In the updated version, we will make the following changes:\n- Move Section 4.1 (Task Decomposition) to the end of Section 3.\n- Swap the order of Section 4 (Benchmark) and Section 5 (Model).\n\nThe revised structure is now:\n- RoG Framework: Task Definition, Task Decomposition\n- Model: VLM, LLM, AR Generation\n- Dataset and Benchmark Design\n- Experiments\n\nWe believe this reorganization better reflects the logical flow from problem formulation to model design, and then to evaluation. Furthermore, due to the complexity and scale of the foundation models used, we include extensive implementation, fine-tuning, and ablation details in the Appendix.\n\n---\n## 2. Short Video Duration in Prediction Results\nThank you for raising this point. To address this concern, we have provided additional longer video prediction results (8+ seconds) on our anonymous project website under the section \"Multi-Round RoG Longer Video Generation\": https://sites.google.com/view/icml-eva#h.5u9bchnr2e41\nThese extended demos demonstrate the model's ability to perform multi-round robotic interaction tasks over longer horizons. We hope this provides a clearer view of the model's temporal reasoning and generation capabilities.\n\n---\n## 3. Visual Artifacts in Generated Videos\nWe appreciate your observation regarding visual artifacts. All evaluation cases were randomly selected to ensure fairness and reproducibility. While some artifacts are indeed present, such as in \"move brown chip bag near blue chip bag_frame_0_sample0.mp4\"—we note that this challenge is not unique to our method. Object consistency in generation, especially deformable objects (e.g., chip bags), remains an open and challenging problem in the video prediction community [2,3].\nOur focus in this work is on generating coherent and temporally consistent robotic behaviors, especially in multi-step tasks. That said, we acknowledge the importance of improving object-level consistency. As part of our future work, we plan to:\n- Introduce spatial consistency constraints during the understanding phase.\n- Integrate additional consistency modules in the video generation phase, particularly for egocentric views (e.g., robot wrist cameras) to better capture fine-grained object interactions.\n\n## Question: Open Source\nYes, we plan to open-source both the code and the benchmark data. The current supplementary code is marked as “NOT RUNNABLE YET” because it is still in the experimental stage and tightly coupled with our internal development environment. To ensure broader usability and reproducibility, we are actively working on:\n\n- Cleaning and refactoring the codebase\n-  Packaging the runnable demo\n- Providing environment setup scripts and installation instructions\n- Writing a detailed user manual\n\nWe are committed to releasing a public version shortly after the review process, in line with community standards for reproducibility.\n\nWe thank the reviewer once again for highlighting this important area for improvement.\n\n---\n# References\n[1] Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models[J]. Advances in neural information processing systems, 2022, 35: 24824-24837.\n\n[2] Ren, Weiming, et al. \"Consisti2v: Enhancing visual consistency for image-to-video generation.\" arXiv preprint arXiv:2402.04324 (2024).\n\n[3] Xu, Ziyi, et al. \"AnchorCrafter: Animate CyberAnchors Saling Your Products via Human-Object Interacting Video Generation.\" arXiv preprint arXiv:2411.17383 (2024).", "claims_and_evidence": "Yes, the claims are basically supported.", "methods_and_evaluation_criteria": "Yes, this paper propose a benchmark for evaluation.", "theoretical_claims": "Yes, I check the algorithm of the paper, which is reasonable.", "experimental_designs_or_analyses": "Yes, I check the benchmark design.", "relation_to_broader_scientific_literature": "The key contributions are related to the world model, which is a hot topic in embodied AI.", "essential_references_not_discussed": "I do not find any.", "other_strengths_and_weaknesses": "Strengths:\n1. The RoG method is interesting and novel to me.\n2. The inclusion of a benchmark is valuable for evaluating the method's performance.\n3. The paper provides comprehensive experimental results, effectively demonstrating the method's effectiveness.\n\nWeaknesses:\n1. I'm a little confused by the overall organization of the paper. The benchmark should be presented to evaluate the baseline methods and the proposed method. However, now the benchmark is presented in Sec.4, the basic method is splitted to Sec.3 and Sec.5, which is strange. Besides, Sec.6 describes various details about the evaluation, I'm confused about the difference between this part with Sec.4 (and Appendix C).\n2. The video prediction results are a bit short (only 2s for most videos). I'm curious about the results for longer prediction.\n3. The video prediction results exhibit noticeable artifacts, e.g., \"move brown chip bag near blue chip bag_frame_0_sample0.mp4\" in the supplementary.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Will you open source the code? The supplementary code is \"NOT RUNNABLE YET\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes the Reflection of Generation (RoG) method, leveraging intermediate reasoning to improve video prediction and use as a world model for multi-step prediction tasks (with a focus on robotics). The model (Embodied Video Anticipator) leverages multi-stage training and a VLM-supervised generation process that can support generation of extended video sequences. In order to test the method, EVA-Bench is proposed which includes both in and out-of-distribution videos within egocentric and robotics domains. 

  Overall, reviewers appreciated the problem, method, and especially the inclusion of the EVA-Benchmark to more rigorously test this problem under both in and out-of-distribution generalization settings. Some concerns were raised, including a better demonstration/justification of the necessity of RoG for long-horizon video prediction, more detailed ablation studies, better documentation of various decisions made in the design of the architecture for reproducibility, and better situation with respect to some prior methods. The authors provided a thorough rebuttal of all of these points, and all of the reviewers stated that their major concerns were addressed. 

  Considering all of the materials, I recommend acceptance of this paper. The idea is an interesting combination of VLMs and generation through reasoning mechanisms, and the benchmark has potential for increasing work in these areas in a comparable and focused manner. I strongly recommend that the authors incorporate the various discussions and paper organization feedback provided during the reviewing process to make the camera-ready version stronger.
Decision: Accept (poster)