WorldSimBench: Towards Video Generation Models as World Simulators
Authors: Yiran Qin
Zhelun Shi
Jiwen Yu
Xijun Wang
Enshen Zhou
Lijun Li
Zhenfei Yin
Xihui Liu
Lu Sheng
Jing Shao
LEI BAI
Ruimao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Embodied Vision
World Model，Dataset and Benchmark
Primary area: applications->computer_vision
TL;DR: Towards Video Generation Models as World Simulators

Abstract: Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulater. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

WorldSimBench: Towards Video Generation Models as World Simulators
Yiran Qin 123*
Zhelun Shi 4*
Jiwen Yu
Xijun Wang² Enshen Zhou Lijun Li Zhenfei Yin 6 Xihui Liu
Lu Sheng
Jing Shao 3
Lei Bai
Ruimao Zhang!" 17!
WorldSimBench
Prediction: Actionable Video
Koywords Video Action Aligmment, Physics Rulcs, Embodied Factions
Open-Ender Embodied Environment
Autonomous Driving
Robotic Manipulation
Text- Observation
S3
Evalcrofter, VBench
Visua Quality
Ancherics Foregrou ounground/ / Barkground Conektency
Prediction: Video
Keywords: Text Video Algemmnt
Condition ComsistancyO
Text Observation
Intraction Scenerio Alkgnmem
LEGO Keyounds
S2
Prediction: tmage
STOP
Text Ohservation
Kay Element
EgoPlen-Bench,
S1
Prediction: Text Text
Keywo dss Task Planning
Embodiment P
Perfpettiity
Interaation
Text Observation
Corresponding Benchmark
So
Task Definition
Traectory
Velncity
Explicit Perceptual Evaluation
Implicit Manipulative Evaluation
Figure 1. Overview of the hierarchical capabilities of the Predictive Models. Models at higher stages demonstrate more advanced capabilities. We take initial step evaluating Predictive Generative Models up to S3 stage, known as World Simulators, by introducing parallel evaluation framework, WorldSimBench. WorldSimBench assesses the models both Explicit Perceptual Evaluation Implicit Manipulative Evaluation, focusing on video generation action transformation across three critical embodicd scenarios.
Abstract
WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use train Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation- aware video can be accurately translated into the correct control signals dynamic envi- ronments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward
Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the function- alities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing dual evaluation framework called
"qqual contribution Sun Yat-sen University "The Chinese Uni- versity of Hong Kong, Shenzhen 'Shanghai Artificial Intelligence Laboratory "Beihang University 'The University of Hong Kong Uniierrsity of Oxford "Guangdong Key Laboratory of Big Data Analysis and Processing. Correspondence to: Jing Shao <shao- jing@ eee@@pllabo.. callabborg.cm org.en>>, Lei Bai <bballci @pw.aa.org.cns...... Ruimao Zhang <ruimao. Eriimao.zhange @ iecc.org>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

WorldSimBench: Towards Video Generation Models as World Simulators
embodied artificial intelligence. Project Page:
integrate robust 3D scene understanding and physical rule priors to provide precise guidance for generating executable actions. These models are closely aligned with the recently proposed concept of World Simulators (Yang et al., 2023).
1. Introduction
To answer the second question, we review the related bench- marks, as listed in Tab. 1. Evaluations on models in So that generate text primarily focus on assessing task planning capabilities, while SJ and S2 assessments on visual output measure aesthetic quality through feature similarity analy- ses with ground truth data. With clearly defined evaluation dimensions and extensive annotated datasets, both types of assessments can be effectively conducted. However, evalu- ating World Simulators introduces complexities due to the intricate physical definitions involved. Additionally, con- ventional evaluation methods are inadequate for assessing the actionablilty of the generated videos, as there is no defi- nite ground truth for actionable videos towards completing specific embodied task. These factors pose significant challenges to the evaluation of World Simulators.
Before taking action, humans make predictions based on their objectives and observations of the current environment. These predictions manifest in various forms, e.g., textual planning, visual imagination of future scene changes, or even subconscious planning at the action level. With the development of generative models, agents driven by these models are exhibiting predictive capabilities that enable them to complete embodied tasks by making human-like predictions, e.g., high-level planning (Driess et al., 2023; Li et al., 2024), image-based guidance (Lai et al., 2023; Black et al., 2023), or future video prediction to drive actions (Du et al., 2023; 2024)). We refer to these models as Predictive Models. Recently, these models have been widely applied across various domains spanning from developing agents to solve inference tasks to leveraging predictions for driving robots to perform specific actions.
We argue that an evaluation aligned with human percep- tion could provide a more intuitive and accurate reflection of the characteristics of the synthesized videos, including their adherence to physical rules. Besides, the actionability can be assessed through a closed-loop manner in simula- tions deployed with a unified work. Considering these aspects, we take the very first step in evaluating World Simulators by proposing a dual eval- uation framework called WorldSimBench. As shown in Fig. 1, WorldSimBench assesses World Simulators through two complementary approaches: Explicit Perceptual Eval- uation, which focuses on the Visual Quality, Condition consistency, and Embodiment of the generated content, and Implicit Manipulative Evaluation, which measures the World Simulator's 's performance through the conversion of video into control signals. We present three representative embodied scenarios: Open-Ended Embodied Environment (OE), Autonomous Driving (AD), and Robot Manipula- tion (RM), to thoroughly evaluate the capability of World Simulators in generating and representing scenario-specific attributes.
Nevertheless, the rich application scenarios and diverse model designs make predictive models broad family. How- ever, without categorizing them based on their inherent char- acteristics, the advancement of predictive model develop- ment remains limited. This leads to our first question: Can we establish reasonable hierarchical system for Predictive Models based on their outpul modality? With a well-defined categorization, we can better target the evaluation of Predic- tive Models from different perspectives in diverse embodied environments, ensuring that their strengths and weaknesses are adequately assessed. In the literature, existing evalua- tions have typically focused on task planning capabilities by assessing text outputs or evaluating visual outputs from an aesthetic perspective. However, such approaches signif- icantly limit the evaluation of highly embodied Predictive Models, as embodied scenarios are more concerned with physical properties (e.g.. perspective consistency, object breakability), which these methods fail to effectively assess. This brings us to our second question: Can we conduct more detailed evaluation of highly embodied Predictive Models from an embodied perspective?
aocotoon policy net-
In the Explicit Perceptual Evaluation, we first define evalua- tion criteria which is used to construct comprehensive set of prompts specific to each scenario. The prompt lists are then used by various video generation models to produce large number of video clips. Following extensive human feedback and annotation, these video clips are compiled into the HF-Embodied dataset which consists of a total of 35,701 tuples with multi-dimensional scores and fine- -aee-graned hu- man feedback. Additionally, we train Human Preference Evaluator, using the HF-EEmodiie dataset to assess World Simulators at the perceptual level, offering robust evalu- ation of both their visual fidelity and contextual accuracy.
To answer the first question, we categorize the functional- ities of Predictive Models into a hierarchy from So to Sa, defined by the model's capabilities and output modality, accompanied by corresponding evaluation benchmarks as illustrated in Fig. 1. Models are classified based on the output modality in their output modalities. From lower to higher stages, the models are capable of generating: text, images, videos, and actionable videos (i.e., the videos that can be translated into actions). It is worth noting that Predic- tive Models at S3 capable of generating actionable videos
2

---

WorldSimBench: Towards Video Generation Models as World Simulators
Table Comparisons between existing Predictive Model benchmarks. Interactive Environment refers to the interaction with the simulation environment during the prediction phase. Task-Level Interaction denotes that each task interacts once, whereas Action-Level Interaction represents the frequency of interactions that occur through the generation actions for control purposes.
Benchmark
Input Modality
Ourput Modality
Based Method
Stage
Interactive Ens.
Evaluation Strategy
AgentBench (1 än et al., 2023b)
Text
Text
L.M
S0
Tack-I Level
Human Judgement
EgoPlan-Bench (Chen et 2023)
Text & Images
Text
MLLM
So
N/A
Multi-choice
MMWorld (He et al., 2024)
Text & Images
Text
MLLM
So
N/A
GPT Judgement
VAB (Liu al., 2024a)
Text & Images
Text
MLLM
So
Task-Level
Human Judgement
LEGO (Lai et al., 2023)
Text & Images
Image
IGM
S1
Task-Level
Feature Similarity
VBench (Huang et al., 2024)
Text
Video
VGM
S2
N/A
Feature Similarity
EvalCrafter (Liu al., 2024h)
Text & Images
Video
VGM
S2
N/A
Feature Similarity
Human Preference Evaluator Embodied Metric
WorldSimBench
Text & Images
Actionable Video
VGM
53
Action Level
For the Implicit Manipulative Evaluation, we deploy three simulation environments for the three embodied scenarios respectively. These environments are used to collect data and train inverse dynamic or goal-based video-to-action models capable of mapping future videos to actions. In each of these embodied scenarios, the World Simulator is tasked with generating situation-aware videos in real-time, based on current observations and provided text instructions. These generated videos are then converted into actions using the pre-trained aadoo-o-actiion models. The effectiveness of the World Simulator is implicitly evaluated by measur- ing the performance of the tasks, using relevant metrics to reflect the quality and accuracy of the generated video.
Stage So: At this stage, predictive models can generate corresponding predictions based on instructions and obser- vations but are limited to textual modality. Benchmarks at this stage conduct text-level and task-completion evaluations through output text planning.
Stage S At this stage. predictive models can generate vi- sual predictions based instructions and observations, but without incorporating temporal information. Benchmarks at this stage conduct aesthetic evaluation for generated images.
Stage S2: At this stage, predictive models can generate corresponding video predictions based on both instructions and observations. Yet, due to limited model capabilities, the evaluation at this level focuses solely on the aesthetic quality of the generated outputs.
In summary, the main contributions are as follows: (1)We categorize the funetionalities of Predictive Models into hierarchy, defined by the model's capabilities and output modality, to advance research and development in the field and take the very first step in evaluating World Simula- tors. We)We propose a dual evaluation framework called WorldSimBench, through Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, we conducted com- prehensive evaluation of the World Simulator's capabilities from an embodied perspective, focusing on both the vi- sual and action levels. We conducted extensive testing across multiple models and performed a thorough analy- sis of the experimental results. Our findings highlight the strengths and limitations of current World Simulators and provide actionable insights for improving future video gen- eration models. (4)We developed HF--Emoodied Dataset, which includes fine-grai gine-grained human feedback across three scenarios and 20 dimensions, with total of 35,701 entries. This dataset, containing both human ratings and the reasons behind them, not only enables the evaluation of World Simu- lators but also provides broader applications (alg..aligmment) for future video generation models.
Stage S3: At this stage, predictive models can generate corresponding video predictions based on instructions and observations, with the predicted video content adhering to physical rules and aligning with the executed actions. These models are known as World Simulators (Ha & Schmid- huber, 2018; Yang et al., 2023), and WorldSimBench is a benchmark specifically designed to evaluate these World Simulators.
The rapidly evolving field of World Simulators offers ex- citing opportunities for advancing Artificial General Intel- ligence, with significant potential to enhance human pro- ductivity and creativity, especially in embodied intelligence. Therefore, conducting comprehensive embodied evalua- tion of World Simulators is crucial.
3. WorldSimBench Construction
WorldSimBench evaluates the embodied capabilities of World Simulators across two distinct levels. The Explicit Perceptual Evaluation assesses the simulators based on numan-perceived quality across different embodied scenar- ios, while the Implicit Manipulative Evaluation implic- itly evaluates the simulators' capabilities by converting the generated videos into control signals and observing their performance in various closed-loop embodied tasks.
2. Predictive Model Category Definition
In this section, we concretely categorize predictive models based on the model's is capabilities and output modality. The detailed categorization stage of Fig. 1 is illustrated below,
3

---

WorldSimBench: Towards Video Generation Models as World Simulators
Instruction Prompt Generation
Massive Video Captions
Diverse Task Instruction Prompt Generation
Task Instruction Prompt List
vides timouarstio rotocis arm
(Neves posttton)
Keyword: Nove Example: "Nove the robot
box
Separate prompts for:
itself (above
LLN: "Rove the rebot ore on top of the white plastie cup"
"Move the ratut are on top of the red button white black buctue"
Each evaluation dimension
"rove the roboc are on top of the black handset of the telophons"
Each embodied scenario
Embodiment Dimensions
Keyward: Press
Exmple: "Press the red desk lamp butten"
'oollect send on the plata"
Velecity
Perspactivity
Pe Human-Level Instruction Prompt Checking
"Hove the robot are on top of pink cloth."
Interaction iors
Trajectory
"Move the robot orn under the flooting red cup." " 'SLide the robot orn through the soLid netol block."
"Exocute right turn et the upcoring Intersection.
Key
HF-Embodied Dataset Generation
Massive Videos with Caption
Data Goneration Model Training
Vikeo
Modi Scree
Diverse Task Instruction Prompt List
P2
Fine-grained Human Feedback Anotation
Oafle
"Move
of pine clecn."
Matste conttorcticiiisr the degree"
Dinension: Trajectory
greee gray
bothoe
Score
1 2
Open
Othir Iroks
Sere
Reason In segnents, the nation trajectory of the are is inconsistent
"CLose the gripper
Figure 2. Overview of Explicit Perceptual Evaluation. (Top) Instruction Prompt Generation. We use a large collection of video captions from the internet and our predefined embodied evaluation dimensions. These are expanded using GPT and manually verified to create corresponding Task Instruction Prompt List for data generation and evaluation. (Bottom) HF-Embodied Dataset Generation. Massive internet-sourced embodied videos with captions are used to train data generation models. Fine- tine-grained Human Feedback Annotation is then applied to the embodied videos according to the corresponding Task Instruction Prompt List, covering multiple embodied dimensions.
The evaluation of World Simulators encompasses three crit- ical embodied scenarios: Open-Ended Embodied Environ- ment (OE), Autonomous Driving (AD), and Robot Manipu- lation (RM). Minecraft serves popular testbed for OE, providing challenging platform for agents to handle com- plex, unstructured tasks. In the context of AD, especially in outdoor settings, ensuring the stability and robustness of the agent's actions is crucial, making it an essential domain for assessing World Simulator's capability in dynamic and uncertain environments. RM, a core task in embodied intelligence, demands precise and adaptive control, testing the world simulator's s ability to generate actionable predic- tions that align with physical interactions. Together, these scenarios provide a comprehensive benchmark for evaluat- ing the effectiveness of World Simulators across a range of real-world tasks.
text instructions for RM (Chen et al., 2024). Using HF- Embodied Dataset, we train a Human Preference Evaluator to perform perceptual evaluations of World Simulators.
3.1.1. HIERARCHICAL EVALUATION DIMENSION
We develop a hierarchical evaluation dimension checklist for the three embodied scenarios, as illustrated in Tab. 2, which can be categorized into three main aspects: Visual Qual- ity, Condition Consistency, and Embodiment. (1) Visual Quality primarily assesses the overall quality of video gen- eration, including Aesthetics, Background and Foreground Consistency. (2) Condition Consistency focuses on the alignment with the input instruction. For tasks in OE that involve distinct scenarios, we additionally define Scenario Alignment to assess the alignment to the specific scenarios outlined in the instruction. (3) Embodiment has different definitions depending on the scenario. As all tasks require movement along certain trajectory, we uniformly define Trajectory to evaluate the rationality of object movement in the video (e.g., whether robotic arm avoids obstacles during motion). In AD and RM, we define Perspectivity to assess whether the video exhibits a clear sense of depth. In OE and RM,, we define Embodied Interaction to evaluate the plausibility of interactions with objects. We also define Velocity in OE to determine whether speed varies appropri- ately across different environments (e.g., slower movement in water). In AD, we define Key Element to evaluate the
3.1. Explicit Perceptual Evaluation
In Explicit Perceptual Evaluation, we propose Hicrarchical Evaluation Dimensions, based on which we build a video assessment dataset annotated through fine-grained human feedback, named HF-Embodied Dataset. The dataset is constructed based on three key resources, each correspond- ing to a specific embodied scenario: a curated dataset of Minecraft videos from the internet for OE (Baker et al., 2022), real-world driving data for AD (Caesar et al., 2020), and worl-world robot manipulation videos annotated with
4

---

WorldSimBench:: Towards Video Generation Models as World Simulators
Table Hierarchical Evaluation Dimension. The dimensions are categorized into three main aspects: Visual Quality for evaluating the overall quality, Condition Consistency for evaluating the alignment to the input instruction, and Embodiment for evaluating embodied related factors like physical rules.
Embodied Scenarios
Visual Quality
Condition Consistency
Embodiment
Velocity (VC)
Background Consistency (BC)
Instruction Alignment (IA)
Open-Ended Embodied Environment (OE)
Trajectory (TJ)
Foreground Consistency (FC)
Scenario Alignment (SA)
Embodied Interaction (EI)
Perspectivityy (PV)
Trajectory (TJ)
Autonomous Driving (AD)
Aestheties (AE)
Instruction Alignment (IA)
Key Element (KE)
Safety (SF)
Aesthetics (AE)
Perspectivilly (PV)
Robot Manipulation (RM)
Background Consistency (BC) Foreground Consistency (FC)
Instruction Alignment (IA)
Trajectory (TJ
Embodied Interaction (EI)
rendering quality and consistency of crucial embodied el- ements, e.g., pedestrians. We also introduce Safety in AD to assess whether the embodied actions comply with traffic rules. More details in Sup. B.
clusively on the generation quality of the motion trajectory. They are instructed not to consider other elements (e.g., g. the rendering quality of the robot arm) or other dimensions (e.g,, consistency with instructions). Additionally, anno- tators are asked to provide fine- tne-grained feedback on any deficiencies, e.g., "inconsistent trajectory". As result, we construct the HF-Embodied Dataset, which consists of a total of 35,701 tuples, each comprising a video, text instruc- tion, multi-dimensional scores, and the potential reasons. More details in Sup. C.I.
3.1.2. INSTRUCTION PROMPT GENERATION
Using the Hierarchical Evaluation Dimension and massive video captions from the key resources, we create a founda- tional but comprehensive prompt list. We utilize the knowl- edge of LLMs, i.e. ChatGPT, to extend the meta-prompts across a wide range. After manual screening for relevance, diversity, and data distribution, struction Prompt List, which separates prompts for each content-embodied scenario and each evaluation dimension, as shown in Fig. 2.
3.1.4. HUMAN PREFERENCE EVALUATOR
compile the Task In-
The objective is to develop a video scoring model that as- sesses videos across multiple dimensions aligning with hu- man perception. The model takes a generated video and a prompt as input and outputs a score ranging from 1 to n (n is defined specifically for each embodied scenario).
3.1.3. HE-EMBODIED DATASET GENERATION
The prompt includes both the video generation instructions and an explanation of the evaluation criteria. Leveraging the strong video understanding capabilities of multimodal large language models, we fine-tune Flash-VStrean VStream (Zhang et al., 2024a), a VideoLLM, aligning it with human per- ception on HF-Embodied Dataset. Only LoRA (Hu et al., 2021) parameters are trained. This enables the model to effectively grasp the evaluation metrics for embodied tasks and produce accurate scores, while maintaining its video perception and reasoning ability. We prove the effectiveness and generalizability of our Human Preference Evaluator in Sec. 4.2.
Data Preparation. We select multiple video generation models and train them using a large corpus of videos and corresponding captions from the key resources. Due to the capabilities of the open-source video generation model, we conduct targeted training for each of the three distinct em- bodied scenarios individually. thereby developing several data generation models for different embodied scenarios. These models are then used to produce a substantial amount of caatuctioo-followinng embodied videos, based on the cor- responding captions, and the initial image condition where applicable (first frame conditioned text-to-video to generate situation-aware videos).
Human Annotation. We use human annotation to label the generated videos. Based on the Hierarchical Evalua- tion Dimension, we establish specific annotation guidelines and numerous in-conttext examples for the annotators. For each dimension, annotators are instructed to score the video solely based its performance within that particular di- mension and provide corresponding reasoning. For instance in RM, as illustrated in Fig. 2, under the dimension of Tra- jectory, annotators are required to evaluate the video ex-
1.5. EVALUATION METRICS.
The evaluation of a video generation model is based on the scores assigned by the evaluator across various dimen- sions. For each dimension, the video generation model generates videos guided by several carefully selected in- structions sourced from Task Instruction Prompt List that are strongly aligned with the specific evaluation criteria, e.g., "explore on the beach" for Embodied Scenario in OE. The
5

---

WorldSimBench: Towards Video Generation Models World Simulators
"Gather row naterials and nake wooden axes in Minecraft."
2. Crafte wood planks Craft wood sticks
I. Collect wood
Generate Current Observation
Environment
"saatt the cany take right turn, and keep going."
J. Stort the can
L[A]: "close the grippen
Monitor
2. Turn right
Evaluate Task Performance
3. Go straight
Highilevel Planner
H
Lift the red block froe the sliding cabinet and store it the drower."
I. Life the red block from the sLtating cabinec
2. the grasped block tn drawor
World Simulator Models
Video-Action Mapping
Video Prediction
Task
Sub-Task List
Interactive Evaluation Process
Figure 3. Overview of Implicit Manipulative Evaluation. Embodied tasks in different scenarios are decomposed into executable sub- tasks. The vidco generation model generates corresponding predicted vidcos based the current instructions and real-time observations. Using pre-trained IDM a goal-based policy, the agent executes the generated sequence of actions. After a fixed timestep. the predicted video refreshed by sampling again from the video generation model, and this process repeats. Finally, the success rates of various embodied tasks are obtained through monitors in the simulation environment.
final metric for each model is computed as the average score across all dimensions. The evaluated dimensions for each embodied scenario are listed in Tab. 2.
include commands e.g., "pull the handle to open the drawer."
3.2.2. EMBODIED TASK EVALUATION
Evaluation Pipeline. As illustrated in Fig. 3, we first lever- age existing or custom-traines video-to-actio models as intermediaries between the World Simulator and the agent performing closed-loop tasks, for the selected benchmarks across three simulation environments. This approach en- ables the transformation of the predicted future videos from the World Simulator into executable control signals in real- time, thereby indirectly evaluating the World Simulator's capability through the successful completion of embodied tasks. The evaluation process is tailored to the specific na- ture of the models under consideration, establishing distinct protocols for closed-loop task evaluation. We fine-tune the models on simulation datasets tailored to each task. These datasets, derived from the three aforementioned benchmarks, include task instructions and corresponding videos, ensur- ing the models are well-adapted to the specific embodied scenarios. Finally, the evaluated World Simulator is inte- grated with the video-to-action model to jointly form an embodied agent that performs the given tasks. The agent's performance across various tasks serves direct measure of the World Simulator's effectiveness.
3.2. Implicit Manipulative Evaluation
The Implicit Manipulative Evaluation assesses the capabili- ties of World Simulators across various embodied scenarios by treating the simulator low-level decision maker for situational contexts. Using pre-trained video-to-action mod- els, we implicitly evaluate the performance of the World Simulators by observing their effectiveness in closed-loop embodied task tests.
3.2.1. SIMULATION CONSTRUCTION
The Implicit Manipulative Evaluation is conducted using the following three simulation platforms, for specific settings, please refer to the Supplementary Material.
OE We employ MineRL as the Minecraft simulator, with the observation space limited to RGB images and the action space confined to keyboard and mouse controls. We adopt the Steve-1 benchmarks (Lifshitz et al., 2024), with task descriptions e.g.. "chop tree."
AD We conduct standard closed-loop evaluations using the CARLA (Dosovitskiy et al., 2017) simulator on the Lan- gAuto Benchmark (Shao et al., 2024). Task descriptions include instructions like "do not deviate from this route." "
Evaluation Metrics. In OE, we track the MineRL (Guss et al., 2019) environment state to calculate metrics e.g., travel distance and early-game item collection. Travel dis- tance is the agent's 's maximum horizontal displacement (X-Z plane) from the spawn point, while dig depth is its maximum vertical displacement (Y axis). We record the maximum number of logs, seeds, and dirt items in the agent's inventory during the episode. In AD, employ eight widely used
RM We employ CALVIN (Mees al., 2022) as the robot manipulation simulator, using only RGB images for the observation space and limiting the action space to the 7-DOF (degrees of freedom) of the robot arm. Task descriptions
6

---

WorldSimBench: Towards Video Generation Models World Simulators
Table The overall performance comparison between Human Preference Evaluator and GPT-40. HPE indicates Human Preference Evaluator. HPE@Lavie means that HPE is trained on videos except those generated by Lavie. The validation is conducted on videos generated by Laive under zero-shot setting.
Embodied Scenario
GPT-4o
HPE
GPT-4o @0eenSra
HPE @ OpenSora
GPT-4o Lavie
HPE@ Lavie
OE@ Ace(T)
72.8
89.4
66.5
71.6
87.9 -0.04
78.5
@ PLCC(t)
0.28
0.60
0.03
0.34
0.49
RM@ (MC((L
0.07
0.43
-0.06
0.47
0.17
0.44
evaluation metrics in Carla (Dosovitskiy et al., 2017), includ- ing Route Completion (RC), Infraction Score (IS), Driving Score (DS), Vehicle Collisions (VC), Pedestrian Collisions (PC), Layout Collisions (LC), Red Light Violations (RV), and Offroad Infractions (OI). In RM, we evaluate the video generation model in the CALVIN (Mees et al., 2022) setting (train on A, B, C - test on D) by running 20 trials and calculating the average success rate.
2024). For the evaluated models, we used the following datasets for fine-tuning: (1) VPT (Baker et al., 2022) and our own collected videos along with corresponding task descriptions as the training set for the OE; (2) the full Calvin(ABC D) (Mees et al., 2022) video dataset and cor- responding robot arm control instructions as the training set for RM; and (3) the full Carla (Dosovitskiy et al., 2017) video dataset and corresponding autonomous driving nav- igation commands as the training set for AD. Since the video-to-action model in our OE setup utilizes a goal-based policy, which interprets the goal from the input video and generates actions based the current observations and the goal, it allows us to additionally evaluate text-to-video models.
4. Experiments
4.1. Experimental Setup
We evaluate 8 popular video generation models, including Open- Laan-Sora-PlaanTTV (Lab & etc., 2024), Lavie (Wang et al., 2023c), ModelScope (Wang et al., 2023b), Open- Sora (Zheng et al., 2024), AnimateDiff (Guo et al., 2023), Spen-Sora-Plan(TI2y) (Lab & etc., 2024), Dynami- crafter (Xing et al., 2023), Easmanimate (Xu et al., 2024) through both Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, across three distinct scenarios: Open-Ended Embodied Environment (OE), Autonomous Driving (AD), and Robot Manipulation (RM), All models finetuned on specific datasets corresponding to three em- bodied scenarios in Explicit Perceptual Evaluation and Im- plicit Manipulative Evaluation. Detailed information on the datasets, training, and testing configurations can be found in the Supplementary Material.
4.2. Experiments on Human Preference Evaluator
We demonstrate the strong capabilities and generalization of Human Preference Evaluator by comparing it with GPT-
4o (OpenAI, 2024), showcasing its applicability for Explicit Perceptual Evaluation, as shown in Tab. 3. We use accuracy (Acc) in OE to assess the alignment of the model with hu- man preferences, given the scoring range of 1-2. In contrast, we employ Pearson linear correlation coefficient (PLCC) for AD and RM their scores range from 1-5.
After fine-tuning on HF-Embodied Dataset, our evaluator consistently surpasses the performance of GPT-4o in terms of alignment with human preferences across all scenarios. Additionally, we conducted zero-shot experiments with two challenging models, i.e. OpenSora and Lavie. GPT-4o exhibits a negative correlation with human preferences in evaluating OpenSora in AD under zero-shot setting, as well as evaluating Lavie in RM under zero-shot setting. Our eval- uator's ' zero-shot performance shows a high correlation with human preferences, further demonstrating its robust general- ization capabilities. Human Preference Evaluator is suitable for Explicit Perceptual Evaluation, and the HF-Embodied Dataset can be leveraged train even more aligned models for assessing video generation models towards World Sim- ulators. For additional details and validation experiments on the Em-onddd Dataset, please refer Sup. C.1 and Sup. C.3.
For Explicit Perceptual Evaluation, we extract five instruc- tions from the Task Instruction Prompt List for each di- mension across the three embodied scenarios, ensuring they strongly align with the specific evaluation criteria, as dis- cussed in Sec. 3.1.5. The selected instruction prompts each model to generate five videos, which are then scored by the Human Preference Evaluator obtain an average score for the model's performance. For the scoring range 1-n, n is set 2 for OE, and set 5 for both AD and RM. We indicate that the generation quality in OE is perceived as binary from a human perspective, while the other two scenarios exhibit a more diverse range of video quality.
For Implicit Manipulative Evaluation, we constructed three video-to-action models for embodied simulation environ- ments, following the designs of Steve-1 (Lifshitz et al., 2024), Susie (Black et al., 2023), and LMdrive (Shao et al.,
7

---

WorldSimBench: Towards Video Generation Models World Simulators
IA
IA
IA
1.0
1.0
1.0
FC
0.8
EI
0.8
BC
0.8
PV
0.6
KE
0.6
PV
0.6
Open-Sora-Plar
0
0.4
0.4
Lavie
0:2
0.2
6
Model55coce
OpenSora
AnimateDiff
BO
SA
FC
AE
DynamicCrafter
SP
AE
EasyAnimate
VC
T
E1
TJ
OE
AD
RM
Figure Result Explicit Perceptual Evaluation aross three embodied scenarios. Scores each embodied scenario are normalized to 0-1. The abbreviations are listed in Tab. 2.
4.3. Design Features and Discussions
bilities. Close-loop Interactive Evaluation addresses this by enabling continuous feedback and adaptation, ensuring that the model's predictions and actions evolve in response to the changing environment, thus providing more accurate and realistic assessment of its performance. Fig. presents the Implicit Manipulative Evaluationnvvaluatin results, showing significant variation in the performance of video generation models across different tasks. In the OE, video generation models conditioned on the first frame have a significantly lower success rate compared to those without image condi- tioning. This suggests that models with image conditioning struggle to generate physical laws and 3D scene representa- tions accurately. Tasks like travel, requiring high-quality tra- jectories and 3D representation, show the greatest variation in model performance, while simpler tasks like collecting wood see similar performance across models, indicating effective handling of minimal background variation. In the AD, models with better trajectory(Open) OOaer--OO------llla genera- tion perform better. In the RM, where background variation is minimal, models perform similarly on simple tasks, but as complexity increases, more robust models achieve higher success rates. Despite some success across scenarios, video generation models still need significant improvements in generating physically consistent content to be reliable for training agents or guiding actions.
In this section, we discuss the Design features and corre- sponding observations we draw from our comprehensive evaluation experiments. More details can found in the Supplementary Material.
Human Prefrence with Feedback. Given the complex- ity and diversity in the representation of physical rules in videos, even a specific dimension may manifest in various ways (for example, both illogical and discontinuous object motion fall under trajectory-related issues). This makes challenging to evaluate using score-based models or a single fixed set of evaluation criteria. WorldSimBench ad- dresses this challenge effectively by employing human preference scoring mechanism and fine-grained feedback system. Fig. 4 illustrates the evaluation results of Explicit Perceptual Evaluation, more detail analyze could be found in Sup. D. In OE, most models struggle with Embodied Interaction, particularly in generating plausible object de- formations, e.g., block shattering, due to the complexity of physical rules. In AD, the variation between models is minimal, with nigh-performing models excelling across all dimensions. The simpler instructions, like moving forward or turming, lead to high Instruction Aligmment, but many gen- erated videos suffer from poor 3D depth (Perspectivity) and fail to depict realistic embodied elements like pedestrians and vehicles, affecting the overall Aesthetic. RMM, models perform uniformly well in static scene depiction, excelling Consistency.
Alignment of Physical Rules and Actions. Ensuring that World Simulators adhere physical laws while generat- ing predictions is crucial for practical application. The alignment of physical rules and actions is essential as it guarantees that the model's outputs are not only visually plausible but also executable in real-world scenarios. This approach allows for the seamless integration of predicted actions with their physical environment, ensuring reliability and effectiveness in real- worldw tasks. Based on exper- imental findings, we observe that most conclusions from the Explicit Perceptual Evaluationand Implicit Manipulative Evaluationevaluations are consistent. Specifically, the visual quality across most dimensions aligns with the results from
in Perspectivity and Foreground/Backgrnnn However, they struggle with Instruction Alignment, often generating aimless actions. Despite this, the lack of un- reasonable trajectories results in relatively high Trajectory scores, though robotic manipulation remains a significant challenge for current models.
Close-loop Interactive Evaluation. Given the dynamic nature and real-time requirements of interactive environ- ments, evaluating World Simulators through static bench- marks often fails to capture the full spectrum of their capa-
8

---

WorldSimBench: Towards Video Generation Models World Simulators
Wood
Seed Travel
Score
Dig
(a)
Lavie
ModelScope
OpenSora
AnimataDiff Open-Sora-Plan (aa.yaannttaatyyyannnnoor
EasyAnimate
RC
80
30
Score
IS
Score
60
F
C 4
20
C_5
40
20
(b) Open-Sora Plan(Ti2v)
DynamicCrafter
EasyAnimate
(c)
Open-Sore Spen--ore-Plen(tiiyy
DynamicCrafter
EasyAnimatee
Figure 5. Result of Implicit Manipulative Evaluation aross three embodied scenarios. The abbreviations are listed in Sec. 3.2.2.
the closed-loop experiments. e.g., Dynamicrafter, which performs well in trajectory generation in Explicit Percep- tual Evaluation, also excels in trajectory-food focused scenarios like AD and RM. However, in other cases susss--uuchae the OE, which requires more frequent interactions, and long- sequence tasks (4, 5) in MMM-Dynamicrafterr underperforms compared to Sora-Plan. This differs from the Explicit Perceptual Evaluation results, likely because these tasks demand stable, high-quality video generation for guidance, where Open-Sora-Plan shows higher robustness. Therefore, comprehensive evaluation of video generation models re- quires a combination of Explicit Perceptual Evaluation and Implicit Manipulative Evaluation assessments to provide the most fair and accurate judgment. Finally, based on the overall Explicit Perceptual Evaluationand Implicit Manip- ulative Evaluationresults, we conclude that current video generation models still fail to effectively capture many phys- ical rules, indicating significant improvements are needed before they can function as true World Simulators.
content from the perspective of embodied intelligence, the World Simulator can be applied more scenarios than just robots, and different scenarios have more physical represen- tations, how to effectively evaluate the World Simulator in other scenarios requires more exploration.
Impact Statement
Our work establishes structured evaluation framework for predictive models, addressing key limitations in their cate- gorization and assessment. By introducing WorldSimBench. we provide a dual evaluation strategy that bridges the gap between visual perception and embodied decision-making enabling a more comprehensive understanding of World Simulators. The introduction of HI-Embodied Dataset enhances model evaluation by aligning assessments with human perception, while our downstream evaluation mea- sures the functional effectiveness of generated videos in real-world-inspired control tasks. These contributions lay the foundation for future advancements in video genera- tion and predictive modeling, driving progress toward more reliable and capable embodied AI systems.
5. Conclusion
In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing dual evaluation framework called WorldSimBench. We conducted comprehensive evaluation and analysis of multiple video generation models as World Simulators through both Explicit Perceptual Eval- uation and Implicit Manipulative Evaluation processes. We summarize key findings from the evaluation and hope these insights will inspire and guide future research on World Simulators.
Acknowledgement
This work is partially supported by Shenzhen Science and Technology Program JCYJ202208 1 8103001002 and by the Guangdong Key Laboratory of Big Data Analysis and Processing, Sun Yat-sen University, China, and by the High- performance Computing Public Platform (Shenzhen Cam- pus) of Sun Yat-sen University.
Limitations. Although evaluate physical rules and 3D

---

WorldSimBench: Towards Video Generation Models as World Simulators
References
Du, Y., Yang. M., Florence, P., Xia, E, Wahid, A., Ichter, B., Sermanet, P., Yu, T., Abbeel, P., Tenenbaum, J. B., et al. Video language planning. arXiv preprint arXiv:2310. 2023.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
Du, Y., Yang, S., Dai, B., Dai, H., Nachum, O., Tenenbaum, J., Schuurmans, D., and Abbeel, P. Learning universal policies via text-guided video generation. Advances in Neural Information Processing Systems, 36, 2024.
Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1 2021.
Ebert, F., Yang, Y., Schmeckpeper, K., Bucher, B., Geor- gakis, G., Danilldis, K., Finn, C., and Levine, S. Bridge data: Boosting generalization of robotic skills with cross- domain datasets. arXiv preprint arXiv: 10%.::2009 2021.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2465 24654, 2022.
Fang, H.-S., Fang, H., Tang, Z., Liu, J., Wang. J., Zhu, H., and Lu, C. Rh20t: A robotic dataset for learning diverse skills in one-shot. In RSS 2023 Workshop on Learning for Task and Motion Planning, 2023.
Black, K., Nakamoto, M., Atreya, P., Walke, H., Finn, C., Kumar, A., and Levine, S. Zero-shot robotic manipulation with pretrained image-editing diffusion models. arXiv preprint ar Ar.iv:2310. 10639, 2023.
Fu, T.J.,, Hu, W., Du, X., Wang, W. Y., Yang, Y., and Gan, Z. Guiding instruction-basse image editing via multimodal large language models. arXiv preprint arXiv:2309. 17.009971002 1012 2023.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang. J., and Li, H. Vista: A generalizable driving world model with high fidelity and versatile controllabil- ity. arXiv preprint arXiv:2405. 1735, 2024.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q.. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631. 2020.
17398,
Goyal, R., Ebrahimi Kahou, S., Michalski, V., Materzynska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Freitar-Freetag, M., et al. The" something something" video database for learning and evaluating visual com- mon sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850 2017.
Chen, Y., Ge, Y., Ge, Y., Ding, M., Li, B., Wang, R., Xu, R., Shan, Y., and Liu, X. Agoplan-benchs Benchmarking egocentric embodied planning with multimodal large lan- guage models. arXiv preprint arXiv:2312.06722, 2023.
Chen, Z., Shi, Z., Lu, X., He, L., Qian, S., Fang, H. S., Yin, Z., Ouyang, W., Shao, J., Qiao, Y., et al. Rh20t- p: A primitive-level robotic dataset towards composable generalization agents. arXiv preprint arXiv:2403. 1962,, 2024.
Grauman, K., Westbury, A., Byrne, E., Chavis, Z., Furnari, A., Girdhar, R., Hamburger, J., Jiang, H., Liu, M., Liu, X., et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 18995-19012, 2022.
Chiang, W.-I.,, Li, 7.,, Lin, 7., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang. S., Zhuang, Y., Gonzalez, J. E., et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See httpx://vicuna. Imsys. org (accessed 14 April 2023), 2023.
Guo, Y., Yang. C., Rao, A., Liang, Z., Wang, Y., Qiao, Y., Agrawala, M., Lin, D.. and Dai, B. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint woXiv:2307.04725 2023.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., and Salakhutdinov, R. Minerl: A large- scale dataset of minecraft demonstrations, arXiv preprint arXiv: 090.1.9974.440 2019.
Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et Palm-e: An embodied multimodal language model. arXiv preprint 233.03377, 2023.
Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv: / 8003. 10122, 2018
10

---

WorldSimBench: Towards Video Generation Models World Simulators
He, X., Feng, W., Zheng, K., Lu, Y., Zhu, W., Li, J., Fan, Y., Wang. J., Li, L.,, Yang. Z.., et al. Mmworld: Towards multi-discipline multi-facete world model evaluation in videos. arXiv preprint arXiv:2406. 2444007700008007,,, 2024.
Lu, C., Qian, C., Zheng, G., Fan, H., Gao, H., Zhang, J., Shao, J., Deng, J., Fu, J., Huang. K., Li, K., Li, L.., Wang, L., Sheng, L., Chen, M., Zhang, M., Ren, Q. Chen, S., Gui. T., Ouyang. W., Wang, Y. Teng, Y., Wang. Y., Wang, Y., He, Y. Wang, Y., Wang, Y., Zhang, Y., Qiao, Y., Shen, Y., Mou, Y., Chen, Y., Zhang, Z., Shi, Z., Yin, Z. and Wang. Z. From gpt-4 to gemini and beyond: Assessing the landscape of milms on generalizability, trustworthiness and causality through four modalities, 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models, 2021.
Huang, Z., He, Y., Yu, J., Zhang, F,, Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q., Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 21807-21818, 2024.
Mees, O., Hermann, I.., Rosete- Beae, E., and Burgard, W. Calvin: benchmark for language- conditioed policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327-7334, 2022.
Janner, M., Du, Y., Tenenbaum, B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. arXiv preprint ar arXiv:2205.0999) 2022.
OpenAL. Gpt-4o. https: hat::://ppeaa.... http:////peeaa..cmmneeeee.eee hello-gpt-40/, 2024.
Lab, P.-YY and etc., T. A. Open-sora-plan, April 2024. URL https: /toi.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 4195 4205, 2023.
10948109.
Lai, B., Dai, X., Chen, I.., Pang, G., Rehg, J. M., and Liu, M. Lego: Learning egocentric action frame generation via vi- sual instruction tuning. arXiv preprint arXiv:2312.03849, 2023.
Qin, Y., Zhou, E., Liu, Q., Yin, Z., Sheng, L., Zhang, R., Qiao, Y., and Shao, J. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16307-16316. IEEE, 2024.
Li, x., Zhang, M., Geng, Y., Geng, H., Long, Y., Shen, Y., Zhang, R., Liu, J., and Dong, H. Manipllm: Embod- ied multimodal large language model for object-centric robotic manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18061-18070, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 188)99, 2019.
Lifshitz. S.. Paster, K., Chan, H., Ba. J., and Mcllraith, S. Steve-1: A generative model for text-to-behavior in minecraft. Advances in Neural Information Processing Systems, 36, 2024.
Shao, H., Hu, Y.. Wang, L., Song, G., Waslander, S. L., Liu, Y., and Li, H. Lmdrive: Closed-loop end-to-end driving with large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15 15120-15130, 2024.
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tuning. In NeurlPS, 2023a.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang, K., et al. Agentbench: Evalu- ating lims as agents. arXiv preprint arXiv:2308.0.R. 03668, 2023b.
Shi, Z., Wang, Z., Fan, H., Zhang, Z., Li, L., Zhang, Y., Yin, Z., Sheng, L.,, Qiao, Y., and Shao, J. Assessment of multimodal large language models in alignment with human values, 2024.
Liu, X., Zhang, T., Gu, Y., Iong, I.., Xu, Y., Song, X., Zhang, S., Lai, H., Liu, X., Zhao, H., et al. Visualagent- bench: Towards large multimodal models as visual foun- dation agents. arXiv preprint arXiv:2408.066. 06327, 2024a.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 2023.
Liu, Y., Cun, X., Liu, X., Wang, X., Zhang. Y., Chen, H., Liu, Y., Zeng, T., Chan, R., and Shan, Y. Evalcrafter: Benchmarking and evaluating large video generation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22139- 22149, 2024b.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302 /3971, 2023.

---

WorldSimBench: Towards Video Generation Models World Simulators
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L.,, and Anandkumar, A. Voyager: An open- ended embodied agent with large language models. arXiv preprint (.x.xx:2305.62990 2023a.
Wang, J. Yuan, H., Chen, D., Zhang, Y. Wang, X., and Zhang, S. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023b.
Wang, Y., Chen, X., Ma, X., Zhou, S., Huang, Z., Wang, Y., Yang, C., He, Y., Yu, Yang, al. Lavie: High- quality video generation with cascaded latent diffusion models. arXiv preprint arXiv:2309.15/10 2023c.
Wang, Z., Cai, S., Chen, G., Liu, A., Ma. X., and Liang, Describe, explain, plan and select: Interactive planning with large language models enables open-wort world multi- task agents. arXiv preprint arXiv:2302.01500 2023d.
Xing, J., Xia, M., Zhang, Y., Chen, H., Wang, X., Wong, T.-T., and Shan, Y. Dynamicrafter: Animating open- domain images with video difftusion priors. arXiv preprint 1rXiv:2310.121900 2023.
Xu, J., Zou, X., Huang, K., Chen, Y., Liu, B., Cheng, M., Shi, X., and Huang, Easyanimate: perforeefo long video generation method based on transformer ar- chitecture. arXiv preprint arXiv:2405. 18991, 2024.
Yang. M., Du, Y., Ghasemipour, K., Tompson, J., Schuur- mans, D., and Abbeel, P. Learning interactive real-world simulators. arXiv preprint arXiv:2310.06114 14, 2023.
Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Huang, X., Wang, Z., Sheng. L., Bai, L., Shao, J., and Ouyang, W. LAMM: language-assisted multi-modal instruction- tuning dataset, framework, and benchmark. In NeurIPS, 2023.
Zhang, H., Wang. Y., Tang, Y., Liu, Y., Feng. J., Dai, J., and Jin, X. Flash-vstream: Memory-base real-time understanding for long video streams, 2024a.
Zhang, Z., Tang, S., Zhang, Y., Fu, I. Wang, Y., Liu, Y., Wang, D., Shao, J., Wang, L., and Lu, Ad-h: Au- tonomous driving with hierarchical agents. arXiv preprint arXiv:2406.0344 2024b.
Zheng, Z., Peng, X., Yang, T., Shen, C.. Li, S., Liu, H., Zhou, Y., Li, T., and You, Y. Open-sora: Democratizing efficient video production for all, March 2024. URL ht htsp: // //github. com/ http://ccateecc/peeeeee Sooa.
Zhou, E., Qin, Y., Yin, Z., Huang, Y., Zhang, R., Sheng, L., Qiao, Y., and Shao, J. Minedreamer: Learning to follow instructions via chain-of-imagination for simulated-world control. arXiv preprint arXiv:2403.12033 2024.
12

---

WorldSimBench:
Towards Video Generation Models World Simulators
A. Related Work
Predictive Models. Predictive models are capable of generating process representations that map the current state to future states by incorporating current state representations and control over future trends. Predictive Text Model, built on LLMs (Radford al., 2019; Touvron et al., 2023; Chiang et al., 2023) and MLLMs (Achiam et al., 2023; Team et al., 2023; Liu et al., 2023a; Yin et al., 2023), generate future predictions in the text modality by accepting current state representations and text instructions. These models have demonstrated impressive performance in high-level planning tasks for embodied agents (Driess et al., 2023; Li et al., 2024; Qin ct al., 2024; Chen et al., 2024; Zhang et al., 2024b; Lu et al., 2024). Similarly, image generation models (Brooks et al., 2023; Fu et al., 2023) as Predictive Image Model (Lai et al., 2023; Black et al., 2023; Zhou et al., 2024) can produce future goal images, showcasing strong capabilities during the recision-making phase of embodied agents. Predictive Video Model (Du et al., 2024; 2023), based on video generation models (Janner et al., 2022), have made some progress in embodied control. However, due limitations in data or models, the generated videos often lack essential physical representations and logical consistency, restricting their applicability to fixed scenarios and single tasks.
With the advancement of diffiuson transformer (Peebles & Xie, 2023) and the extensive utilization of large-scale internet video datasets (Bain et al., 2021; Ebert et al., 2021; Goyal et al., 2017; Grauman et al., 2022), certain Predictive Actionable Video Model (Yang et al., 2023) models, also known as World Simulators, have achieved more precise representations of physical laws and 3D environments.
Evaluation of Predictive Models. With the advancement of predictive models, research has also expanded to evaluate the capabilities of models at different stages. (Liu et al., 2023b; Chen et al., 2023; Shi et al., 2024; Liu et al., 2024a) conducted text-level and task completion evaluations for Predictive Text Model at the So stage. (Lai et al., 2023) performed score-based evaluations from an aesthetic perspective for Predictive Image Model at the S1 stage. (Huang et al.. 2024; Liu et al., 2024b) also assessed the aesthetic quality of videos generated by Predictive Video Model at the S2 stage. We take the first step in evaluating World Simulators through an embodied perspective.
B. Taxonomy in Explicit Perceptual Evaluation
We outline the evaluation dimensions for each embodied scenario below, along with their corresponding explanations. These explanations are used for detailed human annotation documentation and also serve as the explanation of the evaluation criteria in instructions for the Human Preference Evaluator.
B.1. Open-Ended Embodied Environment
Visual Quality. Background Consistency ensures the background remains consistent throughout the video. Foreground Consistency verifies the consistency of the foreground elements.
Condition Consistency. Instruction Alignment assesses whether the video aligns with the provided input instruction. Scenario Alignment checks if the input instruction defines an embodied scenario and whether the video accurately reflects this scenario.
Embodiment. Velocity evaluates if the velocity of the observed object is appropriate. Embodied Interaction evaluates the embodied interaction's 's appropriateness based on the interaction process and target. Trajectory evaluates whether the motion trajectory in the video is logical.
B.2, Autonomous Driving
Visual Quality. Aesthetics evaluates whether the composition, color, lighting, and scene in the video align with human aesthetics.
Condition Consistency. Instruction Alignment assesses whether the video aligns with the provided input instruction.
Embodiment. Perspectivity evaluates the video's perspective, specifically assessing the 3D scene relationships. This includes evaluating whether the video has strong sense of depth and realism (i.e., whether it feels three-dimensional) Additionally, assess the logic of lighting and shadows, including whether the shadow positions are consistent with the light sources. Trajectory evaluates whether the movement and the trajectory of elements in the video is logical. Key Element assesses the generated quality of embodied elements e.g., roads, vehicles, pedestrians, bicycles, lane markings, sidewalks,
13

---

WorldSimBench: Towards Video Generation Models World Simulators
Table 4. Analysis HF-Emmodied Dataset. Samples scored higher than in AD and RM are considered positve.
Embodied Scenario
#instructions
#videos
#dims
#actions
#positive
#negative
Open-Ended Embodied Environment
270
8401
7
11
121249
79965
Autonomous Driving
5
15870
6
5
56768
35044
Robot Manipulation
2556
11430
7
26
70672
9338
traffic signs, and traffic lights. Safety evaluates whether the behavior of the vehicles comply with traffic rules. Are there any instances of running red lights, speeding, or driving outside of permissible areas.
B.3. Robot Manipulation
Visual Quality. Aesthetics evaluates whether the composition, color, lighting, and scene in the video align with human aesthetics. Background Consistency ensures the background remains consistent throughout the video, include the manipula- tion table and the environment. Foreground Consistency verifies the consistency of the foreground elements, including the robotic arm and the object on the manipulation table.
Condition Consistency. Instruction Alignment assesses whether the action of the robot arm the generated video aligns with the provided input instruction.
Embodiment. Perspectivity evaluates the video's perspective, specifically assessing the 3D scene relationships. This includes evaluating whether the video has a strong sense of depth and realism (i.e.., whether it feels ptree-dimessiona)) Additionally, assess the logic of lighting and shadows, including whether the shadow positions are consistent with the light sources. Embodied Interaction judges whether the object's s shape and posture conform to the rules during the collision of objects and the interaction between the robotic arm and the object. Trajectory evaluates whether the trajectory of the robotic arm is reasonable and line with human cognition.
C. Detaild Implementation of Explicit Perceptual Evaluation
C.1. HF-E Un-Emmodded Dataset
Tab. 4 provides an analysis of the HF-Embodied Dataset. In Autonomous Driving scenario, there are only five instructions: move forward, move backward, turn left, turn right, and stop. The other two scenarios include a variety of instructions that combine actions with target objects. Given the diverse instructions, different video generation models generate multiple videos after finetuning on specific datasets. To enhance the Human Preference Evaluator understanding of the autonomous driving context, we also supplement the AD scenario with videos from real-world scenes. Additionally, we list the quantities of positive and negative samples across all dimensions. Samples with human annotated scores of higher in AD and RM are considered positive. Leveraging HF-Embodied Dataset with comprehensive embodied dimensions, we train the Human Preference Evaluator to enable efficient assessment in Explicit Perceptual Evaluation.
To validate the potential value of the dataset, we leverage it to align video generation models with human preferences, thereby enhancing various embodied evaluation attributes. Specifically, we use Open-SORA-PAA as the base model and construct preference data pairs from the dataset's annotated robotic manipulation data for training. Following the DPO training paradigm, which directly utilizes winning and losing preference pairs, the results presented in Tab. 5 demonstrate that the re- reightet model achieves improvements across multiple attributes.
Table 5. HF-Embodied Dataset alignment performance The abbreviations are listed in Tab. 2.
Model
AE
BC
FC
IA
Overall
Open-Sora-Plan Open-Sorz-Plan(PPO)
4.0
4.0
4.0
1.0
4.9
5.0
4.0
3.84
4.0
3.9
4.0
1.6
5.0
5.0
4.3
3.97
Discussion of Future Work. Human Preference Evaluator (HPE) and HF-Embodied Dataset have been effective in aligning generated content with human preferences and evaluating video generation models, and we could explore more about its potential applications. Here are some future work directions to leverage the capabilities of HPE and Hm-oodddded Dataset:
14

---

WorldSimBench: Towards Video Generation Models World Simulators
Interactive Training for Generative Models Utilize HPE as a real-time feedback mechanism during the training of generative models. By integrating HPE and HF--mmoodeed Dataset into a reinforcement learning framework, it could dynamically guide the model to improve alignment with human preferences across various scenarios, and can even make the world simulator perform better in downstream tasks.
C.2. Video Generation Model Finetuning
Table Training Frames of Generation Models.
Model
Open- Open-Sora-Plaan
Lavie
ModelScope
OpenSora
AnimateDiff
DynamicCrafter
EasyAnimate
Short Videos(frames)
16
16
16
16
16
16
Long Videos(trames) 64
48
60
48
64
60
64
We evaluate 8 popular video generation model, including Open-Sora-Plan(T2V)) (Lab & etc., 2024), Lavie (Wang et al., 2023c), ModelScope (Wang et al., 2023b). OpenSora (Zheng al., 2024), AnimateDiff (Guo et al., 2023). Open-Sora Plan(TI2V) (Lab & etc., 2024), DynamicCrafter (Xing et al., 2023), Easy AnimAee (Xu et al., 2024) through both Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, across three distinct scenarios: Open-Ended Embodied Environment (OE), Autonomous Driving (AD), and Robot Manipulation (RM).
In Open-Ended Embodied Environment, we use OpenAl Contractor Gameplay Dataset (Baker et al., 2022) which is created by hiring human contractors to play Minecraft and complete tasks like house building. Keypresses and mouse movements are recorded during gameplay. We apply the same preprocessing steps as VPT, including filtering out null actions. Additionally, we create a supplementary dataset for the task "Explore" by generating trajectories using various pre-trained Steve- I agents. The distribution of this dataset is enhanced by randomly switching between models during trajectories, resetting the agent's memory, and adjusting the agent's is orientation to face new directions at random intervals. For specific in-game events, e.g. "mi ne bl ock", the type of block broken is logged alongside precise timestamps. These timestamps allow for accurate progress tracking and are aligned with the completion of event-related instructions.
"mine
In Autonomous Driving, we fine-tune using the nuScenes training set (Caesar et al., 2020), and following the approach in Vista (Gao al., 2024), we sample video clips consisting of 25 frames frequency of 10 Hz. To classify actions into textual commands, we adhere to established conventions in planning and define ego-vehicle commands as "turn right", "turn left", "go straight", and "stop", consistent with the definitions in Vista.
In Robot Manipulation, we use RH20T-P (Chen et al., 2024), a dataset based on RH20T (Fang et al., 2023) and designed for primitive-level robotic manipulation that features meticulously defined primitive skills and diverse primitive-level spatial knowledge of multiple forms. We use each primitive-level robotic manipulation instruction along with the corresponding video as input for training. Additionally, since this dataset is designed for downstream tasks in specific scenarios, some textual instructions include explicit coordinate information. To enhance the generalization ability of the video model, we excluded these coordinate-specifie instructions during training.
At the model architecture level, we followed Dynamicrafter (Xing et al., 2023) to modify the text-to-video model of Bien-Sora-Plan(T2) (Lab & etc., 2024) by replacing the first frame and expanding the channel dimensions, enabling the model to take the first frame as a condition. This resulted in the Open-Sora-Plae (TI2V) model. No structural adjustments were made other models. During training, we preprocessed the data according each model's default input format and performed fine-tuning following the official implementation without changing the training settings. We fine-tuned each model using two different video lengths to enhance the diversity of the video evaluation set: short videos with approximately 20 frames and long videos with around 60 frames, depending on the model's 's default training video length. The specific lengths are detailed in the Tab. 6.
C.3. Human Preference Evaluator Traing
The Human Preference Evaluator is trained based on 2021) parameters are trained. The model's input consists of sampled vidco, represented as multiple frames, along with a prompt. The prompt includes the current scenario, the instruction input for video generation, the dimension being evaluated, and the definition of that dimension. An example of such a prompt is illustrated in Fig. 6, while the details of the explanation are discussed in Section 2. We don't 't use the annotated reason during training for CoT of the evaluator, as the reason labeled
(Zhang et al., 2024a), where only LoRA (Hu et al.,
15

---

WorldSimBench: Towards Video Generation Models World Simulators
Oc@)Ac((t
BC
FC
EI
Overall
GPT-40
70.4
70.9
67.3
79.6
83.7
85.9
72.8
НPE
81.2
87.5
87.5
96.4
94.5
93.8
88.8
89.4
GPT-4o@ OrenSOra
100
88.8
66.5
HPE@ OpenSora
70
90
60
100
100
22.2
80
71.6
GPT-4o@L Lavie
66.7
88.8
87.5
100
87.5
78.5
HPE@ Lavie
80
80
80
100
100
75
100
87.9
AD@PLCC(1)
AE
IA
PV
TJ
KE
SF
Overall
GPT-4o
0.37
0.22
0.23
0.28
0.37
0.18
0.28
HPE
0.71
0.57
0.50
0.58
0.65
0.58
0.60
GPT-4o@OpenSoe
0.22
-0.39
0.32
0.15
-0.03
-0.12
0.03
HPenOpenSSraa
0.37
0.55
0.34
0.06
0.28
0.41
0.34
GPT-4o@ Lavie
0.17
0.13
-0.34
0.06
-0,09
-0.15
-0.04
HPE@Lavis
0.28
0.49
0.37
0.12
0.69
0.49
RM RM@PLCC(1)
AE
BC
FC
IA
PV
TJ
EI
Overall
GPT-4o
0.07
0.18
0.20
0.32
-0.14
-0.01
-0.14
0.07
HPE
0.52
0.43
0.43
0.43
0.20
0.56
0.44
0.43
GPT-4o@OpenSora
-0.45
-0.03
0.08
0.0
0.04
-0.23
0.14
-0.06
HPE@ OpenOOaenSra
0.25
0.35
0.05
0.42
0.89
0.89
0.44
0.47
GPT-4o@l Lavie
0.11
-0.07
0.42
0.42
0.21
0.31
-0.21
0.17
HPE@Lan Lavie
0.33
0.04
0.69
0.40
0.89
0.67
0.06
0.44
Table Performance comparison between Human Preference Evaluatorand GPTT-4o. HPE indicates Human Preference Evaluator. The other abbreviations are listed in Tab. 2.
<Video>inThe given autonomous driving video is generated by generative model based the input instruction: jinstruction]. Please rate the video based on the following criteria: [Dimension]: [Dimension Explanation]
Figure 6. Prompt template for Autonomous Driving, The (item} } replaced with specific content.
by different human varies a lot, hard for model learn.
We maintain consistent training settings in all three scenarios, with video sampling frequency of 4. The LoRA settings aligned with those in Flash-VStream We use AdamW the optimizer, employ cosine decay for the learning rate scheduler. We train for 4 epochs with a learning rate of 2e-5 and a warmup ratio of 0.03. The training is conducted on 4 A100 80 GPUs. To avoid over-fitting to specific prompts or videos generated by particular models, we carefully filter the HF-Embodied Dataset to ensure balanced distribution across various generation models and evaluation dimensions.
We prove the effectiveness and generalizability of through comparison with GPT-4o arcoss the three embodied scenarios, under both finetuned and zero-shot setting, shown in Tab. 7. After fine-tuning, the Human Preference Evaluator surpasses GPT4-0 in aligning with human preferences across all dimensions in every scenario. This is particularly evident in challenging dimensions, e.g., Embodied Interaction and Trajectory in RM, where GPT4-o shows negative correlation, while the Human Preference Evaluator exhibits strong positive correlation. These results demonstrate the its robust performance, making it suitable for Explicit Perceptual Evaluation. In zero-shot settings, the Human Preference Evaluator also outperforms GPT4-o in nearly all dimensions, further proving our model's aility to understand videos generated by different models.
16

---

WorldSimBench: Towards Video Generation Models World Simulators
Table Evaluation results OE, The abbreviations are listed Tab,
Model
BC
FC
SA
VC
El
Overall
Open-Sora-Plan
1,4
1.9
1.7
1.7
2.0
1.5
1.6
1.69
Lavie
1.3
2.0
1.7
1.7
2.0
2.0
1.8
1.79
ModelScope
1.9
2.0
2.0
1.7
2.0
2.0
1.75
1.91
OpenSora
1.6
1.9
1.6
1.8
2.0
2.0
1.6
1.79
AnimateDiff
1.3
1.3
1.2
1.4
1.38
1.55
1.40
DynamicCrafter
1.9
2.0
1.5
2.0
2.0
2.0
1.45
1.84
Easy Animate
1.4
1.8
1.5
2.0
2.0
1.22
1.45
1.62
Table Evaluation results AD, abbreviations are listed in Tab.
Model
AE
PV
TJ
KE
SF
Overall
Open-Sora-Pie
1.6
5.0
1.55
1.4
1.45
3.2
2.37
Lavie
2.15
5.0
2.2
2.8
2.1
5.0
3.21
ModelScope
2.8
5.0
3.35
4.0
3.0
5,0
3.86
OpenSora
3,55
5.0
4.4
4.8
3.65
5.0
4.40
AnimateDiff
1.55
5.0
1.55
1.0
1.3
3.8
2.37
DynamicCrafter
2.6
4.0
3.4
3.8
2.65
5.0
3.57
Easy Animate
1.5
3.4
1.4
1.4
1.3
2.6
1.93
D. Detailed Result of Explicit Perceptual Evaluation
D.1. Quantitative Results
Tabs. 8-10 present the comprehensive evaluation results for video generation models across three scenarios, including the scores for each dimension and the mean scores representing the overall performance of the models. In OE, although scoring is binary, we display scores scale of 1-2 for consistent comparison. In addition to the conclusions mentioned in the main text, we can observe the following findings.
In OE, most models achieve high scores in Velocity, largely due to the limited occurrences of object movement in the generated videos. Generating dynamic embodied environments with moving objects presents a significant challenge for current models. Additionally, the consistency between the generated videos and the scenarios specified in the instructions higher than the alignment with the task-oriented instructions. This indicates that while the models can generate corresponding scenes, they struggle to reason about the temporal actions necessary for task completion.
In AD, the quality of the generated videos significantly declines due to the complexity of outdoor driving scenarios. The models must understand and generate various traffic elements, e.g., roads, background buildings, pedestrians, and vehicles, while also producing dynamic content, with each element requiring reasonable speed. This presents substantial challenges. However, top-performing models, e.g., OpenSora, manage achieve the highest scores across all metrics.
In RM, the primary issue lies in Instruction Alignment. The video generation models struggle comprehend the input instructions and generate appropriate actions complete the tasks, instead moving aimlessly without clear objectives. This lack of targeted movement reduces potential errors related object interaction penetration, resulting in artificially inflated scores in Embodied Interaction and Trajectory. Current video generation models struggle in effectively addressing robotic manipulation tasks.
D.2. Qualitative Results
We include a qualitative analysis of generated videos under the three embodied scenarios. Each video is represented by three evenly sampled frames, with the corresponding generation instructions listed above the video. To the left of the vidcos, we provide the scores of the key embodied attributes labeled by the human preference evaluator.
17

---

WorldSimBench: Towards Video Generation Models World Simulators
Table 10. Evaluation results in RM. The abbreviations are listed in Tab.
Model
AE
BC
IA
TJ
E
Overall
Open-Sora-Plan
4.0
4.0
4.0
1.0
4.9
5.0
4.0
3.84
Lavie
3.8
3,9
4.0
1.8
4.95
5.0
4.1
3.94
ModelScope
3.63
4.1
4.0
1.18
4.9
5.0
4.0
3.83
OpenSora
3.85
4.0
3.95
1.3
4.75
5.0
4.1
3.85
Difm
3.8
3.9
4.0
1.0
4.95
5.0
3.82
DynamicCrafter
3.97
4.08
4.0
2.6
5.0
5.0
4.31
4.14
Easy Animate
3.55
3.45
3.65
1.2
4.8
4.3
3,45
3.49
Place tarch the stone.
Place torch the stone.
El:
El:
IA: 5
IA:
Ben aaaa
Go straight.
Go straight.
KE:1
KE:4
1
SF:
PV:1
PV:5
Place the brown square.
Place the sponge.
PV:5
PV:5
TJ:
TJ:
El: 4
El:
IA: 1
IA:
Figure 7. Qualitative Results in Explicit Perceptual Evaluation.
Open-Ended Embodiment Scenario. For the open-ended embodiment scenario, the left video demonstrates successful completion of the instructed task, with proper interaction with the stone. In contrast, the right video encounters issues during interaction, specifically crushing the stone when placing the torch, indicating a problematic interaction.
Autonomous Driving Scenario. In the autonomous driving context, the left video suffers from significant distortion and light pollution. Additionally, it exhibits unsafe behavior, such maintaining excessive speed despite the presence of a car ahead. On the other hand, the right video maintains high-quality generation and demonstrates proper adherence to traffic rules, including slowing down at a red light.
Robotic Manipulation Scenario. For robotic manipulation, the left video displays that the robotic arm interacts with a rigid object (wooden block), which appropriately does not deform during the grasping process. However, minor, physically implausible rotations occur during the grasp, resulting in a score of 4 for EI. Additionally, the generated wooden block does match the specified color in the instruction, leading to instruction alignment score of 1. In contrast, the flexible object (sponge) in the right video, unrealistically stretched, violating physical rules. Furthermore, the video depicts the robotic arm moving away from the table, which contradicts the "place" instruction. This mismatch leads to low scores in both trajectory and instruction alignment. Despite these issues, both videos effectively display light reflections and shadows, with a clear sense of depth, earning a PV score of 5.
These qualitative results provide an illustration of "what is good embodied video", and reveal the limitations of the video
18

---

WorldSimBench: Towards Video Generation Models as World Simulators
generation models. More qualitative results could be found in: hutpss/siiissggoogecccoo//////es
E. Implicit Manipulative Evaluation-OE
In this section, we provide additional details about Implicit Manipulative Evaluation-Open--nndee Embodied Environ- ment that are not covered in the main paper due to space limitations. Minecraft has emerged as popular open- world environment for developing generalist embodied agents (Lifshitz et al., 2024; Qin et al., 2024; Zhou et al., 2024) due to its diverse tasks (e.g., survival, harvesting, crafting, combat, and creative tasks), varied environments, and interactive mobs, all of which require generalized agent capabilities. Previous works (Qin et al., 2024; Wang et al., 2023d;a) have primarily focused on exploring the capabilities of LLMs or MLLMs as Predictive Text Modelat the S₁ stage. However, no prior research has conducted closed-loop evaluations of World Simulators the Sy stage within Minecraft. To address this gap, we leverage the Steve-1 pipeline to assess the performance of Video Generation Models as World Simulators in Open-Ended Embodied Environment.
E.1. Detailed Description
In Implicit Manipulative EraauuationOOpenEEndee Embodied Environment, we adapt the action space of Steve-1 (Lifshhtzz et al., 2024) to develop a pipeline for the Video Generation Model, enabling it to function as a low-level embodied controller. Additionally, we employ Programmatic Evaluation to benchmark the low-level embodied control capabilities of the Video Generation Model as World Simulators. These tasks are comprehensive, requiring the combination of multiple atomic actions and smooth scene transitions. Each aspect rigorously tests the coherence of the generated content, the consistency with given instructions, and the model's ability to interact effectively with the environment.
Testing. We evaluated performance in OE using five tasks: collecting wood, collecting dirt, collecting seeds, exploring the area, and vertical digging. To reduce evaluation randomness, we selected the most suitable initialization environments for each task (e.g., the agent is initialized in a forest for the wood collection task). During testing, for each task, we randomly select one description from various task instructions and input it into the World Simulator to generate the corresponding video. The video is then continuously translated into actions by a pre-trained goal-based iideo-to-cctinn model, which executes until the test time expires. Each task runs for 10 trials with distinct environment seeds, with a limit of 3,000 frames (i.e.. 2.5 minutes of gameplay).
Training. Due to the low video quality produced by the open-source video generation model based on the provided instructions, we applied additional fine-tuning using data from the OE simulation environment. For Video Generation Model fine-tuning, we use OpenAl Contractor Gameplay Dataset (Baker et al., 2022) which is the same as OE in Explicit Perceptual Evaluation. The training setting could be found in Sup. C.2. For pre-trained goal-based video-to-action model, we use pre-trained (ieve-1(visual) model without extra fine-tuning.
Metrics. We calculate programmatic evaluation metrics by tracking the MineRL environment state throughout each evaluation episode. Several metrics are measured, including travel distance and early-game item collection. Travel distance is defined the agent's maximum displacement is defined as the agent's 's maximum displacement on the vertical (Y) axis from its initial spawn point. For an early-game inventory, we record the maximum count of logs, seeds, and dirt items observed in the agent's 's inventory during the episode.
the horizontal (X-Z) plane from its initial spawn point. Dig depth
E.2. Actions
We use the part of the action space of (Baker et al., 2022) which encompasses nearly all actions available to human players, including keypresses, mouse movements, and clicks. The specific binary actions used in our setup are listed in Tab 11.
E.3. Full Result
Tab. 12 presents the evaluation results of several models across five specific tasks (collect wood, collect dirt, collect seeds, travel distance, and dig depth), along with the average (AVG) score for each model. The models are evaluated under two different conditions: Text and Text& & Image. Notably, to ensure that each task falls within a similar score range, we divided the score for the travel distance task by 10 to calculate the AVG score.
Performance of Models Under Text Condition. Open-Sora-Pee and Lavie demonstrate strong performance under the text- only condition, especially in the collect dirt and travel distance tasks. Their average scores (26.38 and 26.06, respectively) are very close, indicating consistent and robust performance across tasks. ModelScope shows an excellent score in the
19

---

WorldSimBench: Towards Video Generation Models World Simulators
Table Action Space OE.
Behavior
Action
forward
W key
back
S key
left
A key
right
D key
jump
space key
inventory
E key
sneak
shift key
sprint
ctrl key
attack
left mouse button
collect dirt task (52.20), but it performs poorly in tasks like collect wood (14.00) and travel distance (240.72), resulting in overall lower average score (21.050) compared other text-based models. OpenSora stands with the highest overall average score (27.80 (27.00) excelling particularly in collect dirt (70.20) and travel distance (339.87). This suggests that it is well-adapted to variety of tasks and exhibits strong task performance. AnimateDiff shows the weakest performance across all tasks, especially in collect wood (7.40) and collect seeds (3.30), indicating challenges in handling such tasks.
Performance of Models Under Text& & Image Condition. Open-Sora-Pii Poaa-Plan shows a significant drop in average score under the "Text Image" condition, demonstrating that adding image input reduces its performance compared to the text-only condition. In particular. its travel distance score drops from 342.91 to 195. 14, suggesting that incorporating image data might interfere with certain tasks. DynamICrafter and Animat exhibit poor performance across all tasks, especially in collect wood and collect seeds, where they barely complete the tasks (with scores of 0.40 and 0.20, respectively). This may indicate a lack of generalization ability in these models when combining image input with text. Comparing the "Text" and "Text & Image" conditions, we observe that adding image input does not consistently improve task performance and, in some cases, even degrades it. We also observed that the success rates various tasks significantly decrease when an image added as an additional condition. This indicates that the current video generation models need improvement in handling multiple conditional inputs.
Table 12. Detail Result of Open-Ended Embodied Environment in Implicit Manipulative Evaluation.
Specific Tasks
Model
Condition
AVG
Collect Wood
Collect Dirt
Collect Seed
Travel Dis.
Dig Depth
Open-Sora-Plan
26.38
19.90
50.20
7.30
342.91
20.20
Lavie
26.06
23.50
56.00
11.60
270.20
12.20
ModelScope
Text
21.050
14.00
52.20
6.30
240.72
8.70
OpenSora
27.80
21.20
70.20
10.40
339.87
3.20
AnimateDiff
13.10
7.40
22.90
3.30
274.19
4.50
Open-Sora-Plan
10.28
11.10
12.50
2.60
195.14
5.70
DynamiCrafter
Text & Image
4.06
0.40
0.30
1.30
130.04
5.30
Easyanimate
4.84
0.20
0.70
1.70
157.12
5.90
E.4. Roll Out
Fig. 8 illustrates the downstream execution process in the Open-Ended Embodied Environment, along with the corresponding textual instructions.
F. Implicit Manipulative Evaluation-AD
In this section, we provide additional details about Implicit Manipulative Evaluation Autitiioon-Auuoooouuss Driving that are not covered the main paper due to space limitations.
20

---

WorldSimBench: Towards Video Generation Models World Simulators
Insturction: Collect wood in the forest.
Figure Rollout of Open-Ended Embodied Environment in Implicit Manipulative Evaluation.
F.1. Detailed Description
In Implicit Manipulative Evaluation-Autonn develop pipeline for the Video Generation Model, enabling it to function as a low-level embodied controller. Additionally, we employ LangAuto (Language-guidee Autonomous Driving) CARLA benchmark, to evaluate the low-level embodied control capabilities of the Video Generation Model as World Simulators. These tasks are designed to be comprehensive, spanning all publicly available towns in CARLA, covering a diverse range of scenarios e.g., highways, intersections, and roundabouts. Additionally, they account for 16 different environmental conditions, combining distinet weather settings (Clear, Cloudy, Wet, MidRain, WetCloudy, HardRain, SoftRain) with daylight conditions (Night, Noon, Sunset). Each aspect rigorously tests the coherence of the generated content, the consistency with given instructions, and the model's ability to interact effectively with the environment.
Driving, adapt the action space of LMDrive (Shao al., 2024) to
Testing. We evaluated performance in Autonomous Driving using the LangAuto- Tiny benchmark setting where the route length is shorter than 150 meters. We posit that shorter driving distances provide a more effective test of the low-level control capabilities World Simulators. Longer routes typically involve more instructions, which are prone to misalignment with the real-time simulation environment. Therefore, we opt evaluate performance on shorter routes to minimize these discrepancies. During testing, randomly select one description from various task instructions and input into the World Simulator generate the corresponding video. The video is then continuously translated into actions by pre-trained goal-based video-to-action model, which executes until the test time expires. We use the corresponding LangAuto-Tiny instructions and the first-person view rendered by the real-time CARLA simulation environment input to the video generation model. The generated video is then continuously transformed into downstream control signals using a pre-trained caleo-to-action model until the agent reaches a predefined success zone or the task terminated due to factors e.g., timeouts or collisions.
Training. Due to the low video quality produced by the open-source video generation model based on the provided instructions, applied additional fine-tuning using data from the AD simulation environment. For Video Generation Model training, we use LMDrive Training Dataset (Shao et al., 2024). We preprocessed the training data according to each model's s default input format and performed fine-tuning following the official implementation without changing the training settings. We fine-tuned each model using a short video generation setting with approximately 20 frames. For the video-to-action model, we use pre- tre-trained LMdrive model. Additional fine-tuning was conducted based on the test requirements. We provided the model with arbitrary text instructions and replaced the visual input with the future frame while keeping all
21

---

WorldSimBench: Towards Video Generation Models World Simulators
other training settings consistent with LMDrive.
Metrics. We consider eight key metrics introduced by the CARLA Leaderboard (Dosovitskiy et al., 2017): Route Completion (RC), Infraction Score (IS), Driving Score (DS), Vehicle Collisions (VC), Pedestrian Collisions (PC), Layout Collisions (LC), Red Light Violations (RV), and Offroad Infractions (01). Route Completion refers to the percentage of the total route length that the agent has completed. This metric only accounts for the distance traveled along the predetermined route, where each segment corresponds to a navigation instruction. If the agent strays too far from the route, it is considered to have violated the instruction, resulting the episode being marked tracks any infractions caused by the agent, with penalties applied for collisions or traffic violations through a corresponding discount factor. The Driving Score is the product of the route completion ratio and the infraction score, reflecting both driving progress and safety, and is widely regarded as the primary ranking metric. The precise definitions of the residual metrics can be found in the CARLA documentation (Dosovitskiy et al., 2017).
failure and terminated. The Infraction Score
F.2 Actions
The video generated by the World Simulator is continuously fed into the video-to-action model to obtain the corresponding waypoints. The agent then generates control signals based on the generated waypoints and the conversion strategy used in CARLA.
F.3. Full Result
Tab. 13 presents the evaluation results of several models across eight metrics. The evaluation results highlight significant differences in how video generation models perform in autonomous driving tasks. Open-Sora-Plan stands out in trajectory generation, instruction following, and environment perception, producing high-quality videos that effectively support task execution. In contrast, DynamiCrafter and EasyAnimate struggle with generating detailed and consistent video content, particularly when handling complex or dynamic scenes. These models require improvements video generation quality, scene understanding, and task alignment enhance their performance.
From video generation perspective, several key areas for future development are identified: Improved Trajectory Generation: High-quality trajectory generation is essential for accurate control signals. Models must focus on generating more coherent and precise trajectories, especially in dynamic environments, to ensure vehicles follow instructions and avoid collisions. Enhanced Instruction Following: Generated videos should closely align with task instructions, particularly in changing environments, enabling vehicles to adapt quickly while maintaining task accuracy. Better Environment Perception: Future models need generate videos that accurately represent complex scenes, e.g., interactions with pedestrians, other vehicles, and varied terrains. More detailed and realistic video generation will provide stronger input for real-time decision-making in the control system.
In summary, advancing trajectory accuracy, instruction alignment, and environment representation will be crucial for improving the overall performance these video generation models in autonomous driving tasks.
Table 13. Detail Result of Autonomous Driving in Implicit Manipulative Evaluation.
Model
DS(t)
RC(1)
IS(t)
VC(.)
PC(J)
I.C())
RV(()
OI(L)
Open-Sora- Lenn-Sor--Paan
31.054
38.249
0.767
2.400
0.000
4.401
1.133
3.514
DynamiCrafter
24.491
37.189
0.599
5.030
0.000
4.896
0.937
3.221
EasyAnimate
17.414
28.475
0.607
0.000
0.000
29.344
0.000
1.690
E.4. Roll Out
Fig. 9 illustrates the downstream execution process in the Autonomous Driving, the corresponding text instructions be found in the lower left corner of each frame.
G. Implicit Manipulative Evaluation-Ral RM
In this section, we provide additional details about Implicit Manipulative Evaluation- Robot Manipulation that are not covered in the main paper due to space limitations.
22

---

WorldSimBench: Towards Video Generation Models World Simulators
Figure 9. Rollout of Autonomous Driving in Implicit Manipulative Evaluation.
G.1. Detailed Description
We primarily conduct our experiments on the CALVIN benchmark (Mees et al., 2022), which is specifically designed for long-horizon, language-conditioned manipulation tasks. CALVIN includes four simulated environments (labeled A, B, C. and D) that differ in textures and object placements. Each environment features Franka Emika Panda robot positioned next to desk with various manipulable objects. The evaluation protocol tests model performance across 1,000 unique instruction chains, each consisting of five distinct tasks. By providing an extensive dataset paired with natural language annotations, the CALVIN benchmark can provide close-loop evaluation platform for evaluating World Simulator to test its generation and generalization capabilities.
Testing. We evaluated performance in Robot Manipulation using the CALVIN benchmark benchmark, policy models are trained on demonstrations from environments A, B, and C, and evaluated in zero-shot manner in environment D. During the testing phase, we leverage World Simulators and pre-trained video-to-action model to tackle novel manipulation tasks guided by user-specified natural language commands. Given a current observation, we generate future video predictions using the World Simulator for the manipulation task with text instruction. Once the video is sampled, we then execute the video-to-action policy conditioned on for timesteps, where & is a testing hyperparameter. After timesteps, the video prediction refreshed by sampling from the World Simulator again, and the process is repeated.
Training. Due to the low video quality produced by the open-source video generation model based on the provided instructions, applied additional fine-tuning using data from the RM simulation environment. For Video Generation Model training, we use Calvin(ABC.D) ABC_D) datset (Mees et al., 2022). We preprocessed the training data according to each model's default input format and performed fine-tuning following the official implementation without changing the training settings. We fine-tuned each model using a short video generation setting with approximately 20 frames. For the video-to-action model, we use pre-trained Susie policy without extra fine-tuning.
Metrics. We report the success rates and the average task length completed (out of five tasks) for each evaluation sequence.
23

---

WorldSimBench: Towards Video Generation Models World Simulators
G.2. Actions
For low-level control, we utilize the same action space as Calvin (Mees et al., 2022).
G.3. Full Result
Based on the results shown in Tab. 14, Open-Sora- demonstrates consistent performance, with an average task length of 2.95, indicating its ability reliably complete task sequences. While DynamiCrafter achieves a higher success rate of 0.95 on the initial task, its performance declines as task complexity increases, suggesting limitations in handling longer manipulation sequences. EasyAnimate, although moderately successful in completing early tasks, experiences sharp decline in performance as task difficulty rises, reflected in its lower average task length of 2.05.
Overall, the models' ability to consistently complete multiple tasks in succession showcases their potential in downstream applications, with Open-Sora-Plan emerging as the most capable. However, the observed decrease in success rates as task complexity increases highlights the need for further improvements video-to-action translation, particularly in addressing the challenges posed by longer and more complex manipulation sequences.
Table 14. Detail Result of Robot Manipulation in Implicit Manipulative Evaluation.
Task completed in a row (%)
f
Method
Avg. Len. t
2
3
5
Open-Sora-Pee
0.85
0.70
0.60
0.40
0.40
2.95
DynamiCrafter
0.95
0.75
0.55
0.25
0.25
2.75
Easy 0.90
0.60
0.35
0.10
0.10
2.05
To minimize the impact of randomness caused by the number of experiments, conducted additional 100 trajectories evaluation. The results are presented in Tab 15. Compared to the 20-trajectories setup, the results from the 100-trajectories setup show slight variations but maintain a consistent overall trend. We also compared the performance with Unipi based on the 25-trajectory setup described in SUSIE, and can be observed that the tested video generation models(Open-- Srpen-Sora-Plan DynamiCrafter, EasyAnimate) demonstrate superior capabilities compared to the PVDM (?) latent video diffusion model utilized by Unipi.
Table 15. Detail Result of Robot Manipulation in Implicit Manipulative Evaluation, by running 100 trajectories. Results reported by
Susie (Black et al., 2023).
Task completed in a row (%)
Method
Avg. Len. t
1
2
3
4
5
(HiP)
0.08
0.04
0.00
0,00
0.00
UniPi" (Susie)
0.56
0.16
0.08
0.08
0.04
Open-Sora-Pae
0.89
0.72
0.63
0.34
0.32
3.12
DynamiCrafter
0.93
0.69
0.51
0.27
0.18
2.64
EasyAniman
0.92
0.55
0.32
0.16
0.13
2.08
G.4. Roll Out
Fig. 10 illustrates the downstream execution process in the Robot Manipulation, along with the corresponding textual instructions.
H. Discussion of Vision-Language-Actin Models
The generated videos have the potential significantly enhance the performance of Vision-Language-Action (VLA) models by addressing two key challenges in training such models: the availability of diverse, high-quality training data and the need for effective reward functions in real-world scenarios.
Data Augmentation and Hindsight Relabeling for Imitation Learning. Generated videos can serve as a valuable source of synthetic data for training VLA models. By leveraging the diversity and scalability of generative models, we can create
24

---

WorldSimBench: Towards Video Generation Models World Simulators
Instruction: Press the button to turn on the led light
Instruction: Go push the blue block right
Figure 10. Rollout of Robot Manipulation in Implicit Manipulative Evaluation.
wide array of training scenarios, covering edge cases and rare events that are difficult to capture in real-world datasets. Additionally, these videos enable hindsight relabeling, process where we retrospectively adjust the labels of generated data to align with desired outcomes. This approach particularly effective for imitation learning, allowing VIAA models to learn optimal behavior by mimicking successful trajectories represented in the generated videos. By expanding the data distribution and improving its quality, generative videos can lead to more robust and generalizable VLA models.
Reward Generation for Online Reinforcement Learning. Beyond data augmentation, generated videos Reward Generator in reinforcement learning (RL) contexts. Unlike traditional RL setups that rely pre-defined reward functions within simulator, generative videos enable the creation of dense and context-aware reward signals tailored to real-world tasks. For example, they can simulate desirable outcomes or intermediate goals, providing detailed feedback to the agent. This capability particularly crucial for transferring RL. models to real-world environments, where designing explicit reward functions often impractical. By aligning generated rewards with real-world objectives, we can bridge the gap between simulation and reality, allowing VI.A models achieve higher performance in real-world tasks.
25


Reviews:
[{"review_id": "ilDGi3o8gP", "reviewer": "Reviewer_yx3f", "summary": "This paper proposes WorldSimBench, a benchmark used to evaluate the world simulation performance of video generative models. This paper investigates several video valuation benchmarks and introduces a hierarchy for classifying video models. This paper evaluates several video generative models through proposed Explicit Perceptual Evaluation and Implicit Manipulative Evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper：\n\n- **Solid Experiment**: *\"This paper includes extensive empirical results to support the main claim made in this paper.\"*\n- **Novelty and Soundness**: *\"This paper introduces a human preference evaluator and video-to-action evaluation metrics to evaluate the world simulation abilities of video models, which are interesting.\"*\n- **Reasonable Evaluation Criteria**: *\"Other metrics introduced in this paper also make sense.\"*\n- **Inspiring for the Community**: *\"This paper proposes a new benchmark to evaluate video-based world models, which can be valuable for future research in this area.\"*\n\n---\n\n>More Video Models\n\nWe have added evaluations on **HunyuanVideo**, **Cosmos Diffusion 7B**, and **CogVideoX1.5-5B**:\n\nEvaluation results in **OE**, which can be incorporated into **Table 6** of the supplementary material. The abbreviations are listed in Table 2.\n\n| Model         | BC   | FC   | IA   | SA   | VC   | TJ   | EI   | Overall |\n|---------------|------|------|------|------|------|------|------|---------|\n| HunyuanVideo  | 1.95 | 2.0  | 2.0  | 1.6  | 2.0  | 2.0  | 1.65 | 1.89    |\n| Cosmos        | 1.9  | 2.0  | 2.0  | 1.8  | 2.0  | 2.0  | 1.8  | 1.93    |\n| CogVideoX     | 1.95 | 2.0  | 2.0  | 2.0  | 2.0  | 2.0  | 1.55 | 1.93    |\n\nEvaluation results in **AD**, which can be incorporated into **Table 7** of the supplementary material. The abbreviations are listed in Table 2.\n\n| Model         | AE   | IA   | PV   | TJ   | KE   | SF   | Overall |\n|---------------|------|------|------|------|------|------|---------|\n| HunyuanVideo  | 3.75 | 5.0  | 4.1  | 4.8  | 3.4  | 5.0  | 4.34    |\n| Cosmos        | 4.15 | 5.0  | 4.6  | 4.9  | 3.65 | 5.0  | 4.55    |\n| CogVideoX     | 3.45 | 5.0  | 3.9  | 4.8  | 3.0  | 5.0  | 4.20    |\n\nEvaluation results in **RM**, which can be incorporated into **Table 8** of the supplementary material. The abbreviations are listed in Table 2.\n\n| Model         | AE   | BC   | FC   | IA   | PV   | TJ   | EI   | Overall |\n|---------------|------|------|------|------|------|------|------|---------|\n| HunyuanVideo  | 4.0  | 3.9  | 4.0  | 1.9  | 5.0  | 5.0  | 4.1  | 3.99    |\n| Cosmos        | 4.2  | 4.26 | 4.0  | 2.8  | 5.0  | 5.0  | 4.44 | 4.24    |\n| CogVideoX     | 3.85 | 4.0  | 4.0  | 2.2  | 5.0  | 5.0  | 4.23 | 4.04    |\n\n---\n\n>Quantitative Experimental Results\n\n**D.2. Qualitative Results and Figure 7** in the supplementary material shows some of the Quantitative Experimental Results, and we have also provided corresponding analyses for these results. Additionally, we have showcased more visual results on an **[Anonymous Website](https://sites.google.com/view/worldsimbench)** to address your concerns.\n\n---\n\n>Questions\n\nThere are already quite a few papers and projects based on video models for specific tasks. Methods [1, 2, 3, 4] have demonstrated impressive results in robot manipulation and navigation tasks. Furthermore, future video prediction models can leverage Internet-scale pretraining and visual understanding to guide low-level goal-conditioned policies, potentially showing better generalization in new scenarios. However, during the inference phase, these methods tend to incur additional computational overhead, making them slower compared to traditional methods like imitation learning. Nevertheless, with the continued development of models and acceleration techniques, these approaches may hold significant potential and be worth further exploration.\n\n---\n\nThank you for your constructive and thoughtful comments. They were indeed helpful in improving the paper. We will incorporate the above clarifications, examples, and additional experiments into the paper, and we hope these revisions address your concerns. If you have any further questions, please do not hesitate to continue the discussion with us.We hope this will provide a more comprehensive understanding of the performance of different models on our benchmark.\n\n---\n\n**References**  \n[1] Learning Universal Policies via Text-Guided Video Generation  \n[2] Compositional Foundation Models for Hierarchical Planning  \n[3] VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulation  \n[4] Grounding Video Models to Actions through Goal Conditioned Exploration", "claims_and_evidence": "Yes. This paper includes extensive empirical results to support the main claim made in this paper.", "methods_and_evaluation_criteria": "This paper introduces a Human Preference Evaluator, which can be quite interesting and applicable. Other metrics introduced in this paper also make sense.", "theoretical_claims": "N\\A", "experimental_designs_or_analyses": "Yes. However, only quantitative experimental results are included in this paper, while qualitative results are missing.", "relation_to_broader_scientific_literature": "N\\A", "essential_references_not_discussed": "I encourage the authors to include more advanced open-sourced video models for evaluation, such as Huanyuan and Cosmos.", "other_strengths_and_weaknesses": "## Strengths\n- This paper proposes a new benchmark to evaluate video-based world models, which can be valuable for future research in this area.\n- This paper introduces a human preference evaluator and video-to-action evaluation metrics to evaluate the world simulation abilities of video models, which are interesting.\n- This paper investigates the world simulation performance of several video generative models.\n## Weaknesses\n- Only quantitative experimental results are included in this paper, while qualitative results are missing. Including some qualitative visualization can let readers know whether your benchmark is challenging or easy for nowadays video models.\n- Some world foundation models instead of traditional video generative models should be evaluated by the proposed benchmark, such as the newly introduced Cosmos.", "other_comments_or_suggestions": "NA", "questions_for_authors": "- Since you mainly evaluate some text-based video models, I am curious does these models exhibit superior world simulation ability in the embodied tasks.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8EGEoZlGPh", "reviewer": "Reviewer_outL", "summary": "The lack of categorization based on inherent characteristics hinders predictive model development, and existing benchmarks fail to evaluate highly embodied models effectively. To address this, this paper introduces WorldSimBench, a dual evaluation framework for World Simulators. It includes Explicit Perceptual Evaluation, using the HF-Embodied Dataset and a Human Preference Evaluator for visual fidelity assessment, and Implicit Manipulative Evaluation, measuring video-action consistency in dynamic environments. Authors evaluate across three embodied scenarios: open-ended environments, autonomous driving, and robot manipulation. It provides key insights to advance video generation models and strengthen their role in embodied AI.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper：\n\n- **Inspiring and Novelty**: *\"It provides key insights to advance video generation models and strengthen their role in embodied AI.\"*\n- **Reasonable Evaluation Criteria**: *\"These benchmarks provide a comprehensive assessment of a simulator’s effectiveness in real-world tasks.\"*\n- **Useful Dataset**: *\"HF-Embodied Dataset is useful for constructing evaluation metrics.\"*\n\n---\n\n>Quantitative Experimental Results\n\nWe appreciate your suggestion regarding the presentation of results. To address this, we have created a **[Anonymous Website](https://sites.google.com/view/worldsimbench)** to showcase more visualized results, which we believe will enhance the clarity and impact of our paper. We hope this will provide a more comprehensive understanding of the performance of different models on our benchmark.\n\n---\n\n>Broader Scientific Literature\n\nWe agree that a strong benchmark should clearly expose the limitations of current models and point toward directions for future improvement. While our paper already discusses these aspects—for example, in **Section 4.3** we analyze the physical limitations of current video generation models as world simulators; **Appendix C.1** demonstrates how human feedback from the HF-Embodied dataset can improve video quality via reinforcement learning; and **Appendix H** shows how generated videos can benefit VLA methods and downstream embodied tasks—we acknowledge that these insights could be presented more intuitively. In the revised version, we will make these findings more visually explicit and accessible, helping readers better grasp the limitations of current approaches and the future opportunities they reveal. \n\n---\n\n>Suggestion of Dataset Expansion\n\nThank you for your thoughtful and constructive feedback. We’re glad to hear that you find the HF-Embodied Dataset valuable, especially in terms of evaluation and task diversity. We appreciate your suggestion regarding the inclusion of additional annotations such as camera poses and egomotion data. This is indeed a valuable direction, and we agree that such information could further enrich the dataset and benefit downstream embodied tasks.\n\nDue to the diverse sources and large scale of our current dataset, obtaining accurate camera pose and egomotion annotations presents certain challenges. However, we recognize their importance and are actively exploring methods—both automated and semi-automated—to incorporate such annotations in future versions of the dataset. We believe this will significantly boost its applicability in more complex embodied scenarios.\n\nThank you again for the suggestion, it provides a meaningful direction for our future work.\n\n---\n\n>Effectiveness of Fine-Tuning\n\nThank you for your insightful comment. To ensure the effectiveness and comparability of our fine-tuning process across different models, we used the same dataset for training all models. In order to maintain fairness and optimize performance, we followed the fine-tuning strategies recommended by the authors of each respective model. This approach allows us to ensure that each model is trained in the most optimal way according to its specific architecture and design.\n\nBy adhering to the suggested fine-tuning protocols, we aim to eliminate potential biases and ensure that the models' performances are evaluated under comparable conditions. We hope this approach addresses your concern regarding the impact of different fine-tuning methods on the results. Thank you again for raising this important point!\n\n---\n\n>Choice of Models\n\nWe have added evaluations on HunyuanVideo, Cosmos Diffusion 7B, and CogVideoX1.5-5B. Due to space limits, please see references in our reply to Reviewer yx3f **More Video Models**.\n\n---\n\n>Future Direction\n\nWe appreciate the reviewer’s insightful suggestions regarding failure analysis and prompt categorization. We fully agree that understanding the limitations of the Human Preference Evaluator and identifying failure modes in instruction alignment are meaningful directions for future research.\n\nSpecifically, analyzing when and why video generation models fail, and categorizing prompts to uncover scenario-specific weaknesses, could offer valuable insights into the robustness and generalization capabilities of current models. These efforts would require dedicated experiments and analysis, which we believe merit a standalone investigation.\n\nWe will incorporate these ideas into the **Future Work** section of the paper to inspire and guide follow-up research in this important area. Thank you again for highlighting this valuable direction.\n\n---\n\nThank you for your thoughtful and constructive feedback—it has been instrumental in helping us improve the paper. We will incorporate the above clarifications, examples, and additional experiments into the paper, and we hope these revisions address your concerns, and we’d be glad to continue the discussion if you have any further questions or suggestions.", "claims_and_evidence": "As video generation models continue to advance, we lack a more powerful and comprehensive benchmark. I completely agree with this point, especially since earlier evaluation metrics focused too much on video quality rather than its practical usability—particularly whether the generated videos can serve as a world simulator.\n\nThis viewpoint is somewhat obvious to practitioners in the field. Therefore, the key to evaluating this paper lies in whether the benchmark it provides aligns with researchers' needs. Its contribution ultimately depends on the novelty and relevance of the new dataset and evaluation metrics.", "methods_and_evaluation_criteria": "WorldSimBench evaluates World Simulators at two levels: Explicit Perceptual Evaluation, assessing human-perceived quality, and Implicit Manipulative Evaluation, testing video-to-control translation in closed-loop tasks. It covers three key embodied scenarios: Open-Ended Environments (OE) using Minecraft for complex tasks, Autonomous Driving (AD) for stability in dynamic conditions, and Robot Manipulation (RM) for precise control in physical interactions. These benchmarks provide a comprehensive assessment of a simulator’s effectiveness in real-world tasks.", "theoretical_claims": "No theoretical claims are required for review in this work.", "experimental_designs_or_analyses": "8 popular video generation models were evaluated, including Open-Sora-Plan, Lavie, ModelScope, Open-Sora, AnimateDiff, Open-Sora-Plan, Dynamicrafter, and EasyAnimate. These models are assessed through both Explicit Perceptual Evaluation and Implicit Manipulative Evaluation across three scenarios: Open-Ended Embodied Environment, Autonomous Driving , and Robot Manipulation. Each model is fine-tuned on specific datasets corresponding to these embodied scenarios for a comprehensive evaluation.", "relation_to_broader_scientific_literature": "A strong benchmark should highlight the shortcomings of current models and indicate future directions for improvement. Right now, I think the paper could do a better job of making these aspects more visually explicit.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Overall, the evaluation of this work should focus on 3 key aspects:\n\n1. Data Contribution – The novelty and usefulness of the proposed dataset.\n2. Metrics and evaluation – Whether the evaluation metrics effectively capture the critical aspects of video generation. How well the benchmark reveals the limitations of current models.\n3. Guiding Future Research – Whether it provides clear insights into the next steps for advancing video generation models.\n\n**Data contribution**: HF-Embodied Dataset is useful for constructing evaluation metrics. Large-scale embodied videos sourced from the internet, paired with captions, are used to train data generation models. Fine-grained human feedback annotation is then applied based on the corresponding Task Instruction Prompt List, ensuring coverage across multiple embodied dimensions. \n\nAdditionally, instructions for video generation are expanded using GPT and manually verified to construct a comprehensive Task Instruction Prompt List, which serves as a guideline for both data generation and evaluation.\n\nI think this aspect of the contribution is positive, but there is still room to expand the scope of the dataset. For example, since the paper emphasizes embodiment, it would be beneficial to include additional annotated information, such as camera poses and other egomotion data. This could further enhance the dataset’s utility. This is just a suggestion for the authors to consider in future iterations.\n\n\n**Metrics and evaluation**: The authors correctly focus on instruction alignment and train a Human Preference Evaluator using the HF-Embodied Dataset to evaluate eight models. They then observe that these models struggle with instruction alignment. However, I have two concerns:\n\n1. Effectiveness of Fine-Tuning: The authors fine-tune these models on the new dataset, but different models can exhibit significantly different behaviors depending on how they are fine-tuned. Whether SFT, LoRA, or other techniques are used can also impact the results. How do the authors ensure that their fine-tuning is effective and comparable across models?\n\n2: Choice of Models: The models used in the paper seem somewhat outdated. It would be much more compelling if the evaluation included newer models like CogVideoX, as this would provide a better assessment of current model capabilities.\n\n**Future Direction**: I think this section could be further improved. I would like to see more failure cases of the Human Preference Evaluator, as well as a deeper analysis of when and why current video generation models fail.\n\nThe paper could further categorize different prompts to identify specific scenarios where models are more prone to failure. This would provide a more detailed and actionable understanding of instruction alignment challenges in video generation.", "other_comments_or_suggestions": "NA", "questions_for_authors": "I have mentioned this in the previous sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HoDToutfIg", "reviewer": "Reviewer_sWAY", "summary": "This paper proposes a framework that can be used to evaluate world simulators, such as video generation models. the framework is divided into a couple of components, a perceptual evaluation, and an embodied evaluation. the perceptual one uses a model trained on human collected data to score the results. the embodied on measures task success by combining it with a trained video-to-action model.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your recognition of our paper：\n\n- **Inspiring for the Community**: *\"I believe this paper will have some impact to a sizable audience in computer vision and ML communities.\"*\n- **Comprehensive Literature Review**: *\"References are adequate.\"*\n\n---\n\n>Supplementary Material\n\nOur supplementary material is included in the same PDF file, appended to the end of the main paper.\n\n---\n\n>Quantitative Experimental Results\n\nWe appreciate your suggestion regarding the presentation of results. **D.2. Qualitative Results** and **Figure. 7** in the supplementary material shows some of the Quantitative Experimental Results, and we have also provided corresponding analyses for these results. We have also created a **[Anonymous Website](https://sites.google.com/view/worldsimbench)** to showcase more visualized results, which we believe will enhance the clarity and impact of our paper. We hope this will provide a more comprehensive understanding of the performance of different models on our benchmark.\n\n---\n\n>Discussion of VLA\n\nWe appreciate the reviewer for bringing this up—connecting our work to Vision-Language-Action (VLA) models is indeed a valuable perspective. We would like to clarify that we have already discussed this in the supplementary material, specifically in **Section H. Discussion of Vision-Language-Action Models.**\n\nAs summarized there, the generated videos can contribute to VLA models in two key ways:  \n1. As a source of **diverse and scalable training data for imitation learning**, including through hindsight relabeling, which helps models learn from generated successful trajectories.  \n2. As a means of **reward generation in reinforcement learning settings**, where generated videos can simulate goal states or desirable behaviors, enabling the creation of dense and context-aware reward signals.\n\nThis is particularly beneficial for bridging the sim-to-real gap in real-world tasks where explicit reward specification is often challenging.\n\n---\n\n>Survey Paper Concern\n\nWe respectfully disagree with the assessment that our paper resembles a survey. While our work does provide a structured taxonomy and draws insights from multiple perceptual and task-based settings, these components are not retrospective summaries, but rather integral parts of a novel and operational evaluation framework.\n\nSpecifically, we make the following contributions that go beyond a survey:\n\n- We design and implement a **novel evaluation framework** tailored for assessing video generation models as world simulators—a key open challenge in the community. This framework is not merely a collection of metrics; it includes:\n  - A **human-aligned perceptual evaluation protocol** that maps cognitive dimensions to measurable factors, supported by the **HF-Embodied dataset** with costly human preference annotations that enrich the quality and depth of the evaluation.\n  - A **physics-driven embodied task evaluation**, which leverages embodied agents in simulated environments to test the rationality and functional validity of generated videos—something traditional metrics and human studies often overlook.\n\n- We introduce and apply a **taxonomy of perceptual attributes** to systematically assess different facets of human judgment. This taxonomy is used to guide newly designed human studies, not to summarize existing ones.\n\n- We conduct **extensive experiments** across diverse tasks and settings, showcasing the practical utility and diagnostic power of our framework. Our results not only highlight model behaviors but also reveal consistent patterns between perceptual and task-level evaluations—demonstrating the framework’s effectiveness and explanatory value.\n\nIn summary, rather than surveying existing work, our paper offers a principled and implementable approach to a previously underexplored problem. We believe this constitutes a significant and novel contribution to the evaluation of video generation systems, and we hope the reviewer will reconsider their assessment in this light.\n\n---\n\nThank you for the constructive feedback. We will incorporate the above clarifications, examples, and additional experiments into the paper, and we hope these revisions address your concerns.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "n/a", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "i believe this paper will have some impact to a sizable audience in computer vision and ML communities", "essential_references_not_discussed": "references are adequate", "other_strengths_and_weaknesses": "1. there are limited qualitative results. this makes it very difficult to evaluate the performance\n\n2. can the generated video be applied to improve vision-language-action models?\n\n3. I feel that the proposed method is more like a survey paper, which means that the novelty is limited.", "other_comments_or_suggestions": "n/a", "questions_for_authors": "please see the comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pvY0q6nKhN", "reviewer": "Reviewer_cbEM", "summary": "This paper introduces WorldSimBench, a benchmark for evaluating video predictive models from an embodied perspective, in contrast to previous approaches focused on generative evaluation. It features Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, combining human preference assessments from a visual standpoint with action-level evaluations in embodied tasks. WorldSimBench covers three key embodied scenarios: Open-Ended Embodied Environments, Autonomous Driving, and Robot Manipulation. Experiments on multiple existing models reveal the strengths and limitations of current world simulators. Additionally, it provides the HF-Embodied Dataset, comprising 35,701 entries, to further support research in this domain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive and thoughtful comments. They were indeed helpful in improving the paper. We take this opportunity to address your concerns:\n>Hierarchical Structure of Modalities\n\nThe hierarchical structure we propose is not based on semantic abstraction, but on the practical difficulty of turning different predicted modalities into real-world executable actions—a key consideration for evaluating video generation models as world simulators. Language is high-level but often ambiguous and under-specified for action planning; images are more concrete but lack temporal and causal context; videos, while potentially modeling transitions, may produce physically inconsistent content. To address this, we introduce the notion of actionable video prediction—predicted videos that are both instruction-aligned and physically plausible, making them suitable for downstream decision-making in embodied agents. This task-driven, grounded perspective is essential for advancing video generation toward real-world utility.\n\n---\n\n>CALVIN Evaluation Trials\n\nWhile the main text initially reported 20-trial results on the CALVIN benchmark, we also conducted 100-trial evaluations following prior works such as UniPi and SuSIE (Appendix Table 15). In response to your suggestion, we have now further extended our evaluation to **1,000 trials** to improve statistical reliability.\n\n#### Task completed in a row (%) ↑ for (1,2,3,4,5)\n| Method| 1| 2| 3| 4| 5| Avg. Len. ↑ |\n|-------------------|-------|-------|-------|-------|-------|-------------|\n| **Open-Sora-Plan**|91.8|73.2|61.2|35.3|32.2| 3.06|\n| **DynamiCrafter**|92.4|69.2|50.1|29.3|18.7| 2.71|\n| **EasyAnimate**|88.3|57.3|33.9|17.3|12.2| 2.05|\n\n---\n\n>AD Evaluation Setup\n\nWe would like to clarify that our AD evaluation does not rely on predefined trajectories. Instead, the agent receives language instructions, and a video generation model predicts future frames accordingly. These are passed to a fixed video2action policy—trained only on ground-truth data and shared across all models—to generate control actions. As this policy is not trained on generated videos, task success rates directly reflect the quality and physical plausibility of the predictions. Our setup is designed to evaluate whether models can produce instruction-aligned, physically grounded outputs, rather than reproduce preset trajectories.\n\n---\n\n>Related Work in Autonomous Driving\n\nWe will revise the Related Work section to include these approaches.\n\n---\n\n>Evaluation Persuasiveness and Domain Generality\n\nWe respectfully clarify that our evaluation framework is tailored for embodied settings, where video predictions must be both visually plausible and physically actionable. Unlike prior work focused on perceptual quality, our task-driven protocol assesses a model’s ability to support real-world decision-making by using predicted videos to control agents via a shared video-to-action policy. We introduce the concept of actionable video prediction—emphasizing instruction alignment and physical feasibility—and combine perceptual and task-based evaluations. While video-to-action may not apply universally, it is well-suited for embodied tasks like robotics and autonomous driving, where grounding in action space is essential. This embodiment-centered perspective brings both novelty and practical value to evaluating generative world models.\n\n---\n\n>Suggestions #1\n\nWe address this point in **Appendix C.1**, where we explore how human feedback annotations can be leveraged to further improve the video model’s capabilities through reinforcement learning. Specifically, we conducted experiments using the Open-SORA-Plan framework to fine-tune the model based on human feedback. The results of these experiments are reported in **Appendix Tab.5**, which demonstrate that incorporating human feedback can lead to measurable improvements in planning and control performance.\n\n---\n\n>Suggestions #2, #3\n\nWe will fix the typos in the revised version.\n\n---\n\n>Questions: Conclusions and Insights\n\nOur paper highlights several key insights across different sections. In **Section 4.3**, we analyze limitations of current video generation models as world simulators with several physical aspects. **Appendix C.1** shows how human feedback from the HF-Embodied dataset can enhance video quality via reinforcement learning, offering a path to integrate human preferences into training. **Appendix H** further demonstrates how generated videos and our benchmark can benefit VLA methods, revealing broader utility in downstream embodied tasks. Together, these findings provide actionable guidance for future model design, including better multi-modal supervision, improved control fidelity, and the use of synthetic data for generalization.\n\n---\n\nThank you for the constructive feedback. We will incorporate the above clarifications, examples, and additional experiments into the paper, and we hope these revisions address your concerns.", "claims_and_evidence": "This is a benchmark paper and does not make traditional claims.\n\nThe authors propose:\n1. Evaluating the capability of predictive models from a hierarchical perspective.\n2. Assessing models from an embodied perspective by leveraging video-action models to evaluate driving and manipulation tasks, revealing significant shortcomings in existing models.\n\nOverall, evaluating from an embodied perspective is a valuable approach, as current generative models are clearly insufficient. However, regarding the first point, I have some doubts. As the authors mentioned, prediction can occur across different modalities with varying focuses, but it does not necessarily imply a clear hierarchical structure.", "methods_and_evaluation_criteria": "The authors evaluate video prediction models from explicit perceptual and implicit manipulative perspectives. They assess models based on visual quality, condition consistency, and embodiment, which is a reasonable approach. However, the work lacks significant innovation, serving more as an integrative evaluation of previous research.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "1. Running only 20 trials on CALVIN is not reasonable, as the standard evaluation setting typically involves 1,000 trials.\n2. The evaluation of AD also feels somewhat problematic. It essentially involves predefining trajectories as instructions, using those instructions to guide video generation, and then converting them into corresponding actions. Throughout this process, information loss occurs at each transformation step.", "relation_to_broader_scientific_literature": "This work attempts to evaluate the capabilities of existing video generation models from an embodied perspective, aiming to make them more practically applicable.", "essential_references_not_discussed": "Predictive models have been widely explored in both autonomous driving and robotics. For instance, in autonomous driving, works such as GAIA-1, Drive-WM, and DriveDreamer focus on world models. However, this paper's related works section lacks discussion on these contributions.", "other_strengths_and_weaknesses": "I appreciate the authors' contributions, as they have conducted extensive experiments across gaming, driving, and robotics. Although some metric evaluations may not be entirely reasonable, their work still contributes to the advancement of the field.\n\nHowever, the evaluation lacks strong persuasiveness and innovation, as it primarily follows existing methods to assess current models. Additionally, video-to-action approaches are not universally applicable across different domains.", "other_comments_or_suggestions": "1. How to use the human feedback annotations for further improvement？\n2. Line84-85, duplicate “output modality”\n3. Line1238, cite PVDM", "questions_for_authors": "1. This paper evaluates multiple video generation methods. What are the key conclusions and insights? How can the findings guide improvements in model design and methodology?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark for "World Simulation", essentially, applying text-to-video models to embodied AI tasks. This is becoming an increasingly popular application so it seems sensible to have strong benchmarks to test capabilities here.

The authors introduce three problem settings, Minecraft, driving and robotic manipulation, and conduct a series of evaluations in each. In general there are some issues with respect to these being somewhat disjoint tasks, but the diversity could also be seen as a strength. In the rebuttal, the authors included the recent Cosmos work which makes the paper stronger given its recent prominence at this exact task.

Side note: The authors incorrectly cite Ha and Schmidhuber for this - who worked on "World Models" as opposed to "World Simulators". The prior has a specific meaning, predicting action conditioned dynamics. The latter is largely a marketing term from Sora which is not really defined.
Decision: Accept (poster)