WorldSimBench: Towards Video Generation Models as World Simulators
Authors: name: Yiran Qin; name: Zhelun Shi; name: Jiwen Yu; name: Xijun Wang; name: Enshen Zhou; name: Lijun Li; name: Zhenfei Yin; name: Xihui Liu; name: Lu Sheng; name: Jing Shao; name: Lei Bai; name: Wanli Ouyang; name: Ruimao Zhang
Published: 2024-10-23 17:56:11

Abstract: Recent advancements in predictive models have demonstrated exceptional
capabilities in predicting the future state of objects and scenes. However, the
lack of categorization based on inherent characteristics continues to hinder
the progress of predictive model development. Additionally, existing benchmarks
are unable to effectively evaluate higher-capability, highly embodied
predictive models from an embodied perspective. In this work, we classify the
functionalities of predictive models into a hierarchy and take the first step
in evaluating World Simulators by proposing a dual evaluation framework called
WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and
Implicit Manipulative Evaluation, encompassing human preference assessments
from the visual perspective and action-level evaluations in embodied tasks,
covering three representative embodied scenarios: Open-Ended Embodied
Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit
Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment
dataset based on fine-grained human feedback, which we use to train a Human
Preference Evaluator that aligns with human perception and explicitly assesses
the visual fidelity of World Simulators. In the Implicit Manipulative
Evaluation, we assess the video-action consistency of World Simulators by
evaluating whether the generated situation-aware video can be accurately
translated into the correct control signals in dynamic environments. Our
comprehensive evaluation offers key insights that can drive further innovation
in video generation models, positioning World Simulators as a pivotal
advancement toward embodied artificial intelligence.