OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling
Authors: Yang Zhou
Yifan Wang
Jianjun Zhou
Wenzheng Chang
Haoyu Guo
Zizun Li
Kaijing Ma
Xinyue Li
Yating Wang
Haoyi Zhu
Mingyu Liu
Dingning Liu
Jiange Yang
Zhoujie Fu
Junyi Chen
Chunhua Shen
Jiangmiao Pang
Kaipeng Zhang
Tong He
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-Domain
Multi-Modal
World Modeling
Primary area: datasets and benchmarks

Abstract: The field of 4D world modeling—aiming to jointly capture spatial geometry and temporal dynamics—has witnessed  remarkable progress in recent years, driven by advances in large-scale generative models and multimodal learning.  However, the development of truly general 4D world models remains fundamentally constrained by the availability  of high-quality data. Existing datasets and benchmarks often lack the dynamic complexity, multi-domain diversity,  and spatial-temporal annotations required to support key tasks such as 4D geometric reconstruction, future  prediction, and camera-controlled video generation. To address this gap, we introduce OmniWorld, a large-scale,  multi-domain, multi-modal dataset specifically designed for 4D world modeling. OmniWorld consists of a newly  collected OmniWorld-Game dataset and several curated public datasets spanning diverse domains. Compared with  existing synthetic datasets, OmniWorld-Game provides richer modality coverage, larger scale, and more realistic  dynamic interactions. Based on this dataset, we establish a challenging benchmark that exposes the limitations of  current state-of-the-art (SOTA) approaches in modeling complex 4D environments. Moreover, fine-tuning existing  SOTA methods on OmniWorld leads to significant performance gains across 4D reconstruction and video generation  tasks, strongly validating OmniWorld as a powerful resource for training and evaluation. We envision OmniWorld  as a catalyst for accelerating the development of general-purpose 4D world models, ultimately advancing machines’  holistic understanding of the physical world.

Published as a conference paper at ICLR 2026
OmniWorld: A MULTI-DOMAIN AND MULTI-MODALL DATASET FOR 4D WORLD MODELING
Yang Zhou Yifan Wang? Jianjun Zhou Wenzheng Chang Haoyu Guo Zizun L
Kaijing Ma Xinyue Li Yating Wang?? Haoyi Zhu Mingyu Liu Dingning Liu 1.2 Jiange Yang2.
Zhoujie Fu Junyi Chen 2.3 Chunhua Shen + Jiangmiao Pang Kaipeng Zhang? Tong He2.5.
1 Fudan University Shanghai AI Laboratory SShanghai Jiao Tong University
*Zhejiang University
Shanghai Innovation Institute University of Science and Technology of China
Tongji University
8 NNanjing University
Nanyang Technological University Corresponding Author
OMNI WORLD
MULTI DOMAIN
MULTI- MODAL
300M FRAMES
Covvnnman Mastam vattng
Figure 1: We introduce Omni World, a large-scale, multi-domain, and modai dataset. Omni- World provides rich resource for 4D world modeling by integrating high-quality data from multiple domains and offers a variety of data types, including depth maps, camera poses, text captions, op- tical flow and foreground masks. OmniWorld is designed to accelerate the development of more general models for modeling the real physical world.
ABSTRACT
The field of 4D world modeling-ainee to jointly capture spatial geometry and temporal dynamics has witnessed remarkable progress in recent years, driven by advances in large-scale generative models and multimodal learnin.. However, the development of truly general 4D world models remains fundamentally con- strained by the availability of high-quality data. Existing datasets and benchmarks often lack the dynamic complexity, muliidomain diversity, and spatial-tempott annotations required to support key tasks such as 4D geometric reconstruction, fu- ture prediction, and camera- camera-controlled video generation. To address this gap, we introduce OmniWorld, large-scale, multi-domain, multi- modti dataset specif- ically designed for 4D world modeling. OmmiWorld consists of newly col- lected Omni Homiiorrd--Gar dataset and several curated public datasets spanning di- verse domains. Compared with existing synthetic datasets, AmmiWorld-Gain provides richer modality coverage. larger scale, and more realistic dynamic in- teractions. Based on this dataset, we establish a challenging benchmark that exposes the limitations of current state-of-the-art (SOTA) approaches in model- ing complex 4D environments. Moreover, fine-tuning existing SOTA methods on OmniWorld leads to significant performance gains across 4D reconstruction and video generation tasks, strongly validating OmniWorld source for training and evaluation. We envision OmniWorld as a catalyst for ac- celerating the development of general-purpose 4D world models, ultimately ad- vancing machines' holistic understanding of the physical world. Project Page: httpss://yangzhou2// .t..ii......ttt thub.. 10/0mn World/
powerful re-

---

Published as a conference paper at ICLR 2026
Table I: Comparisons between AmniWWrlddGaam and existing synthetic datasets. OmniWorld- Game surpasses existing public synthetic datasets in modal diversity and data scale.
Type
Reselution
Depth
Duta modalit
Optical
Fp. masks
MPI Sintel
Dynamic
1024 436
RyingThinaosz (Mayer
Horley
Cuakvor
$40
TaranAir Wap 2VA0
Dynamic
BlendaM Y2000
Mixed
Staie
17K
HyperSim Rabeen 2
Indace
Sutie
77K
Dynamis Radier
Indrcr
Dynsmic
1280 720
169K
Sycing MMhl al 2043
Mixed
Dynamic
1920 1990
23K
EDEN L2 2 MMM1
Oukloor
Statie
6411 x 480
300K
6/
Pointklpsser Ziinired
2023
Mixed
Dyussic
9611 x 540
216K
6/
S-Kai-Came
[Frera:: 2005
Oukkoor
Dymeiic
1920 x 1080
4.320K
World Gante (Dlrs)
(Dur))
Mixed
Dynanic
1280 x 720
18,515K
INTRODUCTION
The development of world models (DeepMind, 2025 Ha & Schmidhuber 2018 Agarwal et al,
2025 LeCun 2022, Hafner et al. (023) has become central pursuit in visual intelligence systems, aiming to build systems that can simulate and reason about the physical world. This capability goes beyond simple static perception, demanding models that can simulate dynamic environments, predict object motion, infer causality, and generate content that adheres to physical laws. Such spatio-temporal modeling is a cornerstone for effective world models, with development critically
dependent large-scale, multi-domain and multi-modal datasets (Feng al. 2024: Team et al.
2025a Chen et al. 2025 He et al. 2025b; Team et al. 2025b Yu et al. 2025b a)
Two fundamental tasks that reflect a model's world modeling capability have drawn widespread attention: 3D geometric foundation models (Wang et al.
2024c
Leroy et al.]
2024
Zhang et al.
2024 Yang et al. 2025a Tang al. 2024 Wang et al.
2025b
Zhang et al. 2025 Wang et al.
2025a d), and camera-controlled video generation models
Wang et al. 2024d|
He et al. 2024
Zheng et 2024 Bahmani al., 2024 Bai et al. 2025 YU et al.] 2025). The former aims to
extract comprehensive 3D geometric information from 2D image inputs, while the latter focuses on generating dynamic video content that follows precise spatio-temporal instructions. Both tasks heavily rely on large-scale, high-quality datasets with rich modalities, including RGB images, depth maps, and camera poses.
However, existing benchmarks and datasets for evaluating and training these models have signifi- cant limitations. In the domain of 3D geometric foundation models, existing benchmarks suffer from short sequence lengths, which constrain the evaluation of model's 's long-term robustness. For ex- ample, Sintel (Butler et al. 2012), which is a widely used dataset, consists of videos with an average length of only 50 frames. Furthermore, the limited motion amplitude and single-action types within these datasets (e.g.. Bonn's (Palazzolo et al. 2019) focuses on indoor human motion, Kitti's (Geiger et al. 2013) focuses on outdoor street scenes) fail to comprehensively evaluate model performance in complex, dynamic environments. Similarly, in the field of camera-conee camera-controlled video generation, mainstream datasets like RealEstate 1OK (Zhou al.) (018)) primarily consist of static scenes with smooth camera trajectories. This lack of diverse object motion and complex camera operations re- sults in noticeable gap between the dataset's content and real-world scenarios, thereby hindering comprchensive assessment of a model's 's true capabilities.
From the perspective of training data, there is critical scarcity high-quality, multi-domain, multi- modal datasets that include rich geometric annotations. For instance, in image or video generation,
while there are numerous image-text (Schuhmann et al., 2022 Gadre et al., 2023) or video-text datasets (Chen et al., 22244] Nan et al.) 2024] Ju et al.] (2024). they often lack critical geometric modalities such as depth maps, camera poses, and optical flow. Similarly, the demand for large-
scale, diverse datasets with accurate geometric annotations is increasingly urgent for 3D geometric foundation models.
To address these shortcomings. we introduce OmniWWrld, a large-scale, multi-domain, and multi- modal dataset composed self-collected high-quality OmniWorld-Game synthetic dataset and several public datasets. Its core characteristies are: 1) High-Quality 4D Data. OmniWorld- is a massive synthetic video dataset comprising over 96K clips and more than 18M frames, with a total duration of over 214 hours. It is captured from diverse game environments with 720P RGB images, dense ground truth depth maps, accurate camera poses, and annotations for text captions, optical flow and foreground masks. As shown in Tab. I the dataset significantly surpasses existing
2

---

Published as a conference paper at ICLR 2026
Table 2: Omni structure. smiling face (0) indicates the modality newly (re- Jannotated by us, green check (V) denotes ground-truth data that already exists in the original dataset, and red cross (X) marks missing modalities.
Data modality
Dataset
Domain
# Seq.
FPS
Resolution # Frames
Depth Camera
Text Opt. flow Fg. masks
OmniWorld-Gomm Gome
Simulator
96K
24
1280xx20
18,515K
AgiBot Bu el 2025
Robot
20K
30
640x480
39,247K
DROID Khazalsky cl
2024
Robot
35K
60
1280x720
26,643K
RH20T Fang al. 2024
Rohot
109K
10
640x360
53,453K
X
RH20T-
Human (rang el
2024
Humun
10
640x360
8,875K
HO14D
et
Human
15
1920x1080
891K
Epic-Kitchent DDamnn al.
2018
Human
15K
30
1280x720
3,635K
X
X X AXX
Ego-Exo4D Grauman et
2024
Human
4K
30
1024x1024
9,190K
X
HoloAssist Wang ct 2023
Human
1K
30
896x504
13,037K
X
Assembly101 Sener ct al. 2022
Human
60
1920x1080
110,831K
XXX
EgoDex Hoque 2025
242K
1920x1080
76,611K
CityWalk Ll
Internet
30
1280x720
13,096K
X
public synthetic datasets in modal diversity and scale. 2) Multi-Domain Coverage. By integrating datasets from four key domains including simulator, robot, human, and the internet, OmniWorld covers a wide range of real-world and virtual scenarios, greatly enhancing data diversity. 3) Multi- Modality Annotations. OmniWorld provides rich suite of multi-modal annotations, crucial for detailed world modeling, as shown in Tab.
Based on OmniWorld- we propose a new benchmark for both 3D geometric foundation models and camera-controlle, video generation models. Our OmniWorld-Gamm benchmark pro- vides challenging, complex scenarios and dynamics that accurately reflect model's true world capabilities, revealing the limitations of current SOTAs. By fine-tuning existing SOTAs (e.g., DUSt3R (Wang et al. (024c), CUT3R (Wang et al. 2025b). Reloc3r (Dong et al.) (0024), AC3D (Bahmani et al. 2024)) with OmniWorld, we demonstrate significant performance improve- ments on public benchmarks. This strongly validates Onnll as powerful training resource for enhancing world modeling capabilities.
In summary, our contributions are as follows:
1. We introduce OmniWorld, a multi-domain and multi-modal dataset designed to address the lack of diversity in existing datasets. Its self-collected subset, OmniWorld Gamme, surpasses current synthetic datasets both modality diversity and data volume.
2. We establish comprehensive benchmark for 3D geometric foundation models and camera- controlled video generation models based on Omni Worli SnnnWWrld-Gaaee Gamee, providing a unified plat- form for evaluation.
3. We fine-tune several SOTAs on OmniWorld and observe significant performance gains, underscoring its value as training resource.
OmniWorld DATASET
2. DATA ACQUISITION
Our data acquisition strategy is centered on our novel, self-collected OmniWorld-Grr dataset, which strategically supplemented with curated data from three other distinct domains: robot, human, and internet, as illustrated in Fig. 2 This strategy allows to integrate the strengths of diverse data sources to comprehensively capture weal-world complexity.
Simulator domain. To acquire the high-precision and temporally consistent multi-modal data that hard to obtain the real world, we collect Omni Hamiiorld-Gaam from game environments. Follow- ing prior works (Richter et al. 20166; Yang et al. 2024a; Feng et al. 2024 Team et al.) 2025a), we utilize ReShade (ReShade Contributors (024) to access depth information during the rendering process, and simultaneously capture synchromized RGB images from the sereen using OBS (Con- tributors] 2024). This approach offers significant advantages: 1) High-Precision Modal Data. We precisely control the environment and acquire accurate depth data, which is often unattainable
3

---

Published as a conference paper at ICLR 2026
Data Collection
Video Slicing
Text Captions
Depthhipa
Depth Maps
CHANA
PasoPue
Camera Poses
Insufliciet feabures
M SAM
Mask
aulti-Domain
Human
Internet
Data
APPlow
Optical Flow
DepthPipe
Course
RuShade
Drgais Rays
Prler
DroidCatib
CoTracker
cloed
Foundation
Starmo
OmniWorld acquisition and annotation pipeline. We collect raw data from diverse domains and apply a video slicing filter to obtain high-quality RGB sequences. These sequences are then processed through a suite of specialized pipelines to generate multi-modal annotations, including text captions, depth maps, camera poses, foreground masks, and optical flow.
Figure 2:
real-world settings. 2) Rich Real-Worke Scene Simulation. Modern virtual environments provide highly realistic graphies and diverse simulations of real-world scenarios, such complex settings from wilderness to urban areas.
Robot domain. We integrate public datasets from robot manipulation and human-robot interaction tasks, including AgiBot (Bu et al. 2025), DROID [Khazatsky et al., 2024), and RH20T (Fang et al., 2024 These datasets provide valuable sequences of rohot-ennironment interactions and navigation, which are essential for tasks involving robotic manipulation and physical world understanding.
Human domain. We incorporate public datasets describing various human activities, including (Damen et al.] 2018). Ego-Exo4D (Grauman al.) 2024), HoloAssist (Wang al. 2023), Assembly101 (Sener al.,
RH20T-Human (Fang et al., 2024). HO14D (Liu et al. (022). ,
2022]. and EgoDex (Hoque et al. [2025). These datasets capture diverse human behaviors, ranging from daily activities complex assembly tasks, from both egocentric and exocentric perspectives.
Internet domain. To acquire large-scale, realistic, and diverse in-the-wild scene data, we utilize the CityWalk dataset (Li et al. 2025). It offers rich real-world street view videos from the internet.
To prepare the raw data, we perform video slicing to ensure high quality and temporal coherence. This crucial preprocessing step filters out unsuitable frames (e.g., those with motion blur insuf- ficient features) and segments long recordings into shorter, manageable clips. The resulting high- quality video segments are then passed our multi-modal annotation pipeline.
2.2 DATA ANNOTATION
We primarily annotate the following key modalities: depth maps, camera poses, text captions, optical flow, and foreground masks (see Fig. 2 for the overall pipeline). Here we briefly introduce the annotation method of each modality.
Depth maps. Accurate depth information is paramount for geometric modeling. To ensure the quality and consistency of depth maps, we adopt a tailored approach based on the data source. For the self-collected dataset OmniWorld-Game, information during the rendering process using tools like ReShade (ReShade Contributors 2024).
mentioned in Sec. 22. directly access depth
For public datasets like AgiBot (Bu et al. 2025) and HOI4D (Liu et al. 2022), which often provide noisy and sparse raw depth maps, we employ Prior Depth Anything (Wang et al., 2025e) to robustly optimize them, yielding denser and more reliable depth annotations. For the public stereo dataset DROID (Khazatsky et al. 2024), we leverage FoundationStereo (Wen et al., 2025) for stereo depth estimation this dataset.

---

Published as a conference paper at ICLR 2026
Mind
Ourdion
Eye- Eowat
Holekeziat
Assembby100
Butkten
Velicee
Urian
Gurdser
Spickinchens
Natorrll
H0240
Human
Ege
Deninant
Indiorr
Percentage (%)
JHuman
Object
cnywok
intermet
Sieelater
Futoriticc
Hatarical
Perspective
Cameng
WenWeold
Modern
First-
RAROTT
Avcian
Agiet
DROTO
Person
500
150
200
250
(a) OmniWorld Compositional Distribution
(b) OmniWorld-Ganm Internal
Composition
(c) Caption Tokens Distribution
Figure 3: Statistical information OmniWWrld. (a) displays compositional distribution of data from different domains within OmniWorld, (b) presents internal composition of (c) shows caption tokens distribution of OmniWorld.
Foreground masks. To provide precise, temporally consistent masks of primary subjects, de- velop specialized automated pipelines. For robot domain data, we use RoboEngine (Yuan et al.) 2025) to generate initial masks for keyframes, followed by temporal tracking and fusion with SAM (Ravi et al. 2024). For Omni WorldWooldGaaadd (e.g.. player characters in third-person view), we lever- age Grounding DINO (Liu al.. 2023) detect initial bounding boxes within predefined regions of keyframes, which then serve as prompts for SAM (Kirillov et al. (023). These generated masks can be used dynamic foreground masks to guide camera pose estimation.
Camera poses. Accurate camera pose annotation in dynamic videos is highly challenging due to transitions, weakly textured areas, and abrupt movements that hinder traditional Structure-from- Motion methods (Rockwell et al. 2025 Li et al., 2024). Following prior work (Team et al. (022a)) we develop robust, automated, two-stage pipeline for dynamic camera pose annotation, whose principles are validated across diverse data types.
The pipeline leverages the pre-computed foreground masks to focus on static background regions. The stages include: 1) Coarse camera pose estimation leveraging VGGT (Wang et al. 2025a) for videos without depth or DroidCalib (Hagemann al., (023) with depth constraints; 2) Camera pose refinement through dense point tracking (SIFT (Lowe (2004), SuperPoint (DeTone al. 2018) with CoTracker3 (Karaev al., 20244) on static regions and subsequent bundle adjustment to min- imize reprojection errors, optionally enhanced by forward-backwer reprojection with depth infor- mation (Chen et al. 2019).
Text captions. We generate text descriptions using semi-automated approach centered the Dwen2-VL-72B-Insteee model (Wang et al. 2024a). We employ domain-specine prompting strate- gies for each 81-frame video segment. For instance, in the Omni OmmiWorld--F Gamee domain, generate multi-faceted deseriptions covering different viewpoints (e.g., first- and third- soird-person,, character ac- tions, background details, and camera movements.
Optical flow. We generate optical flow annotations using DPFlow (Morimitsu et al. 2025) to capture dense, pixel-level motion. Unlike models that require downsampling high-resolution inputs (Teed & Deng 2020), DPFlow processes videos at their original resolution. This makes it ideal for our high resolutions dataset.
2.3 DATA STATISTICS
OmniWorld is large-scale dataset composed of 12 distinct datasets from four domains: simulators, robots, humans, and the internet (see Tab. 2 for a summary). It contains over 600K video sequences and 300M frames with high resolutions. The dataset is richly annotated with multiple modalities, including depth, camera poses, text, optical flow, and foreground masks.
As shown in Fig. 3a the human domain constitutes the largest portion of OmniWorld, highlighting its focus on roal-world activities. Our self-collected Omni OmniWorld subset is particularly diverse, detailed in Fig. 3b It spans various scene types (e.g., outdoor-urban, indoor), camera perspectives
5

---

Published as a conference paper at ICLR 2026
Table 3: Monocular depth & video depth estimation on OmmiWorld-Gane benchmark.
Mono-Depth
Video-Depth
Method
scale
scale
scale&shift
FPS
Abs Rel +
8.2.25
Abs Rel
6<1,225
Abs Rel 4
8<1.25
DUSL3R
Wang et al. 2024c
0.742
0.460
0.709
0.447
0.379
0.560
0.96
MASI3R
Leroy et al. 2024
0.485
0.560
0.482
0.579
0.217
0.724
0.79
MonST3R
(Zhang cf al. 2024
0.670
0,493
0.669
0,505
0.272
0.648
0.95
Fast3R
Yang et al.
2025a
0.755
0,404
0.741
0.384
0,464
0.531
14.99
CUT3R
Wang ct al.
2025b
0.624
0,518
0.690
0.479
0,429
0.603
10.75
FLARE
Zhang ct al. 2025
0.664
0.475
0.757
0.453
0.511
0.527
4.24
VGGT
Wang et al.
2025a
0.531
0.554
0.440
0.625
0.194
0.755
18.75
MoGe-1
Wang
2024b
0.459
0.586
MoGe-2
Wang
20256
0.401
0.589
(first-person and hhird-person), historical eras (ancient to futuristic), and dominant objects (natural terrain, architecture, vehicles). This culti-dimennsooall diversity ensures the data is both challenging and comprehensive.
Furthermore, OmniWWrll features structured and detailed text annotations. The captions typically range from 150 to 250 tokens (Fig.Be), density that significantly surpasses other harge-scale video- text datasets like Open Vid-IM (Nan et al. (2044) and Panda-70M (Chen et al. 2024).
3 OmniWorld-Game BENCHMARK
3.1 3D GEOMETRIC PREDICTION BENCHMARK
Benchmark design. Existing benchmarks for 3D Geometric Foundation Models (GFMs) often feature short sequences and limited motion dynamics. For example, Sintel (Butler et al. 2012 quences average only 50 frames, while datasets like Bonn (Palazzolo et al.) 2019) and KITTI (Geiger et al. 2013) confined to specific scenarios (e.g.. indoor human motion outdoor street views). These limitations hinder the comprehensive evaluation of a model's long-term and complex-scene modeling capabilities. To address this, our WeiiWWorr-Gmmt benchmark provides more challeng- ing testbed featuring extended, high-resolution sequences (up to 384 frames at 720P) with diverse and complex motions.
Evaluation details. We evaluate a suite of recent GFMs, including DUSt3R (Wang et al.) 2024c MAS13R Leroy et al., 2024). MonST3R (Zhang al. (024). Fast3R (Yang al. 2025a).
CUT3R WWag et al. 2025b). FLARE (Zhang et al. 2025), VGGT (Wang et al.) (025a), and MoGe (Wang al.J 2024b] 2025c). The evaluation is conducted on two tasks: monocular depth
estimation and video depth estimation.
Analysis. Our evaluation on OmniWorld-Game reveals significant challenges for current GFMs (Tab 3) For monocular depth estimation, MoGe-2 (Wang et al.) 2025c) achieves the best quan- titative results. This finding is supported by qualitative results in the supplementary materials, where it produces visibly sharper depth maps. In the more demanding video depth estimation task, VGGT (Wang et al. (2025)) demonstrates superior accuracy and efficiency (FPS). Point cloud visual- izations (Fig.4) (4) confirm that VGGT generates
coherent 3D structures, yet it produces ticeable artifacts in highly dynamic scenes. No single GFM masters all tasks on OmnnWWrlee Game. The results highlight that current SOTA models still struggle with long-sequence consistency and complex dynamics, validating our benchmark as challenging testbed for advancing future research.
3.2 CAMERA-CONTROLLE VIDEO GENERATION BENCHMARK
Benchmark design. Existing benchmarks for camera-controlled video generation, such as RealEstate 10K (Zhou et al., 2018). are often limited to static scenes with smooth camera paths, failing to reflect real-world complexity. In contrast, OmniWorld Game benchmark provides more challenging evaluation environment, featuring rich dynamic content, complex camera trajec- tories, and diverse scenes.

---

Published as a conference paper at ICLR 2026
Input Imuges
Ground Truth
Fast3R
CUT3R
FLARE
VGGT
Figure 4: Qualitative comparison of multi-view 3D reconstruction on OmmiWorld-Game bench- mark.
Table 4: Camera-controlled video generation evaluation on OmniWorld-Ganm benchmark.
FVD
Method
TransErr
RotErr
CamMCI
VideoGPTI
StyleGAN
AC3D (T2V) (aahmani al.
2024
6.2788
0.8867
6.6965
1745.778
1594.885
MotionCull (12V) Wang et al.
2024d
7.8633
1.1402
8,2710
694.342
745.652
CamCtrl (12V) (e et al. 2024
1.2882
0.2022
1.3856
615.417
637.574
CAMI2V (12V) Zheng of al.
2024
5.9626
0,5087
6.2010
837.185
742.594
We evaluate several recent models, including the Text-to- Video model AC3D (Bahmani et al. 2024) and Image-to- Video models like CamCtrl (He et al. (024), MotionC- trl (Wang et (022dd) CAMI2V (Zheng et al., 2024). Following prior work (Zheng et al. 2024))
Evaluation details.
we assess performance using two sets of metrics: camera-controlled accuracy (RotError, TransError, CamMC) and perceptual quality (Fréchet Video Distance, FVD) (Unterthiner et al.) 2018).
Analysis. Our analysis on OmniWorld Game reveals current models failing to achieve either high Text-to-Video
generation quality precise camera control. For instance, the Text-to- Video model AC3D (Bahmani jtt al. 2024) generates subtle dynamics and fails to follow camera paths, resulting in poor quantitative and qualitative scores (Tab. 4 Fig. 5). Among Image-to- Amage-to-Video (I2V) models, CamCtrl (He al.,
2024) shows better quantitative performance. However, its generated videos often suffer from blurry moving characters shown Fig. 5 Other methods, including MotionCtrl (Wang et al. 2024d) and CAMI2V (Zheng et al. 2024). ) face similar quality degradation issues. These results underscore
the unique challenges posed by our benchmark in evaluating spatio-temporal generation capabilities.
4 MODEL FINE-TUNING AND EFFICACY VALIDATION
4.1 IMPROVING 3D GEOMETRIC PREDICTION WITH OmniWorld
To demonstrate OmniWorld' value as a training resource, fine-tune three baseline models, DUSt3R (Wang et al.) 2024c), CUT3R (Wang et al. 2025b). and Reloc3r (Dong et al. (024). on subsets of our dataset. The fine-tuned models consistently surpass their original performance across monocular depth estimation (Tab. 5), video depth estimation (Tab. (6). and camera pose estimation (see supplementary). Notably, for monocular depth, the fine-tuned DUS(3R not only improved upon its baseline but also outperformed MonST3R Zhang et al. (0024). which is fine-tuned on several ex- isting dynamic datasets. The enhancements in video depth estimation also underscore OmniWorld's effectiveness in improving temporal consistency. These results validate that OmniWorld scale and diversity provide powerful resource for boosting the generalization and robustness of 3D geometric foundation models.
7

---

Published as a conference paper at ICLR 2026
AC3D
man aut maki along tracks urion sefting statiancry und ebserving surroundngs"
Mo
Figure 5: Qualitative comparison of camera-controlled video generation on Worti-Worrraa benchmark. AC3D takes text as condition signal. MotionCtrl, CamCtrl, CAMI2V take an image as condition signal. Condition images are the first images of each row.
Table 5: Comparison of original and fine-tuned models for monocular depth estimation
Sintel (Butler et al. 2012), Bonn (Palazzolo et al.) 2019). KITTI (Geiger et al. 2013) and NYU- denotes models that have been fine-tuned Omni World.
v2 (ilberman et al.
2012
Sintel
Bonn
KITTI
NYU->2
Method
Abs Rel.
< 1,25
Abs Relj
<. 1,25
Abs Rel,
< 1.25 f
Abs Rell
5 < 1.25 f
DUSER Wang al. 2024c
0.488
0,532
0.139
0,831
0.109
0.873
0.081
0.909
MonSTaK (Zhang et nl. 2124
0.402
0.525
0.069
0.954
0.098
0.895
0.094
0.887
DUSL3R
0.370
0.529
0.067
0,948
0.088
0.932
0.089
0,902
CUT3R WWang et al. 20256
0.420
0.520
0.058
0.967
0.097
0.914
0.081
0.914
CUT3R
0.522
0.935
0.075
0,920
4.2 ENHANCING CAMERA
CAMERA-CONTROLLED CONTROLLED VIDEO GENERATION WITH OmniWorld
Existing datasets for camera-controlled video generation, such as RealEstate 10K (Zhou et al. 2018), often limited static scenes with simple camera movements, which restricts model's ability to handle dynamic content. To address this data bottleneck, we fine-tuned the AC3D (Bahmani
al. 2024) baseline on OmniWorld. This approach aligns with prior findings (He al., 2025a) that highlight the critical role of dynamic data in improving camera control. As shown in Tab. our fine-tuned model significantly outperforms the original baseline on both the RealEstate 10K and our OmniWorld-Game benchmarks. This result validates Omni World as powerful training resource for enhancing a model's capability to follow complex camera instructions dynamic environments.
5 RELATED WORK
World model dataset. The ability of models to perform world modeling is intrinsically linked to the availability of scage, high-quality spatio-temporal datasets. Static 3D datasets (Dai al.;
2017 Silberman et al.) 2012 Li & Snavely 2018) have advanced 3D reconstruction by providing precise geometric information. However, their static nature limits their utility for modeling
tion. In video generation, large-scale video-text datasets (Chen et al. 2024 Bain et al., [202]] Nan al. 2024: Ju al., (024) offer rich semantic annotations but lack geometric information (e.g., depth), making them unsuitable for 4D world modeling. To bridge this gap, researchers have cre- ated dynamic real-world datasets for autonomous driving (Geiger et al. 2013 Sun al. (020)) and human-robot interaction (Palazzolo et al., 2019; Liu et al.) 2022 Damen et al., 2018 Fang et al.) 2024). While valuable, these datasets often suffer from a lack of scene diversity and noisy geo- metric annotations. With advancements in modern rendering technology significantly reducing the sim-to-real gap (Wang et al. 2020), synthetic datasets have emerged as a valuable alternative provid- ing precise annotations. However, recent synthetic datasets (Butler et al. 2012) Mayer et al. 2016 Harley et al.) 2022; Wang et al. 2020. Karaev et al. 2023 Mehl et al., 2023) still fall short in terms of scale, diversity, and modal richness compared to HomniWorld-Gama dataset (Tab. T
8

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of original and fine-tuned models for video depth estimation on Sin- tel (Butler et al., 2012), Bonn (Palazzolo et al. 2019) and KITTI (Geiger et al. 2013 denotes models that have been fine-tuned on OmniWorld.
Sintel
Bonn
KITTI
Method
Align
Ahs Rel 4. 8< 1.25
Abs Rel 4 8< 1.225
Abs Rel 4 0< 1.25 f
DUS(3 (Wang et 2024c
0.652
0,436
0.151
0.839
0.143
0.814
scale
DUS(3R
0,512
0.456
0.083
0.920
0.135
0.800
CUT3R Wang al. CUT3R
2025b
0.417
0.510
0.078
0.937
0.123
0.875
scale
0.396
0.516
0.078
0.938
0.107
0.907
DUS(3 Wang el
2024c
0.570
0.493
0.152
0.835
0.135
0.818
scale&shift
DUSt3
0.520
0,480
0.084
0.914
0.136
0.808
CUT3R Wang al. CUT3R
2025b
0.537
0.556
0.075
0.944
0.111
0.884
scale&shift
0.314
0.574
0.067
0.964
0.103
0.912
Table 7: Comparison of original and fine-tuned models for camera-controlled video generation evalua- tion RealEstatelOKK (Zhou 2018) and OmniWorld-Gamm benchmark. * denotes models that have been fine-tuned on OmniWorld.
FVD
Method
Benchmark
TransErr
RotErr.
CamMCJ
VideoGPTI
StyleGAN,
AC3D (Bahmani et al. 2024 AC3D
3.4433
0.6308
3.6615
479.320
409.795
RealFstate
2.8648
0,5314
3.0518
472.683
416.948
AC3D Bahmani ct al. 2024
6.2788
0.8867
6.6965
1745.778
1594.885
OmniWorld-Gane
AC3D
4.1428
0.7610
4.4854
1437.247
1249.186
3D geometric foundation models. 3D geometric foundation models have recently emerged as data-driven alternative traditional methods. Early works like DUS&3 (Wang et al. 2024c) and MonST3R (Zhang et al. 2024) operate on image pairs, requiring expensive global alignment for larger scenes. Further research has introduced diverse architectural innovations to overcome this, including parallel processing (Fast3R (Yang et al.) 2025a)), decomposing the learning task (FLARE (Zhang et al. 20255), online processing for image streams (CUT3R (Wang et al. (025b)). multi-task learning (VGGT (Wang al. 2025a)), and cetuuuattioonquvvariii designs (л (Wang et al. 2025dh). However, the efficacy of these models fundamentally tied to large-scale, multi- powerful training resource that fulfills this need.
modal training data. We validate OmmiWorld
Camera-controlled video generation. Most methods in this field inject camera parameters (such as Plücker embeddings) into pre-trained video diffusion model (Blatimann al. 2023 Chen al.,
2023 Yang et al. 2024b) with representative works including MotionCtrl (Wang et al. 2024d),
CameraCtrl (He et al. 2024), CAMI2V (Zheng et al. (2024), AC3D (Bahmani al. 2024 Despite this progress, these methods still struggle to generate dynamic content with complex camera control. They are typically trained on datasets like RealEstate 10K (Zhou et al.) 2018) DL3DV-10K (Ling et al.] (202)., which consist of static scenes with smooth camera motions. This data limitation inher- ently restricts them to handle dynamic scenes (He al. 2025a). The performance gap is evident on our challenging HomniWold-GGond Game benchmark.
6 CONCLUSION
We introduce OmniWorld, large-scale, multi-domain. and multi-modal dataset designed address the critical data bottleneck for 4D world modeling. By integrating self-collected Omni World-Game dataset and several public datasets from various domains, we create comprehensive data resource for 4D world modeling. We demonstrate that OmniWorld-Game serves as a challenging benchmark for 3D geometric prediction and camera-controlled video generation, revealing the limitations of current methods. Furthermore, we provide strong evidence that fine-tuning with OmniWorke sig- nificantly boosts the performance of these models, underscoring its value as a powerful training resource. We believe that OmniWorle will serve crucial data resource for the community, accel- erating the development of more general and robust models for understanding and interacting with the real physical world.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work supported by Shanghai Artificial Intelligence Laboratory.
ETHICS STATEMENT
Our work, the OmniWorld dataset, a composite dataset consisting a newly collected game- derived dataset (OmniWorid-Game) and several curated public datasets. We have undertaken multi- faceted approach to ensure our practices for both collecting new data and curating existing data are legally compliant.
1) Responsible Acquisition of games using standard. non-invasive tools (e.g.. OBS, ReShade) without any reverse engineering or cheating. To respect the source material, we automatically remove UI elements and text, and manually filter for sensitive content such as story spoilers.
Data. Game data was captured from legally purchased
2) Adherence to Terms of Use for Game Content. Our use of game content is strictly non- commercial, aligning with publisher terms of service (e.g., Rockstar Games (Rockstar Games)
2024 The dataset is intended solely advance academic research and does not compete with or infringe upon the economic interests of the copyright holders.
3) Curation of Public Datasets. OmniWorla also incorporates public datasets to enhance domain diversity. We have strictly adhered to the original license of each dataset, ensuring proper attribution and compliance with all usage terms.
REFERENCES
Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chat- topadhyay, Yongxin Chen, Yin Cui, Yifan Ding, al. Cosmos world foundation model platform for physical ai. arXiv preprint 1rxxi:2501.03576 2025.
Sherwin Bahmani, Ivan Skorokhodov, Guocheng Qian, Aliaksandr Siarohin, Willi Menapace, An- drea Tagliasacchi, David B. Lindell, and Sergey Tulyakov. Ac3d: Analyzing and improving 3d camera control in video diffusion transformers. arXiv preprint arXiv:2411.18673 2024.
Jianhong Bai, Menghan Xia, Xiao Fu, Xintao Wang. Lianrui Mu, Jinwen Cao, Zuozhu Liu, Haoji Hu, Xiang Bai, Pengfei Wan, et al. Recammaster: Camera-controlled generative rendering from single video. arXiv preprint arXiv:2503. 1647, 2025.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes diverse real- world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fifth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round I), 2021. URL htR.ttts::// htttps:://ppererr./
net. forum? d=t. www.fooddtttiiiiaii.ooo d=t.jijv_ghhhhec..
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023.
Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Xindong He, Xu Huang, et al. Agibot world colosseo: large-scale manipulation platform for scalable and intelligent embodied systems. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025.
Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, 2012. URL https://ap1: semanticscho art.aa..r/oorpsst:::33333330
10

---

Published as a conference paper at ICLR 2026
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang. et al. Videocrafterl: Open diffusion models for high- quality video generation. arXiv preprint arXhv:2310.19512, 2023.
Junyi Chen, Haoyi Zhu, Xianglong He, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Zhoujie Fu, Jiangmiao Pang, et al. Deepverse: 4d autoregressive video generation world model. arXiv preprint arXiv:2506.011 103, 2025.
Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka. Hsiang-wer wei Chao, Ming Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming- Hssan Yang, and Sergey Tulyakov. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu. Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7063-7072, 2019.
OBS Contributors. Obs studio, 2024. URL
com/
Angela Dai, Angel X Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nicßnnr. Scannet: Richly-annotanl 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scal- ing egocentric vision: The epic-kitchens dataset. In European Conference on Computer Vision (ECCV), 2018.
Google DeepMind. Genie 3.
attpr//deepuissc.ogg/////oooooo 2025. Accessed: 2025-08-27.
Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and deseription. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 224-236, 2018.
Siyan Dong, Shuzhe Wang, Shaohui Liu, Lulu Cai, Qingnan Fan, Juho Kannala, and Yanchao Yang. Reloc3r: Large-scale training of relative camera pose regression for generalizable, fast, and accu- rate visual localization. arXiv preprint 24Xiv:2412.08377 2024.
Hao-Shu Fang, Hongjie Fang, Zhenyu Tang. Jirong Liu, Chenxi Wang, Junbo Wang. Haoyi Zhu, and Cewu Lu. Rh20t: A comprehensive robotic dataset for learning diverse skills in one-shot. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 653-660. IEEE, 2024.
Ruili Feng, Han Zhang, Zhantao Yang, Jie Xiao, Zhilei Shu, Zhiheng Liu, Andy Zheng, Yukun Huang, Yu Liu, and Hongyang Zhang. The matrix: Infmiite-hrizon world generation with real- time moving control, 2024. URL httppp/s//xxxvvoor///////
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the generation of multimodal datasets. Advances Neural Information Processing Systems, 36:27092-271 12, 2023.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The kitti dataset. International Journal of Robotics Research (IJRR), 2013.
Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyllos Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19383-19400, 2024.
David Ha and Jürgen Schmidhuter. World models. arXiv preprint arXiv: 1803. /0/22, 2(3). 2018.

---

Published as a conference paper at ICLR 2026
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04004, 2023.
Annika Hagemann, Moritz Knorr, and Christoph Stiller. Deep geometry-awarr camera self- calibration from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3438-3448, 2023.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference on Computer Vision, pp. 59-75. Springer, 2022.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint arXiv:2404.02111, 2024.
Hao He, Ceyuan Yang, Shanchuan Lin, Yinghao Xu, Meng Wei, Liangke Gui, Qi Zhao, Gordon Wetzstein, Lu Jiang, and Hongsheng Li. Cameractrl ii: Dynamic scene exploration via camera- controlled video diffusion models. arXiv preprint arXiv:2503. 20552, 2025a.
Xianglong He, Chunli Peng, Zexiang Liu, Boyang Wang, Yifan Zhang, Qi Cui, Fei Kang, Bíao Jiang, Mengyin An, Yangyang Ren, et al. Matrix-game game 2.0: An open-source, real-time, and streaming interactive world model. arXiv preprint arXiv:2508. 13009, 2025b.
Ryan Hoque. Peide Huang, David Yoon, Mouli Sivapurapu, and Jian Zhang, Egodex: Learn ing dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11700 2025.
Nan Huang, Wenzhao Zheng, Chenfeng Xu, Kurt Keutzer, Shanghang Zhang, Angjoo Kanazawa, and Qianqian Wang. Segment any motion in videos. In Proceedings of the Computer Vision and Patlern Recognition Conference (CVPR), pp. 3406-3416, June 2025.
Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng. Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions, 2024. BRLhhts:s///xxii wttp://////b////...o//
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dynamicstereo: Consistent dynamic depth from stereo videos. CVPR, 2023.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint 4rXiv:2410.11833 2024.
Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty El- lis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.1 12945, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643. 2023.
Hoang-An Le. Partha Das, Thomas Mensink, Sezer Karaoglu, and Theo Gevers. EDEN: Multi- modal Synthetic Dataset of Enclosed garDEN Scenes. In Proceedings of the IEEE/CVE Winter Conference of Applications on Computer Vision (WACV), 2021.
Yann LeCun. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1-66 62, 2022.
Vincent Leroy, Yohann Cabon, and Jerome Revaud. Grounding image matching in 3d with mast3r, 2024.
Zhen Li, Chuanhao Li, Xiaofeng Mao, Shaoheng Lin, Ming Li, Shitian Zhao, Zhaopan Xu, Xinyue Li, Yukang Feng, Jianwen Sun, Zizhen Li, Fanrui Zhang, Jiaxin Ai, Zhixiang Wang, Yuwei Wu, Tong He, Jiangmiao Pang, Yu Qiao, Yunde Jia, and Kaipeng Zhang. Sekai: A video dataset towards world exploration. arXiv preprint arXiv:2506. 15675, 2025.
12

---

Published as a conference paper at ICLR 2026
Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In Computer Vision and Pattern Recognition (CVPR), 2018.
Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast and robust struc- ture and motion from casual dynamic videos. In arxiv, 2024.
Philipp Lindenberger. Paul-Edouard Sarlin, and Marc Pollefeys. LightGlue: Local Feature Matching at Light Speed. In ICCV,C 2023.
Lu Ling, Yichen Sheng, Zhi Tu, Wentian Zhao, Cheng Xin, Kun Wan, Lantao Yu, Qianyu Guo, Zixun Yu, Yawen Lu, et al. D13dv-10k: A large-scale scene dataset for deep learning-based 3d vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22160- 226602 2024.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.0544 050499, 2023.
Yunze Liu, Yun Liu, Che Jiang, Kangbo Lyu, Weikang Wan, Hao Shen, Boqiang Liang, Zhoujie Fu, He Wang, and Li Yi. Hoi4d: A 4d egocentric dataset for category-level numan-object interac- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21013-21022, June 2022.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Nikolaus Mayer, Eddy Igg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Henrique Morimitsu, Xiaobin Zhu, Roberto M Cesar, Xiangyang Ji, and Xu-Cheng Yin. Dpflow: Adaptive optical flow estimation with dual-pyramid framework. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17810-17820, 2025.
Kepan Nan, Rui Xie, Penghao Zhou, Tichan Fan, Zhenheng Yang, Zhijic Chen, Xiang Li, Jian Yang. and Ying Tai. Openvid- 1 m: A large-scale high-quality dataset for text-to-video generation. arXiv preprint arXiv:2407.023:
2024.
E. Palazzolo, J. Behley, P. Lottes, P. Giguère, and C. Stachniss. ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residuals. arXiv, 2019. URL https:
//arxiv.org abs 905 02082
Nikhila Ravi, Valentin Gabeur, Yuan-Ting -ing Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Va- sudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Fe- ichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 2024. URLpttps::////// aww.a. abs/ 2408. 00714
Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10901-10911, 2021.
ReShade Contributors. ReShade, 2024. URL Bollhttps://reshadde .ee.adeeeeee/eeeee..//.....
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference computer vision, pp. (02-118. Springer, 2016.
13

---

Published as a conference paper at ICLR 2026
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10912 10912-10922, 2021.
Rockstar Games. Policy on Posting Copyrighted Rockstar Games Material, 2024. URL hetps: httppppttooooaaaaaaa.ss..ssorrtr
Chris Rockwell, Joseph Tung, Tsung- Yi Lin, Ming-Yu Yu Liu, David F Fouhey, and Chen-Hsuan Lin. Dynamic camera poses and where to find them. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12444-12455 2025.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models, 2022. URL
https://arae //t.//aaavvv.//aa///// org/ abs 2210. 08402
Sener, D. Chatterjee, D. Shelepov, K. He, D. Singhania, R. Wang, and A. Yao. Assembly A large-scale multi-view video dataset for understanding procedural activities. CVPR, 2022.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In European conference on computer vision, pp. 746-760. Springer, 2012.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. bench- mark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 573-580. IEEE, 2012.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for tonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pp. 2446-2454, 2020.
Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds. arXiv preprint arXiv:2412.0697. 2024.
Aether Team, Haoyi Zhu, Yifan Wang, Jianjun Zhou, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Chunhua Shen, Jiangmiao Pang. and Tong He. Aether: Geometric-aware unified world modeling. arXiv preprint arXiv:2503. /tt.::5503./8945.... 2025a.
Hunyuan World Team, Zhenwei Wang. Yuhao Liu, Junta Wu, Zixiao Gu, Haoyuan Wang, Xuhui Zuo, Tianyu Huang, Wenhuan Li, Sheng Zhang, et Hunyuanworld 1.0: Generating immersive, explorable, and interactive 3d worlds from words or pixels. arXiv preprint arXiv:2507.21809 2025b.
Zachary Teed and Jia Deng- Raft: Recurrent all-pairs field transforms for optical flow. In European conference on computer vision, pp. 402-419. Springer, 2020.
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv: 18/2.0111 2018
Jannyua Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2025a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing Dision-languee model's perception of the world any resolution. arXiv preprint arXiv:2409.12199 2024a.
14

---

Published as a conference paper at ICLR 2026
Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Eftos, and Angjoo Kanazawa. Con- tinuous 3d perception model with persistent state. arXiv preprint arXiv:2501.12387 2025b.
Ruicheng Wang. Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for domain images with optimal training supervision, 2024b. URL https: http:p/://axxvvoor/////..
Ruicheng Wang, Sicheng Xu, Yue Dong, Yu Deng, Jianfeng Xiang, Zelong Lv, Guangzhong Sun, Xin Tong, and Jiaolong Yang. Moge-2: Accurate monocular geometry with metric scale and sharp details, 2025c. URL batps://arxxyy: www.xiiw..gg/a/////oaaaaa
02546
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In CVPR, 2024c.
Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. Tartanair: A dataset to push the limits of visual slam. In 2020 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 4909- 4916. IEEE, 2020.
Xin Wang, Tacin Kwon, Mahdi Rad, Bowen Pan, Ishani Chakraborty, Sean Andrist, Dan Bohus, Ashley Feniello, Bugra Tekin, Felipe Vicira Frujeri, Neel Joshi, and Mare Pollefeys. Holoassist: egocentric human interaction dataset for interactive ai assistants in the real world. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20270-20281, October 2023.
Yifan Wang, Jianjun Zhou, Haoyi Zhu, Wenzheng Chang, Yang Zhou, Zizun Li, Junyi Chen, Jiang- miao Pang. Chunhua Shen, and Tong He. 33. Scalable permutation-equivariae visual geometry learning, 2025d. URL
13377
Zehan Wang, Siyu Chen, Lihe Yang. Jialei Wang, Ziang Zhang, Hengshuang Zhao, and Zhou Zhao Depth anything with any prior, 2025e. [RLlhttss://arxxtt www.:oorgaab///0/0oo/ 25w..
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024d.
Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foun- dationstereo: Zero-shot stereo matching. CVPR, 2025.
Hongchi Xia, Yang Fu, Sifci Liu, and Xiaolong Wang. Rgbd objects in the wild: Scaling real-world 3d object learning from rgb-d videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22378-22389, 2024.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint 2rXiv:2310.121900 2023.
Honghui Yang, Di Huang, Wei Yin, Chunhua Shen, Haifeng Liu, Xiaofei He, Binbin Lin, Wanli Ouyang, and Tong He. Depth any video with scalable synthetic data. arXiv preprint arXiv:2410. 10815, 2024a.
Janning Yang, Alexander Sax, Kevin J. Liang, Mikael Henaff, Hao Tang, Ang Cao. Joyce Chai, Franziska Meier, and Matt Feiszli. Fast3r: Towards 3d reconstruction of 1000+ images in one forward pass. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025a.
Rujia Yang. Geng Chen, Chuan Wen, and Yang Gao. Fp3: A 3d foundation policy for robotic manipulation. arXiv preprint arXiv:2503.08950, 2025b.
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. arXiv preprint 2xXiv:2408.06072 2024b.
15

---

Published as a conference paper at ICLR 2026
Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren, Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A large-scale dataset for generalized multi-view stereo networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1790-1799, 2020.
Mao Ye, Peifeng Yin, Wang-Chien Lee, and Dik-Lun Lee. Exploiting geographical influence for collaborative point-of-interest recommendation. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pp. 325-334, 2011.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: high-
fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12-22, 2023.
Jiwen Yu, Jianhong Bai, Yiran Qin, Quande Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Xi- hui Liu. Context memory: Scene-consistent interactive long video generation with memory retrieval. arXiv preprint urXiv:2506.03147 2025a
Jiwen Yu, Yiran Qin, Xintao Wang, Pengfei Wan, Di Zhang, and Xihui Liu. Gamefactory: Creating new games with generative interactive videos, 2025b.
Mark YU, Wenbo Hu, Jinbo Xing, and Ying Shan. Trajectorycrafter: Redirecting camera trajectory for monocular videos via diffusion models. In ICCV, 2025.
Chengbo Yuan, Suraj Joshi, Shaoting Zhu, Hang Su, Hang Zhao, and Yang Gao. Roboengine: Plug- and-play robot data augmentation with semantic robot segmentation and background generation. arXiv preprint arXiv:2503./877 2025.
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming- Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arxiv:2410.03825, 2024.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Shangzhan Zhang, Jianyuan Wang, Yinghao Xu, Nan Xue, Christian Rupprecht, Xiaowei Zhou, Yu- jun Shen, and Gordon Weizstein. Flare: Feed-forward geometry, appearance and camera estima- tion from uncalibrated sparse views, 2025. URL attpp::/aaxi...or///aa/..
Guangcong Zheng, Teng Li, Rui Jiang, Yehao Lu, Tao Wu, and Xi Li. Cami2v: Camera-controlled image-to-video diffusion model. arXiv preprint arXiv:241O. 15957, 2024.
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018.
16

---

Published as a conference paper at ICLR 2026
A OVERVIEW
This appendix provides comprehensive supplementary materials, organized follows:
Dataset & Benchmarks: Sec. B details the OmmiWorld dataset construction. Extended
details on our benchmark and fine-tuning experiments are provided Sec. C and Sec. D
supplemented by comprehensive benchmark statistics in Sec.
Data Processing & Validation: Sec. details our strategy for filtering high-quality video clips. Sec. Sec.GG and See. | provide qualitative and quantitative validations of our annotation pipelines, respectively.
Analysis: Sec. limitations.
presents comprchensive failure case analysis to discuss the current
B
OmniWorld DATASET
Nature & Outdoors
Tourist Attractions
Parks and Recreation
Hotels and Accommodations
Primary POI Category
Religious Institutions
Transportation Hubs
Restaurants and Cafes
Automotive Services
Education Institutions
Shopping Centers
Financial Services
Tech & Business
Medical Facilities
Sports & Fitness
Government Civic Services
Events & Conferences
0%
4%
8%
12%
16%
56%
Scene Percentage (%)
Figure 6: The OmniWorld-Game distribution of scene category (the primary POI locations).
To quantitatively analyze the scene diversity of OmniWorld- we adopt the methodology from DL3DV (Ling et al. 2024) to classify and count scenes across 16 Point-of-Interest (POI) cate- gories (Ye et (011). The statistical results are shown in Fig. OmniWorld- Game encompasses wide variety of scene categories, including "Nature & Outdoors" "Tourist Attractions". "Parks and Recreation". and "Hotels and Accommodations" "Nature & Outdoors" represents the largest share, reflecting its dominant presence in the dataset. The distribution of these scene categories aligns with their prevalence in the real world and the characteristics of the games themselves. For instance, scenes related to "Govermment & Civic Services" and "Events & Conferences" are typically less fre- quent in games, leading to their lower representation in our dataset. These statistics further validate the richness and real-world attributes of SouniWorldGaame
To provide a more detailed analysis of the dominant "Nature & Outdoors" scenes in OmniWorldd- Game, we further subdivide this category into 5 second-level and 40 third-level categories. The detailed distribution shown in Fig. / Our statistics reveal that "Natural Landforms & Ecosys- tems" is the dominant second-level category. Within this category, scenes depicting "Forests &
17

---

Published as a conference paper at ICLR 2026
0.0
Artruk of Lnnteeee
Figure 7: Scene diversity within the "Nature & Outdoors" category. A quantitative breakdown of second- and third-level scene categories in Ommi0mmoon Warnd-orddaaa Game dataset, demonstrating the high internal diversity and distribution of natural environments.
Rainforests" and "Cliff's & Rock Formations" are the most prevalent. "Outdoor Sports & Scenic Routes" is the second-largest category, with scenes of "Rock-Climbing Areas" and "Scenic Drives & Viewpoints" being particularly prominent. Additionally, "Urban Outdoor Spaces & Activities" and "Agricultural & Rural Landscapes" also make up small portion of the data. These detailed statistics confirm that the "Nature & Outdoors" scenes in Omni World World-Game are not only abundant but also internally diverse. This rich composition provides a diverse data source for world modeling in complex natural environments.
C OmniWorld-Game BENCHMARK
C.1 3D GEOMETRIC PREDICTION
Experiment details. We adhere to the default configurations of each evaluated model. The entire evaluation process is conducted on single A800 GPU. All images are consistently resized long side of 512 pixels while preserving aspect ratio.
For the monocular depth estimation, we evaluate the first 200 frames of 18 test sequences from the OmniWorld-Gand benchmark. Following the evaluation protocols of prior works (Zhang et al., 2024 Wang al. 2),5b(d), we focus on scale-invariant monocular depth accuracy. The primary evaluation metrics are Absolute Relative Error (Abs Rel) and threshold accuracy (8 < 1.25). Under this setting, the depth map of each frame is independently aligned with its corresponding ground truth.
For the video depth estimation, we select the first 100 frames of the same test sequence from the OmniWorld- benchmark. To ensure fair comparison across all models, we cap the input sequence length at 100 frames, as some models (e.g., FLARE (Zhang et al. (0255) cannot handle longer sequences without errors. Similar the monocular depth estimation,
report Abs Rel and 8 1.25. To more comprehensively evaluate depth consistency across video sequences, we provide results under two different alignment settings: (i) scale-only alignment (scale) and (ii) com- bined scale and translation alignment (scale & shift). These settings test model's depth estimation capabilities under different constraints, particularly in handling motion and viewpoint changes.
is important to note that since the benchmark data is included in the training set of 3 (Wag et al. 2025d), we did not evaluate it in our benchmark.
18

---

Published as a conference paper at ICLR 2026
RGB
Ground Truth
DUSER
MASGR
MonST3R
Fast3R
CUTER
FLARE
VGGT
MoGe-
MoGe-2
Figure 8:
Qualitative comparison of monocular depth estimation OmniWorle Game bench-
mark across various methods.
Table 8:
Comparison of original and fine-tuned models for camera pose estimation on Sin-
tel (Butler et al. 2012). TUM-dynamics (Sturm et al. 2012) and ScanNet denotes models that have been fine-tuned on Omni World.
(Dai al. 2017).
Sintel
TUM-dynamies
ScanNet
Method
ATBS
RPE trans]
RPE rot.
ATE.
trans]
RPE cot.
ATES
RPE trans]
RPE rot
CUT3R Wang al. 2025b
0.210
0.071 0.055
0.627
0.045
0,014
0.441
0.096
0,022 0.022
0.733
CUT3R
0.178
0.651
0.041
0.374
0.095
0.604
C.2 CAMERA-CONTROLLED VIDEO GENERATION
Experiment details. AC3D (Bahmani et al.
2024) uses Cog Video-5BB (Yang et al. 2024b) as
base T2V model, it generates 25 frames per inference at a resolution of 480 x 720. CamCirl (He
ct al.
2024 and MotionCtrl (Wang et al. 2024d) use Stable Video Diffusion (SVD) (Blattmann 2023) base I2V model and generate 14-mmame video sequences
ct al.
resolution of 320 x CAMI2V (Zheng et al. 2024) uses DynamiCrafter (Xing et al.) 2023) as base I2V model. It generates 16-frame video sequences
512.
resolution of 320 x 512. For fair comparison with CamCtrl and MotionCtrl, we use the first 14 frames of its generated videos for evaluation. We use
3 (Wang al. 2025d) get camera poses of the generated videos. All methods evaluated
an A800 GPU.
C.3 ADDITIONAL QUALITATIVE RESULTS
We provide additional qualitative results that complement the quantitative analysis in Sec. 3.1 of the main paper. Fig. 1 provides visual comparison of monocular depth estimation results from various methods on our HommiWorld-Gard al.) 2023c) generates depth maps with significantly sharper details and more coherent geometric structures compared to its counterparts.
benchmark. These visualizations confirm that MoGe-2 (Wang
D MODEL FINE-TUNING
D.1 CAMERA POSE ESTIMATION.
Following prior work Wang et al. 2025b we report the Absolute Trajectory Error (ATE), Rel- ative Pose Error for translation (RPE trans), and Relative Pose Error for rotation (RPE rot) on Sin- tel (Butler et al. 2012), TUM-dynamics (Sturm et al. 2012) and ScanNet (Dai et al. 2017). The results in Tab. g show that CUT3R's performance notably improved after fine-tuning on OmniWorld in camera pose estimation.
We perform relative camera pose evaluation on the D0nPose-100K (Rockwell et al. 2025 and the OmniWorld-Ciin Walk test set. Following prior work (Dong et al. 2024), we assess performance with three indicators: AUC@5/10/20, which measure the area under the pose accuracy curve. This curve is based minimum thresholds of 5, 10, and 20 degrees for rotation and translation angular errors. Reloc3r (Dong et al.) 2024) demonstrated substantial improvements in its ability to estimate dynamic camera poses after fine-tuning OmniWorld in relative camera pose evaluation (Tab.
19

---

Published as a conference paper at ICLR 2026
Table Comparison of original and fine-tuned models for relative camera pose evaluation on
DynPose-1000 100K (Rockwell et al., 2025), OmniWorld-Con (Li et al. 2025). denotes models that have been fine-tuned on Omni World.
DynPose-1008
TmnnWWolld-Cre Walk
Method
AUC@5t
AUC@10
AUC@201
AUC@51
AUC@
AUC
Relocãr (Dogg et al. 2024)
15.4
27.1
33.3
49.4
63.1
Reloc3r
14,4
25.5
37.8
42.5
58.0
70.3
D.2 IMPLEMENTATION DETAILS
We conduct comprehensive fine-tuning experiments several SOTAs validate the efficacy of our OmniWorld as training resource. All experiments are performed on 8 A800 GPUs.
DUS13R [Wang et al. 2024c). For fine-tuning, DUS(3R's s original training sets, including ARKitScenes (Baruch et al. 202T), MegaDepth (Li & Snavely 2018). , and Waymo (Sun al.,, 2020). We load the pre-traines weights of DUSt3R and per- formed full fine-tuning. The model is fine-tuned on images with random resolutions (e.g.. 288x512, 384x512 22, 336x512). The training runs for 40 epochs, with each epoch consisting of 800 iterations. We use the AdamW optimizer with an initial learning rate of 2.5 x 10-5 and weight decay of 0.05. Each GPU had a batch size of? 7, with each batch containing two images.
OmniWorldd-aam alongside portion of
CUT3R (Wang et al., 2025b). We fine-tune CUT3R using OmmiWorld-Gamm and a subset of its original training data, including CO3Dv2 (Reizenstein et al., (021), WildRGBD (Xia et al. 2024),
ARKitScenes (Baruch et al. 2021), Waymo (Sun et al. (020), and TartanAir (Wang et al. 2020
We load the pre-trained weights and follow the training strategy from CUT3R's training stage 3. We fine-tune higher-resolution images with varied aspect ratios, setting the maximum side to 512 pixels. The encoder frozen, with only the decoder and heads being trained on longer sequences of 4 to 64 views. The model is fine-tuned for 2,000 iterations with total batch size of 96 and learning rate of 1.0 x 10 6 optimized by Adam with a weight decay of 0.05.
Reloc3r Dong et al. 2024). For fine-tuning Reloc3r, we utilize OmniWorld- Gamee, Omni World- CityWalk, Omni World www.oolddooooAssso HoloAssist, and OmniWorld Eiiccc--ppiKtthhens along with a portion of its original
training sets, including CO3Dv2 [Reizenstein al.) (021). ARKitScenes (Baruch al.) 2021 Scannet++ (Yeshwanth et al.) 2023). BlendedMVS' ( Yao et al. (2020). and MegaDepth (Li & Snavely] 2018). We load the pre-trained weights, freeze the ViT encoder, and only update the weights for the decoder and pose regression head. Fine-tuning is performed on images of random resolutions, including 288 x 512, 384 x 512, and 336 512. The model is trained for 80 epochs, with each epoch comprising 400 iterations. We use the AdamW optimizer with a learning rate of 5.0 x 10-6 and a weight decay of 0.05. Each GPU has batch size of 32, with each batch containing two images.
AC3D (Bahmani et al. 2024). We fine-tune AC3D using OmniWorld- Gaame,,, OmniWorld- EpicKitchens, Omni HO14D, SooniWorld-Egolee as well as the original training set, RealEsttae (Zhou et al.] 2018). We load the pre-trained weights of the AC3D ControlNet (Zhang et al., 2023), which is based on Cog Vog (Yang et al.] 2024b). Only the ControlNet model is fine-tuned, with other network structures frozen. The fine- tuming performed on video clips of 49 frames with resolution of 352 x 640. The model is fine-tuned for 6,000 iterations with a total batch size of and a learning rate of 5.0 x 10-5, optimized by AdamW with weight decay of 0.0001. The fine-tuned and original models are evaluated on two distinct benchmarks: a random subset of 150 video samples from the RealEstate 10K (Zhou et al. 2018) test set and OmmiWorld-Ge 200 video samples. For a fair comparison, all models are configured to output videos at uniform resolution of 720 480 with sequence length of 25 frames.
OnmiWorld-HoloAssiis. OmniWorldd SounWWolddEEgoExee EgoExo4D. and
benchmark, which consists of
D.3 VISUAL RESULTS.
Fig. 9 provides qualitative comparison of DUS13R (Wang et al., 2024c) and CUT3R (ang al. 2025b) on the Sintel (Butler et al.) 2012) subset of the Video Depth Estimation benchmark, evalu- ated both before and after fine-tuning OmniWorld. After fine-tuning, both models recover finer
20

---

Published as a conference paper at ICLR 2026
RGB
DUSt3R
DUSI3
CUT3R
CUT3R
Figure 9: Qualitative comparison of original and fine-tuned models for video depth estimation on Sintel (Butler al. (012). * ddnotees models that have been fine-tuned on OmniWorld. After fine-tuning, both models recover finer geometric details and produce more accurate depth maps, highlighting the efficacy of OmmiWorle as a geometric supervision source.
"A character rides horse dirt path through lush forested surrounded tall pine trees distant snow-covered
mountains under partly cloudy sky.."
AC3D
AC3D
Figure 10: Qualitative comparison of original and fine-tuned models for camera- camera-controlled video generation. denotes models that have been fine-tuned on OmniWorld. The visualizations show that fine-tuning with our dataset significantly improves the model's 's ability to generate videos that more accurately follow camera trajectories and maintain higher temporal consistency for mov- ing objects.
geometric details and generate more accurate depth maps. These results indicate that OmniWorld offers strong geometric supervision and substantially enhance model's geometric prediction capability.
Fig. 10 presents visual comparison of AC3D (Bahmani et al. 2024) the OmniWorld- benchmark before and after fine-tuning on the OmniWorld dataset for the camera-controlled video generation task. The visualizations clearly show that after fine-tuning, the generated videos more closely follow the desired camera trajectory and exhibit higher temporal consistency for moving objects. This demonstrates that OmniWorld significantly enhance model's ability to model dynamics.
E STATEMENT ON LLM USAGE
In the preparation of this manuscript, Large Language Models (LLMMs) only polish writing.
21

---

Published as a conference paper at ICLR 2026
Texture Scarcity
Severe Underexposure
Dynamic Occlusion
Motion Blur
Figure 1I: Examples of low-quality frames discarded by the filtering pipeline. Each column illustrates a specific failure case: Texture Scarcity, Severe Underexposure, Dynamic Occlusion, Motion Blur.
F HIGH-QUALITY VIDEO CLIP SELECTION
To ensure the precision and geometric consistency of camera pose annotations within the OmniWorld dataset, we implemented sophisticated, multi-stage automated filtering pipeline. This pipeline is designed to extract high-quality, texture-rich, and smoothly moving video segments (Splits) from raw, long video sequences, while discarding frames unsuitable for robust reconstruction. The filtering process encompasses two primary stages: frame-level quality assessment and temporal consistency validation.
F.1 FRAME-LEVEL QUALITY A ASSESSMENT
Before sequence segmentation, each individual frame undergoes a rigorous evaluation. Any frame failing to meet the following criteria is designated "invalid framee" triggering truncation of the current video segment.
Texture Scarcity. Robust feature matching is fundamental for accurate camera pose estimation. We employ the SIFT (Lowe. 2004) algorithm for feature point extraction. A frame is discarded yields few valid keypoints, indicative of bomogenous surfaces extreme blur.
Geometric Invalidity. For source data including depth information, we check the integrity of the depth maps. A frame deemed geometrically invalid if it contains invalid values if the area of invalid depth (zero values) exceeds 60% of the total image pixels.
Severe Underexposure. Extremely dark scenes significantly degrade the signal-to- noise ratio, ad- versely affecting reconstruction quality. We quantify the proportion of pixels with values below 20 in the RGB image. If dark pixels constitute a specific threshold of the frame, it is marked as unusable.
Dynamic Occlusion. Large-area dynamic foreground objects can confound SIM algorithms, which typically assume a static scene. Leveraging semantic masks generated by Grounding DINO (Liu al., 2023) and SAM2 (Ravi al., (024), we calculate the screen occupancy of dynamic entities (e.g., characters, vehicles). If the dynamic region exceeds 60% of the frame, it is consequently excluded.
F.2 TEMPORAL CONSISTENCY AND MOTION FILTERING
To guarantee the continuity of inter-frame relationships within video segments, we integrate optical flow estimation, utilizing RAFT (Teed & Deng 2020), to impose constraints on adjacent frame motion characteristics.
Flow Consistency. We compute bidirectional optical flow between frames at time and t + 1, and employ the bawkadd error to detect occlusions and matching inaccuracies. If the proportion of pixels satisfying geometric consistency constraints falls below 50%, it indicates sudden scene change or severe occlusion, leading to the truncation the currett sequence.
22

---

Published as a conference paper at ICLR 2026
Motion Magnitude Limitation. Excessive camera motion often results in motion blur and reduced inter-frame overlap. which can severely impede feature tracking. We calculate the mean magnitude of normalized optical flow. If the average motion exceeds 10% of the image dimensions (a threshold set at 0.1), is classified as rapid motion, and the current segment is truncated. This criterion is crucial for preventing tracking loss during SfM
Fig. 11 provides visual examples various "bad cases" that our filtering pipeline effectively iden- tifies and discards. These examples illustrate frames affected by texture scarcity, severe underexpo- sure, dynamic occlusion, and significant motion blur, all of which compromise the quality of camera pose estimation.
Through this stringent filtering methodology. we effectively eliminate low-quality data, thereby en- suring that each selected video segment within OmniWorldt suitable for generating high-precision camera pose annotations.
G QUALITATIVE VALIDATION OF ANNOTATION PIPELINES
To visually substantiate the effectiveness of our annotation pipelines, we provide qualitative valida- tion.
G.1 CAMERA POSE ESTIMATION
We compare our two-stage pipeline (DroidCalib (Hagemann et al.) 2023) initialized, refined via Co- Tracker3 (Karaev et al. 2024 and Bundle Adjustment) against the baseline DroidCalib (Hagemann al., 2023). To visualize the accuracy of the estimated camera poses, we project the ground truth depth maps into unified 3D point cloud using the estimated camera extrinsic parameters. As shown Fig. 12 the baseline DroidCalib often suffers from drift or misalignment
dynamic scenes, re- sulting "ghosting" artifacts structural inconsistencies (highlighted by red arrows). In contrast, our pipeline produces globally consistent point clouds where static background geometries (e.g., walls. pillars, trees) are sharply aligned, demonstrating superior pose accuracy in the presence of dynamic foregrounds.
G.2 FOREGROUND MASK GENERATION IN DYNAMIC SCENARIOS
We compare semantic segmentation pipeline (Grounding DINO (Liu et al., 2023) with SAM (Ravi et al., 2024)) against the recent SOTA video-specific segmentation model SegAnyMo (Huang al. 2025).
As illustrated in Fig. 13 our pipeline demonstrates superior mask results and robustness in complex dynamic environments. In the second row (narrow alley), our method successfully segments multi- ple walking pedestrians that are missed by the baseline. In the third row (road scene), SegAnyMo incorrectly segments static traffic signs as dynamic foregrounds, whereas our method correctly iden- tifies and segments the moving vehicles. These results validate our choice of the Grounding DINO with SAM pipeline for generating reliable foreground masks, which are crucial for filtering dynamic interference in downstream tasks.
H FAILURE CASE ANALYSIS
While our automated pipeline (Grounding DINO (Liu et al. 2023) with SAM Ravi et al. 2024) demonstrates high reliability across diverse domains, we provide transparent analysis of typical failure cases to highlight potential limitations. We identify two primary scenarios where the pipeline may exhibit imperfections: extreme crowd density and color ambiguity in close-range interactions.
As shown in the top row of Fig. [4 in highly cluttered outdoor scenes with numerous dynamic agents, the pipeline successfully segments the majority of pedestrians. However, due resolution limitations and severe occlusion, detection may fail for distal subjects (i.e., small figures in the far background).
In ego-centric robotic manipulation scenarios (bottom row of Fig. 14 the camera often operates in close proximity objects. When the robotic arm interacts with objects that share similar color
23

---

Published as a conference paper at ICLR 2026
Reference Image
DroidCalib
Ours
Figure 12: Qualitative comparison of point cloud reconstructions between DroidCalib (Hage- mann al., 2023) and our camera pose estimation pipeline. We visualize the consistency of camera poses by accumulating point clouds over a video sequence. The left column shows the refer- ence frame. The middle column shows the reconstruction using poses from the baseline DroidCalib, where red arrows indicate significant misalignment and ghosting artifacts static structures. The right column shows the reconstruction using ours, which effectively resolves these artifacts, result- ing in sharper and more geometrically consistent 3D structures.
textures are spatially adjacent, the segmentation mask may exhibit semantic leakage inadvertently covering nearby static objects along with the moving arm.
Crucially, empirical observations suggest that these localized annotation imperfections have a neg- ligible impact on our primary downstream task: dynamic camera pose estimation. As long as the dominant dynamic foregrounds are masked and sufficient portion of the static background remains visible, the pose estimation remains accurate and stable.
1 DETAILED BENCHMARK STATISTICS AND ANALYSIS
I.1 GEOMETRIC PREDICTION BENCHMARK
To ensure comprehensive evaluation of SOTA models, employed "Attribte-Ballan Sam- pling" strategy for the OmniWorld-Gan Game benchmark. This section details the quantitative distribu- tions and qualitative definitions of our test samples.
We curated set of 90 long-sequer sengeee samples, each containing 384 frames, specifically designed to evaluate long-term geometric consistency, Each sample lasts for 16 seconds.
24

---

Published as a conference paper at ICLR 2026
ALOS
SIA
Mo
A2OS
GroundingDNNO SAM
STA
WN
MogAnyMo
GroundingINO SAM
Figure 13: Qualitative comparison of foreground mask generation on in-the-wild videos. We compare SegAnyMo (Huang et al., 2023) with our pipeline. SegAnyMo struggles to consistently track and segment all moving subjects, while our method provides precise masks for the moving subjects.
Figure 14: Failure case analysis of foreground mask annotations. We visualize the segmentation masks (green overlay) in challenging scenarios.
Scene Distribution. As illustrated in Fig. 15a the dataset maintains balance across Outdoor- Urban (58%), Outdoor-Nature (18%), and Indoor (24%) environments. This distribution ensures that models are rigorously tested on both unbounded scenes with complex backgrounds and bounded scenes featuring intricate internal structures.
Dynamic Complexity. We categorize motion into three distinct levels based on the intensity of object and camera movement. High--Dnnami (48%): Features intense motions such as running, flying, or rapid vehicle movement. Medium- Dynamic (38%): Includes standard motion (walking, running) and regular interactions. Low-Dynamic (14%): Primarily consists of background environmental motion still character animations.
1.2 CAMERA-CONTROLLEE VIDEO GENERATION BENCHMARK
For the camera-controlled video generation task, we selected 200 samples, prioritizing trajectory complexity and environmental richness to ensure a challenging evaluation.
Camera Trajectory Complexity. As depicted in Fig. 15b we split the samples based on camera (51%): Features rapid translations combined with rotations, compound high-speed movements. Medium-Com Comuum--m (49%): Represents distinct but stable motions, typical of cinematic tracking shots.
motion.
Environmental Diversity. The benchmark spans a wide range of lighting conditions (Day 60%, Night 31%, Dusk 9%) and weather scenarios (Sunny 53%, Cloudy 25%, Rain/Snow 22%)
Perspective. We include both First-persoo (64%) and Third-person (36%) views to test generation robustness across different field-of-view dynamics.
25

---

Published as a conference paper at ICLR 2026
Indoor
Day
Disk
High- Dynamic
Night
Diggh-Dnmmic
Outdoor- Natural
Lighting
Sunny
Scene Distribution
Dynamic Complexity
Comera Trajectory
Weather
Perapectives
Cloudy
Rain/ Snow
Medium- Dynamic
Outdoor-
Urban
Low Dynammc
First-Person
Low- Dynamic
Third-
Person
(a) Geometric Prediction Benchmark
(b) Video Generation Benchmark
Figure 15: Statistical distribution of the OmniWorld-Game benchmark samples. (a) The geomet- ric prediction benchmark balanced across scene types (Indoor, Outdoor) and dynamic complexity levels. (b) The video generation benchmark covers diverse range of camera trajectory complexi- ties, environmental conditions, and perspectives.
As shown in the visual examples in Fig. 16 OmmiWorld- high-dynamic sequences introduce complex camera trajectories and drastic geometric changes, posing substantial challenge for both geometric prediction and camera-controlled video generation.
J QUANTITATIVE VALIDATION OF ANNOTATION PIPELINES
To ensure the trustworthiness of the OmniWorld dataset, we conducted extensive quantitative analy- in key modalities.
J.1 DEPTH ANNOTATION VALIDATION
The accuracy our ground truth (GT) depth in OmniWorld- is intrinsically guaranteed by the rendering engine and validated by the fine-tuning experiments in Sec. 4.1 Here, focus validating the quality of our pseudo-labeled depth for the DROID (Kaaztsky et al. 2024) dataset, which was generated using FoundationStereo (Wen al. 2025).
To assess the utility of these annotations in downstream tasks, we pre-trained the FP3 (Yang et al. 2025b) model on point clouds projected from two sources: (1) the original DROID depth, and (2) our refined depth annotations. We then evaluated these models on four real-world tasks.
As presented in Tab. TO the model treitrained success rates across all tasks compared to the baseline using original DROID depth. This result demonstrates that our depth annotations preserve better geometric consistency.
annotated depth yields significantly higher
Table 10: Real-worid manipulation task success rates (%). Comparison of the FP3 pre-trained on the original DROID depth and our refined depth annotations. Our annotations consistently lead higher success rates.
Method
Open Drawer
Stack Cups
Pick up Toy
Put Toy into Basket
FP3 (Original DROID depth)
60
FP3 (Ours refined depth)
40
35
90
55
26

---

Published as a conference paper at ICLR 2026
J.2 CAMERA POSE ANNOTATION VALIDATION
We evaluate our camera pose annotation pipeline in two scenarios: datasets without GT depth (e.g.. in-the-wild videos) and datasets with GT depth (e.g., OmniWorld-Game)
Scenario 1: Evaluation on Data without GT Depth. We compared our full pipeline (VGGT (Wang
al., 2025a) initialization followed by CoTracker3 (Karaev et al. (024) and Bundle Adjustment)
against the baseline VGGT on the Sintel benchmark (Butler et al. 2012). As shown in Tab, [I our method significantly outperforms the baseline, reducing the Absolute Trajectory Error (ATE) by over 50%. This confirms that our optimization strategy effectively refines coarse initializations into precise trajectories.
Table 11: Pose estimation performance on the Sintel benchmark. Comparison between the base- line VGGT and our full annotation pipeline.
Method
ATE L
RPE trans J
RPE rot |
VGGT (Baseline)
0.167
0.062
0.491
Ours
0.082
0.042
0.246
Scenario 2: Evaluation on Data with GT Depth. To evaluate the reliability of our camera pose an- notations where GT depth is available, we adopted the rigorous validation protocol following (Rock- well et al. 2025). We conducted a large-scale evaluation across 8,345 randomly sampled frame pairs from dataset.
For each pair, we extracted high-quality sparse correspondences on static regions using Super- Point (DeTTone al. 2018) and LightGlue (Lindenberger al., (023), explicitly masking out dy- namic objects to ensure geometric validity, We then computed the geometric consistency (via Samp- son error) for poses estimated by our pipeline compared to the DroidCalib (Hagemann et al. 2023) baseline.
The results, summarized Tab. 12 demonstrate the superiority of our approach. Our pipeline re- duces the mean reprojection error to 1.09 (from 1.30 px) and improves fine-grained precision, with 78.36% of correspondences across frame pairs falling within a 1-pixel error threshold, com- pared to 69.85% for the baseline. Beyond numerical metrics, our pipeline produces visibly more consistent point cloud reconstructions and the visualizations in Appendix G.] have already demon- strated this.
Table 12: Evaluation of camera pose annotation reliability. We report the geometric consistency (reprojection error) on 8,345 sampled pairs, comparing our pipeline against the DroidCalib baseline.
Method
Mean Error (px)
% < 1 Pix
% < 3 Pix
% 5 Pix t
DroidCalib (Baseline)
1.30
69.85% 78.36 %
91.42% 93.90 %
96.02%
Ours
1.09
96.66%
27

---

Published as a conference paper at ICLR 2026
Figure 16: Visual examples of the OmniWorld-Game Benchmark. Each row displays sample RGB frames from a long sequence alongside the corresponding ground truth 3D point cloud and camera trajectory (indicated by yellow frustums). The samples demonstrate high diversity in scene types and complex camera motions, reflecting the challenging nature of the benchmark.
28


Reviews:
[{"review_id": "EeIsOmJ33H", "reviewer": "Reviewer_NEvh", "summary": "This paper introduces a large-scale, multi-domain, multi-modal 4D dataset. It comprises over 214 hours of videos, includes simulation, human, robot and internet, and provides rich annotations including depth, camera, text, flow and masks. Based on this dataset, the paper also establishes a challenging 4D benchmark and validates the dataset as a powerful resource for training.", "questions": "See Weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The dataset is large-scale, multi-domain and multi-modal, while previous datasets are limited.\n* The paper proposes an annotation pipeline for the data from different domains and different outputs.\n* The paper validates that the dataset can be a powerful resource for both training and evaluation.", "weaknesses": "* It's hard to validate the quality of the dataset. The paper only leverages the pretrained models for annotation. The open-source real-world datasets are very noisy and the pretrained models can not handle them perfectly. How to filter the bad cases?\n* The paper lacks the ablation analysis of the annotation pipeline. Which pretrained model is most suitable for each dataset and why.\n* The paper lacks the analysis of how to select the benchmark samples.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkWNNE0atr", "reviewer": "Reviewer_XxMY", "summary": "The paper introduces OmniWorld, a large-scale multi-domain and multi-modal dataset (RGB, depth, pose, text, optical flow, mask) designed for 4D world modeling. It unifies new synthetic data (OmniWorld-Game) and re-annotated public sets, supporting benchmarks for 3D geometry prediction and camera-controllable video generation. The dataset features higher resolution, richer modalities, and more complex dynamics than existing resources like Sintel or TartanAir. Its automated annotation pipeline and unified benchmark design expose clear weaknesses of current geometric and generative models, and fine-tuning on OmniWorld improves performance across multiple datasets.\n\nHowever, the paper lacks clarity on how dynamic objects and camera motions are modeled, leaving its 4D claim partially unsubstantiated. The work also relies heavily on synthetic data, with limited validation on real-world domains (human, robot, internet). Moreover, the quality and uncertainty of automatically generated annotations are not quantitatively analyzed.\n\nOverall, OmniWorld is a valuable and ambitious contribution that could serve as a strong foundation for 3D and 4D perception research. Clearer handling of dynamic scenes, stronger real-world evaluation, and annotation-quality validation would further improve the paper. My current recommendation is Borderline Accept.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Significant scale and modality improvement.**\nCompared with existing synthetic datasets, OmniWorld-Game offers clear advantages in resolution, frame count, and modality diversity. It also covers more dynamic and complex scenes. Table 1’s comparison against Sintel, TartanAir, and HyperSim highlights this superiority.\n\n2. **Cross-domain integration and automated annotation pipeline.**\nThe paper clearly details an end-to-end annotation pipeline, from four-domain collection, video slicing and filtering, to modality annotation, and provides domain composition statistics, which support reproducibility and future extension.\n\n3. **Challenging benchmark design.**\nCompared with prior benchmarks featuring short sequences and smooth camera motion, OmniWorld-Game introduces longer sequences, complex trajectories, and dynamic scenes. On this benchmark, most state-of-the-art GFM/video generation models exhibit weaknesses in temporal consistency and camera control (as shown in Table 3/Table 4 and qualitative examples).\n\n4. **Demonstrated training value.**\nModels such as DUSt3R, CUT3R, and Reloc3R, when fine-tuned on OmniWorld, show consistent performance gains on standard public benchmarks (Sintel, KITTI, NYUv2), indicating that the dataset provides practical benefits for improving 3D geometry and temporal coherence.", "weaknesses": "1. **Unclear handling of dynamic 4D content.**\nGiven the central claim of 4D world modeling, the paper should more explicitly describe how dynamic objects and camera motions are represented during dataset construction, particularly in the synthetic (simulation) domains. It is unclear whether the dataset systematically varies object motion patterns or camera trajectories, or includes a taxonomy distinguishing static versus dynamic regions. In evaluation, the study focuses mainly on static 3D foundation models; incorporating 4D reconstruction or dynamic scene methods (e.g., MegaSAM) would make the benchmark more consistent with its 4D ambition and strengthen its empirical validity.\n\n2. **High reliance on synthetic data and limited real-world evaluation.**\nAlthough the authors highlight the multi-domain nature of OmniWorld, both training and evaluation are largely centered on OmniWorld-Game, the synthetic/game-based subset. Systematic experiments on the human, robot, and internet domains are limited, which makes it difficult to assess how well the dataset and benchmarks generalize to real-world conditions. A stronger cross-domain analysis would clarify the practical robustness of the proposed data.\n\n3. **Lack of annotation-quality assessment and uncertainty analysis.**\nMany modalities (depth, pose, optical flow, and foreground masks) rely on automatic or pseudo-labeling pipelines, yet the paper provides no quantitative evaluation of their accuracy or reliability. It would be valuable to include error quantification (e.g., comparisons with manual ground truth or high-confidence models), failure-case analysis, and sensitivity studies showing how annotation noise affects training and evaluation outcomes. These analyses would help validate the trustworthiness of the dataset’s supervision signals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QkXoIcDmZG", "reviewer": "Reviewer_xQoM", "summary": "This paper proposes OmniWorld, a large-scale dataset designed to address the lack of high-quality data for 4D world modeling.\nThis paper proposes OmniWorld-Game, a benchmark for 3D geometry prediction and camera control video generation.\nExperiments show that fine-tuning existing state-of-the-art models on OmniWorld significantly improves their performance in 4D reconstruction and video generation tasks, validating the value of this dataset.", "questions": "Regarding the problem of uneven data distribution, are there any plans to supplement data from fields such as robotics and the Internet, or use data augmentation technology to balance the data proportions in each field to improve the model's generalization capabilities in multiple scenarios?\n\nOmniWorld-Game is derived from a game environment. How can we quantify the difference between this synthetic data and real-world data? Are there any experiments validating the performance of models trained on OmniWorld when transferred to real-world datasets (such as real indoor/outdoor 4D datasets)?\n\nThe current benchmark does not involve the evaluation of extreme scenarios (such as intense motion, severe occlusion, and low light). Will you consider expanding the benchmark's scene coverage in the future to more comprehensively measure the model's 4D modeling capabilities under complex real-world conditions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "OmniWorld covers four major areas: simulators, robotics, humans, and the internet, and contains over 600,000 video sequences and 300 million frames of data. OmniWorld-Game significantly surpasses existing synthetic datasets in modal diversity (such as depth maps and optical flow) and data volume, providing a rich resource for 4D modeling.\n\nChallenging benchmarks are constructed for 3D geometry prediction and camera control video generation. Multi-model evaluation reveals current state-of-the-art shortcomings. Fine-tuning experiments demonstrate that the dataset effectively improves model performance and verifies the integrity of the logic.", "weaknesses": "Human-domain data accounts for the largest proportion in OmniWorld, while data from other domains (such as robotics and the internet) is relatively scarce. This may lead to insufficient training of the model's generalization ability in non-human-related scenarios.\n\nThe core subset, OmniWorld-Game, is derived from a gaming environment. While it strives to simulate real-world scenarios, differences exist between game rendering and the real physical world, potentially affecting the model's transferability to real-world scenarios. Furthermore, the impact of this \"simulation-to-reality\" gap on task performance has not been thoroughly analyzed.\n\nIn the evaluation of 3D geometry prediction and camera control video generation, the main reliance is on quantitative indicators (such as Abs Rel, FVD) and some qualitative comparisons. There is a lack of evaluation of the model's performance in extreme scenarios (such as fast motion and complex occlusion), which cannot fully reflect the model's robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)