DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling
Authors: Kairun Wen
Yuzhihuang
Runyu Chen
Hui Zheng
Yunlong Lin
Panwang Pan
Chenxin Li
Wenyan Cong
Jian Zhang
Junbin Lu
Chenguo Lin
Dilin Wang
Zhicheng Yan
Hongyu Xu
Justin Theiss
Yue Huang
Xinghao Ding
Rakesh Ranjan
Zhiwen Fan
Venue: NeurIPS 2025 poster ()
Keywords: Metric‑Scale 4D Reconstruction
Automated Vision–Language Data Curation
Multimodal Dataset
Primary area: applications

Abstract: Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human‑like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structure-from-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity of foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet. To bridge these gaps, we introduce **DynamicVerse**, a physical‑scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models to interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-based Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D multimodal format. DynamicVerse delivers a large-scale dataset consists of 100K+ videos with 800K+ annotated masks and 10M+ frames from internet videos. Experimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsics estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.

DemamicVersee A Physically-waare Multimodal Framework for 4D World Modeling
Kairun Wen 1tt Yuzhi Huang" Runyu Chen', Hui Zheng', Yunlong Lin', Panwang Pan',
Chenxin LI, Wenyan Cong Jian Zhang' Junbin Lu Chenguo Lin", Dilin Wang Zhicheng Yan Hongyu Xu Justin Theiss Yue Huang', Xinghao Dingl Rakesh Ranjan", 8 Zhiwen Fan
Equal Contribution; Project Leader, Corresponding Author
XMM "CUHK SUT Austin
*UW 'PKU "Meta
Project Website: https://ddaaniic-eeereeeee/ www.eeeeeee.......eeeeeeee github.io/
Video Frames
Dynamic Content (Moving Object Camera)
Annotations
Object Mask & Category
Object Caption
Caption
Caption
Metric-scale Camera
PolntMap
Object Category: Running Lady
Camera Caption: The camera moves forward, following running lady from behind, resulting slightly unsteady motion with shakinng As the video concludes, the camera stops tracking, tilts up, and pans left reveal the scene ahead.
Object Caption: elderly lady
wearing vibrant multicolored
pants,
steady, rhythmic gait. Their arms are slightly bent, holding a small object. Maintaining an upright posture with head tilted forward, they move at consistent
suggesting focus, purpose, familiarity with their path.
Scene Caption: lively
inside spacious, well-lit restaurant, characterized by wooden floors, large windows with natural light, and mix of modem rustic decor including exposed brick walls and furniture. An elderfy lady walks purposefully through the warm, inviting space, her stride steady, busiting with diverse patrons seated enjoying meals conversations many tables with cutlery glasses. The ambiance cazy yet sophisticated.
Figure 1: The overview of physically-awar multi-modal world modeling framework Dynamic Verse.
Abstract
Understanding the dynamic physical world, characterized by its evolving 3D structure, real-world motion, and semantic content with textual descriptions, is crucial for human-agent interaction and enables embodied agents to perceive and act within real environments with human-like capabilities. However, existing datasets are often derived from limited simulators or utilize traditional Structure- from-Motion for up-to-scale annotation and offer limited descriptive captioning, which restricts the capacity foundation models to accurately interpret real-world dynamics from monocular videos, commonly sourced from the internet.
To bridge these gaps, introduce Dynamic Verse, physical-scale, multimodal 4D world modeling framework for dynamic real-world video. We employ large vision, geometric, and multimodal models interpret metric-scale static geometry, real-world dynamic motion, instance-level masks, and holistic descriptive captions. By integrating window-baa Bundle Adjustment with global optimization, our method converts long real-world video sequences into a comprehensive 4D mul- timodal format. Dynamic Verse delivers large-scale dataset consists of 100K+
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

videos with 800K+ annotated masks and 10M+ frames from internet videos. Ex- perimental evaluations on three benchmark tasks, namely video depth estimation, camera pose estimation, and camera intrinsies estimation, demonstrate that our 4D modeling achieves superior performance in capturing physical-scale measurements with greater global accuracy than existing methods.
Introduction
Humans inhabit a dynamic 3D world where geometric structure and semantic content evolve over time, constituting 4D reality (spatial with temporal dimension). Understanding this dynamic environment fundamental for developing advanced Al applications in fields such as robotics 2 3 4 3. 6). extended reality 7 g 2), and digital twins [10 [T]. However, building generalizable foundation models for these downstream tasks faces a longstanding challenge: acquiring high-quality, ground- truth 4D datasets from real-world environments, given that data-driven solutions increasingly demand 4D data while its collection using multiple sensors remains non non-scalable. This raises the question: Can develop automated pipeline capable of generating a real-world 4D dataset at scale?
Current real-world 4D data primarily focus on indoor scenes 12 13 or autonomous driving scenar- ios [14, where geometry capture is straightforward, but their diversity is limited. Even synthetic 4D data [15] 16 [8), while controllable, often lack the fidelity and complexity required to truly represent the real world, resulting in notable simulation-to-rea gap. Moreover, physically-awe aware multimodal annotations- -nniuuuuinnnnnnnnnininn including metric-scale 3D geometry, detailed representations of non-rigid actors (e.g., object size, mask and bounding box. etc.), and descriptive captions of dynamic contents (i.e., object, camera and scene) are often absent [9 20). This limited data landscape, especially when contrasted with the progress fueled by large-scale datasets in modalities like images, videos, and language, underscores the compelling need for a large-scale, diverse, physically-awes awwae, and semantically rich annolated multi-modal datasel for 4D scene understanding.
Against this background, this paper aims to generate scalable, physically-awan awware, and multimodal annotations from massive monocular video data (see Fig. [ for numerous potential applications, such as enhancing 4D Vision-Language Models [21], facilitating advanced 3D-aware video generation [22]. and enabling linguistic 4D Gaussian Splatting (23). However, achieving this goal is not trivial. To the best of our knowledge, there is currently a significant lack of rich and diverse 4D datasets (see Tab. D adequate for these demanding tasks. To address this data scarcity, we introduce DynamicGen, novel multi-modal 4D data at scale. This pipeline contains two main stages: (1) metric-scale geometric and moving object recovery (i.e., object category and mask) from raw videos, and (2) hierarchical dynamic contents (i.e., object, camera and scene) detailed caption generation. Specifically, the pipeline curates diverse real-world monocular video sources; employs a filtering strategy to remove outliers such as camera motion intensity; integrates multiple foundation models (i.e., VFMs, VLMs, LLMs, GFMs) for initial frame-wise annotation; applies dynamic bundle adjustment to jointly minimize global photometric error; and concludes with dynamic content captioning at three granularities and muman-in-the-loop quality review to ensure annotation semantic accuracy.
automated data curation pipeline (see Fig. 3) designed to generate physically-awan
The resulting multi- modal 4D dataset, termed Dynamic Verse (see Fig. D.. comprises over 100K distinct 4D scenes, 800K masklets, and 10M video frames. Each scene extensively annotated with multiple modalities: metric-scale point maps, camera parameters, object masks with corresponding categories, and detailed descriptive captions. We evaluate DynamicGen through three benchmarks: video depth estimation, camera pose estimation, and camera intrinsics estimation. We demonstrate the generalization capability of DynamicGen to process web-scale video data and extract multi-modal information gualitatively. We also conduct human study and GPT-assited evaluation validate the quality of generated captions.
Our main contributions are summarized as follows:
We develop DynamicGen, a novel automated data curation pipeline designed to generate physically-awar multi-modal 4D data at scale. This pipeline contains two main stages: (1) metric-scale geometric and moving object recovery from raw videos, and (2) hierarchical detailed semantic captions generation at three granularities (i.e,, object, camera and scene). Powered by foundation models (i.e., VFMs, VLMs, LLMs, GFMs), DynamicGen efficiently generate 4D data

---

at scale, thus addressing the critical scalability, physical reality and modality diversity limitations of traditional 4D data curation.
We introduce Dynamic a large-scale 4D dataset featuring diverse dynamic scenes accom- panied by rich multi-modal annotations including metric-scale point maps, camera parameters, object masks with corresponding categories, and detailed descriptive captions. Dynamic Verse encompasses 100K+ 4D scenes coupled with 800K+ masklets, sourced through combination of massive 2D video datasets and existing 4D datasets. This represents a significant improvement in terms of data scale, scene and modality diversity compared to prior 4D datasets.
We validate DynamicGen through three benchmarks: video depth estimation, camera pose and intrinsics estimation. We demonstrate the generalization capability of DynamicGen to process web-scale videos and extract multi-modal information qualitatively. We also conduct human study and GPT-assited evaluation to validate the quality of generated captions.
2 Related Work
Table I: Comparison of Dynamic Verse with large-scale 2D video datasets and existing 4D scene datasets. Dynamic Verse expands the data scale and annotation richness compared prior works.
Numerical Stataitis
Provided Amtviations
Detailed Feniures
# Vidoss
Frames
Musklets
Canera
Depitmmp
Instance Madk
Semaniic Mask
Object Category
Object Caption
Scane Captinn
Camera Caption
Type
Dynumic Type
Real-world?
Metric scale?
Datasct Name
2D Video Datazet
DAVIS2017
24
0.2K
10.7K
D.4K
X
X
Youtube-VIS
25
3,8K
UVO-dense
1.0K
68.3K
10.2K
VOST 27
0.7K
75.5K
1.5K
X
BURST
2.9K
195.7K
MOSE 29
2.1K
638.8K
5.2K
SA.V [30
50.9K
4,2M
642.6K
MinaDATA
31
330K
Scene Dutaver
T.Air Shibuya
32
7
0.7K
Mixed
Street
Synthetic
Yes
MPI Sintel 31
14.
0.7K
Scripted
Synthetic
FlyinTThiees Things3D
220
2K
Mixed
Objects
Syuthetic
Waymo 14
1,150
200K
Outdoor
Driving
Real-world
Yes
CnP3D 12
4,200
600K
Mixed
Pets
Real-world
Sterco4D
110,000
10,000K
Mixed
fisheye
Real-world
Yes
PointOdysscy
151
159
200K
Mixed
Realistic
Synthetic
Yes
Spring [T]]
47
6K
Mixed
Realistic
Synthetic
Yes
Dynamic Replica
524
145K
Indoor
Realistic
Synthetic
Yes
MVS-Syath [8
120
12K
X
Ouidoor
Urban
Synthetic
Yes
RealCam- Vid
[9
100K
Mixed
Realistic
Synthetic
Yes
1ynPose-1100k
20
180K
6,806K
Realistic
Synthetic
Yes
DynamiieVesse
100K
13.6M
800K
7
7
7
Mixed
Realistic
Real-world
Yes
Multi-modal foundation models. The development of numerous large foundation models in recent
years has yielded remarkable performance across multiple tasks such as depth estimation [6 37
38
39
40]
multi-view stereo [4]
42
43 detection and segmentation [44]
45
46
30), human
We propose that these models are highly applicable achieving holistic 4D understanding, and unifying them within single framework represents a promising direction for advancing tasks like nonrigid structure from motion. Our DynamicGen pipeline implements this idea by integrating the following pretrained components: UniDepthv2 [52) for geometry initialization, CoTracker3 [51] and UniMatch [49] for correspondence initialization, and Qwen2.5-VL [53] and SA2VA [54] for dynamic object segmentation. This integration, coupled with multi stage optimization and regularization, allows us to extract accurate metric-scale camera poses and 4D geometry from monocular video. Similar to our method, the concurrently developed Uni4D [55] captures 4D geometry and pose, but it suffers from limited data modalities and discontinuous geometric estimates. In contrast, our DynamicGen pipeline not only produces globally refined dense 4D geometry but also supports moving object recovery (i.e., object category and mask) and provides fine-grained dynamic content (i.e., object, camera and scene) caption annotations.
parsing 47
optical flow estimation
48 49
and point tracking (50) 51 38
3

---

Chald seea
Deiver
PointOdyssey(159)
Shopping ni
Hot Car Vehicle Rider
Right
Spring(47)
Pant Quickty
DynamicReplica(524)
Annotating Object Mask and Category
Pedestrial
oer
Soenent
Shaking
MVS-Synvth(1220
Dog
DynPose-100K(100K)
Moving objects type.
(b) mution type.
Captioning Rephrasing
Dynamic Content
RARRATE TRANSMIT SUPPORT
PRPOMARE URBAN SHSPPING
DAVIS(O.2K]
Youtabe-V1S(2.80
UVO-dense(L.OK,
Annotating Camera and Pointmap
MOVE FLA
OUTDOOR INDOOR AREA
SCENE
VOST(0.7K)
SLAOE
MARIE
BURST(2.9K)
DRIVE
100 PIBRATE ROTATE
MOSE(2.1K)
STREET
SA-V/50.9K)
Data Filter
(c) Object actions upp.
(d) Dynemee Seone (environments) type.
(e) source of DynamieVersc.
Figure 2: The statistics and data source of Dynamic Verse.
Multi-modal datasets.
The development of large-scale multi-modal datasets has proven essential for advancing model performance across numerous domains, including language, image-text (e.g.,
LAION [56 [57. Conceptual Captions [58], WebImage (59)) and video understanding (e.g.,
DAVIS2017 (24), Youtube- VIS [25], UVO-dense [26]. VOST (27. BURST 28]] MOSE [29], SA-
30. MiraDATA BI]. Extending this success to holistic 4D understanding requires datasets that capture the dynamic 3D world with rich, mulli-modde annotations. Existing 4D datasets, whether from early reconstruction efforts 13 [6] 17
[8] (limited diversity) or recent large-scale posed video collections like RealCam Vid [99 and DynPose- 100K (0) (lacking detailed geometry and semantics beyond pose), and even OBJAVERSE (6) (limited content), fall short of providing the comprehensive multi-modal information needed. Our Dynamic dataset bridges this gap by offering extensive multi-modal annotations, including metric-scale depth, camera parameters, instance segmentation with labels, and descriptive captions, specifically designed facilitate advanced 4D research.
3 DynamicVerss
Overview Dynamic a physical-scale, multi-modal 4D modeling framework for real-world video, which contains novel automated data curation pipeline and corresponding large-scale 4D
dataset. The DynamicGen pipeline (see Fig. 3 contains two main stages: (1) metric-scale geometric and moving object recovery (i.e., object category and mask) from raw videos, and (2) hierarchical dynamic contents (i.e., object, camera and scene) detailed caption generation. This pipeline primarily consists of five steps: 4D scene curation (in Sec. 3.1 data filter strategy (in Sec. 3.2), moving
object recovery (in Sec. 3.3). dynamic bundle adjustment (in Sec. 3.44 and dynamic content caption generation (in Sec. B.5 The resulting Dynamic Verse dataset comprises over 100K distinct 4D
scenes, 800K masklets and 10M video frames. The data statistics and collection of Dynamic Verse
illustrated in Fig. 2
3.1 4D scene curation
To address the scarcity of available 4D scene data, DynamicGen unifies video data from vari-
ous real-world video datasets, including DAVIS2017 (24). Youtube-VIS [25). UVO-dense [26),
VOST (27, BURST (28), MOSE 29] and SA-V (30, alongside existing synthetic 4D datasets from
PointOdyssey [5), Spring [6]. Dynamic Replica [17]. MVS-Syme (18), RealCam- Vid [19] and
DynPose- 100K 20). The inclusion of these datasets is mainly motivated by their potential as scalable
data sources for 4D scene understanding.
3.2 Data filter strategy
Data filtering is a critical step for identifying video data suitable for subsequent dynamic bundle adjustment. This process presents challenges due to the noisy quality and inherent variability video data, which impedes the precise selection of high-quality sequences. To address this, we developed filtering strategy incorporating several distinct criteria: proximal depth, focal-length stability, video blur, camera motion smoothness, and non-perspective distortion. Each of these aspects is quantified

---

Stage 1 Generating Metric-scale Camera, Video Depth, Object Category, Object Mask and Size
Callection
Poverful Foundation Mudeés
Data Filter Stralagy
Dynamic A dijsstment
Depti Vacilication
Dynamic Masiing
Object Coverage
VaDepu-V2
Focal engt Stabiity
Course Camera Intrliizttio
Video Blur Degree
Naw 20 vidro datasets with GT
Iestanco mask & semantic label
UniMatzh
VoIrackae-V3
6
VLM Jndgemmnnt
Static Area Bundle Adjustninnt
Moving Object Recovery
Non Rigid Bundle Adjustment
SANIK Vartents
VGGT
VIM based object recequttonn
Semantie-he hased Segmentation
Global Rutinoment
Raw drnamic
Phystral-wware Size Extraction
deyth
Eppolar Language Models
Stage Generating Fine-grained Captions Moving Objects, Cameras Dynamic
ara Moving Object
Dynamic Scone Captioa
Camera Metlen Caption
Frergt tiee the mo adoe convng intence wask flew
und
Previees diei ondeetinnel The
Prenge theee wiee koaah autuane cigh Provite descripiic
Fooa oe in action mimiunee you it ortn shope motemsl
Regoonee (nbjectiewel)
Resporse sennellaal
mine A righ. let breac cown the steribueer stop by step"
mec First obeeme the bos ochan mand crasch
Mert,
sola 1h4 insrrcctien antitie young bon
Raspaas
Videm
edker anpeliie park orharo valuein] J-leeeen
sote dems na iphowight Prealy
We it. Feuily, lake aiceast the opaiial
mocking Bigacc
performin vaniral acrabol i oot orenn npaavin (omaemmn
shik
exphrased
sequercs snanding to facuaed crouch thatr aovenants
Beyrevig
and cark potty bur
wntver|
the pork fasturia numppl frangel hhs aits blas pasdiig and aie sheegh pnirt crybe ledy. battos lig
Curde yytoore Indoor
rendenir
LLM--aiined
Jestnce Madks
right tie verg lax park with ose red teriing
enbodsiing otileticien and groce thickgg tour tiee
Young Roy" Titnkanse Label
Figure 3: The physically-awee multi-modal 4D data generation pipeline DynamicGen.
by a normalized score. We combine these scores as features and employ a Random Forest model to predict a video quality score ranging from 0 to 5. For model training, manually annotated approximately 1,000 videos, assigning scores between 0 (indicating largely unsuitable, poor quality insufficient dynamics) and 5 (indicating highly suitable, good quality and sufficient dynamies). We further apply VLM-based judgment to automatically exclude unsuitable videos before reconstruction.
3.3 Moving object recovery
To accurately identify the main dynamic objects within a video, we integrated multiple foundation models to achieve reliable segmentation. Specifically, our pipeline first employs Qwen2.5-VV VL [61] to identify moving objects and determine their semantic categories. These categories are then used to prompt SA2VA (54) for generating corresponding object masks. Leveraging the obtained object masks and geometric annotations, we can apply physical-aware size extraction to annotate the 3D bounding box for moving objects.
3.4 Dynamic bundle adjustment
Leveraging the high-quality RGB filtered videos, we employed a robust dynamic bundle adjustment method for annotating metric-scale camera parameters and point maps. This task is challenging due to dynamic objects occluding the static scene and static scene appearance changes hindering correspondence estimation. To effectively addresses both difficulties, we design a multi-stur stage optimization framework, see Fig. 3 including: (1) dynamic masking, (2) coarse camera initialization, (3) tracking-based static area bundle adjustment, (4) tracking-based non-rigid bundle adjustment, and (5) flow-based sliding window global refinement. Compared with traditional Sruucure-from-A from-Motion techniques [62] and DUSt3R -bsse methods [63], our framework not only can handle massive video data with different resolutions but also yield metric-scale results by leveraging the full power of various foundation models.
Formulation Given 7 video RGB frames I I= (T,,... IT) with resolution H x W, we aim to estimate for each timestep = 1,..., T: per-frame pointmap X e Hx x3, camera intrinsics K and camera pose p R T where R and T denote the t-th camera's 's rotation and translation.
5

---

Video
SegAnyMo
GPT+DEVA
DynamicGen
SWIS
RAVK
DENDOFF SPRINGS
IAAF ORG
Canon
5
VT
SINOP
LAAF.oRD
IAFF.om ORG
IAFFcom oRG
Figure 4: Qualitative Results of Moving Object Segmentation. We show qualitatively some of our segmentation results on the Youtube-VIS dataset compared with other methods.
respectively. Here, X contains static points Xetatic and dynamic points Xdyn- We assume all frames share the same intrinsics K where we optimize focal lengths fo and fy. The overall cost function is formulated follows:
CBA (P, Xstatie) + Citow (Xstatic) + CNR (Xaya) + Cmotion (Xyyn) Ceam R
(I)
where CBA Xsuicc) and Cnow (Xanic) are bundle adjustment terms measuring the reprojection error between static correspondences and the static 3DD structure X ONR CNR(Xayn) is non-rigid fructure-from-mottin term evaluating the consistency of the dynamic point cloud with its tracklets. Regularization is applied camera motion smoothness through Ceam (P) and the dynamic structure and motion via Omotion (Xyn). Each term participates in different optimization stages, which are described below. Detailed explanations of the cost terms are provided in the supplementary material.
Stage I: Dynamic masking We first extract dynamic masks to filter out the dynamic points for static area bundle adjustment, Specifically, we use semantic- based and motion- based method to obtain dynamic masks M {M') {M
)T= For the segmentation-bant based approach, use the generated moving object masks [M in Sec. 3.3 For the flow-based approach, we employ Unimatch 49 to obtain dense optical flow predictions and compute per-frame epipolar error maps (64, which indicate the likelihood of pixels belonging the dynamic foreground. Then we obtain dynamic masks M foow [E, E2....... Er] by thresholding these epipolar error maps.

---

Stage II: Coarse camera initialization In this stage, we start camera initialization by obtaining video depth D (D+) Di and dense pixel motion = { For video depth estimation, we use UniDepth V2 [52] a monocular depth estimation network, to estimate initial depth maps D and initial camera intrinsics Kinit For dense pixel motion estimation, we utilize Co-Trrao TrackerV3 [S]] for its robustness. We apply Co- Co-Tracker bi-directionally on dense grid every 10 frames to ensure thorough coverage. We filter and classify tracklets using segmentation masks yielding a set of correspondent point trajectories {Z E KT*'2] j at visible time steps determined by Co-Tracker. Combining DD and 2 allows us to establish 2D-to-3D correspondences. This allows us to initialize and tune camera parameter P by minimizing the following cost function with respect to camera parameters only. Specifically, we can unproject each video frame's 's depth at time t back to 3D and minimize the following cost function:
min Σ
ΣΣ
E
[Z,,e' - ( (Z De, E2
(1)
(0 ZLE .M
where "K is the unprojection function that maps 2D coordinates into 3D world coordinates using estimated depth D,. We perform this over all pairs within a temporal sliding window of frames. Given camera initialization P, we unproject our depth prediction into common world coordinate system, which provides an initial 4D structure X. This is used as initialization for later optimization.
Stage II: Static area bundle adjustment We jointly optimizes camera pose and static geometry by minimizing the static component-related energy in a bundle adjustment fashion. Formally speaking, we solve the following:
min CBA M, + Ceam (R)
(2)
P.Xauaic
By enforcing consistency with each other, this improves both the static geometry and the camera pose quality. We perform a final scene integration by unprojecting correspondences into 3D using improved pose and filtering outlier noisy points 3D.
Stage IV: Non-rigid bundle adjustment Given the estimated camera pose, this stage focuses on inferring dynamic structure. Note that we freeze camera parameters in this stage, as we find that incorrect geometry and motion evidence often harm camera pose estimation rather than improve it. Additionally, enabling camera pose optimization introduces extra flexibility in this ill-posed problem, harming robustness. Formally speaking, we solve the following:
min CNn P, M) + Cmotton dya)
Xáya
We initialize Xdyn using video depth and our optimized camera pose from last step. This energy optimization might still leave some high-energy noisy points, often from incorrect cues, motion boundaries, or occlusions. We filter these outliers based on their energy values in final step. To further densify the global point cloud, enabling each pixel to correspond to a 3D point, we perform depthbased interpolation by computing scale offset.
Stage V: Sliding window global refinement Given the estimated optical flow, this stage focuses on refining static structure. Note that we freeze camera parameters in this stage. Formally speaking, we solve the following:
min Chow (Xutatie)
Xatatic
With consideration for accuracy and efficiency, the sliding window global refinement is capable of significantly enhancing the multi-view consistency of static points and generalizing effectively to real-world 4D scenes. The detailed process can be found in the appendix.
3.5 Dynamic Content Caption Generation
Drawing upon the emphasis placed by LEO 65 and Scene [66] on the criticality of caption quality and granularity for comprehensive scene understanding, we design captions three specific levels: object, scene, and camera. Object captioning focuses on detailed object motion, scene captioning describes object-scene interactions, and camera captioning conveys intricate camera movement. To argument the caption, Large Language Models (LLMMs) are employed to automatically rephrase initial captions and align them with these three granularity levels. Finally, to ensure data quality, human verification is conducted to filter out low-quality caption annotations.
7

---

Moving object captioning. Moving object captions provide detailed descriptions crucial for object grounding. However, prior datasets often have incorrect temporal alignment [66] or insufficient detail 15 67 while current video captioning methods yield only simple (e.g., Panda-70M (68])
or non-localized descriptions (e.g.. Qwen2.5-VI. [6]]. To address these limitations and generate detailed, accurate captions for individual objects, we utilize DAM 69). known for its superior capabilities. Given RGB videos and corresponding object masks, DAM [69] generates detailed and temporally aligned object descriptions through carefully designed prompts, enabling precise grounding and richer scene understanding.
Dynamic scene captioning. Scene-level captions are designed capture global information, depicting the key objects within the scene along with their associated actions, interactions, and functionalities. For comprehensive understanding of the entire dynamic scene, we utilize Qwen2.5- VL [6] for dynamic scene captioning. To obtain more detailed, fine theeggrined, and accurate captions, we propose the use of structured captions. This process involves leveraging the fine- ane-grained moving object captions as auxiliary input and employing specific prompting to generate the final scene-level descriptions. the design of the prompts, we discovered that an explicit Hierarchical Prompt Design [70] significantly aids the Qwen2.5-VI.ft and its operational boundaries. This approach contributes to the stabilization of the output's 's format and enhances the overall quality of the results.
in comprehending its role, its expected format,
Camera motion captioning. Camera Motion Captioning aims to describe the camera's 's trajectory and movement patterns. Using the powerful VLM (7]) we analyze the sequence of inter-frame trans- formations to identify key motion types like panning, tilting, zooming, and dolly movements. This kinematic information is then used to generate natural language descriptions, potentially leveraging template-based generation or LLM prompting, to convey how the viewpoint changes over time.
Caption rephrasing. Following the generation of three distinct caption types (object, scene, and camera motion), Large Language Model (L.LM) 61 is employed to jointly process them. This step aligns the dynamic content descriptions across caption types and refines their phrasing to enhance overall consistency and readability.
Hluman-in-the-lop quality review. To provide a faithful comparison against larger pretrained models, human evaluation was used. Addressing persistent errors from source annotation inaccuracies, we implemented an iterative human-in -ine--i--tttoooy verification during caption construction identify errors, trace sources, and revise/remove problematic data.
4 Experiments
In this section I we present experimental results to evaluate the robustness of our DynamicGen pipeline. Due to the page limit, we direct readers to the appendix for implementation details, more qualitative results, and more experimental analyses.
4.1 Video Depth Estimation
To evaluate video depth estimation accuracy, we assess several baseline methods, including metric depth predictors such as Metric3Dv2 (72), Depth-Pro (36), DepthCrafter (37], and Unidepth [39].
which operate without scale or shift alignment. We also consider joint 4D modeling approaches, in- cluding MonST3R (63] and RCVD 73.. Evaluations are conducted on the Sintel [33] and KITTI (75] datasets, following standard protocols 37 by applying global shift and scale alignment to the pre- dicted depth maps. We report absolute relative error (Abs Rel) and the percentage of inlier points (8 1.25), with all methods undergoing least-squares alignment in disparity space. As shown in Tab. 2 DynamicGen achieves the best overall performance across all datasets and evaluation metrics. In particular, it consistently outperforms prior approaches in both absolute accuracy and geometric consistency, demonstrating strong generalization to diverse and dynamic scenes. As illustrated in Fig. 5 MonST3R consistently struggles with object geometry reconstruction, producing distorted
All research undertaken at Meta Al was limited to general guidance on model architectural design. Meta did not participate in any model training activities. Fan, Z. contributed to this project prior to the NeurlPS submission deadline.
8

---

Monst3r
Uni4D
Ours
Monst3r
Uni4D
Ours
Monst3r
Uni4D
Ours
Monst3r
Uni4D
Ours
Monst3r
Uni4D
Monst3r
Uni4D
Ours
Figure 5: Visual comparisons of 4D reconstruction on in-the-wild data.
Table Video depth evaluation on Sintel and KITTI datasets. Bold and underlined values indicate best and second best results.
Sintel
KITTI
Alignment
Category
Method
Abs
51.251
Abs
81.251
Per-sequence scale
Joint depth pose
Monstär 63
0.344
55.9
0.089
91.4
Uni4D
55
0.289
64.9
0.086
93.3
Single-frame depth
Depth-pro
36
0.280
60.5
0.080
94.2
Metric3D
72
0.205
71.9
0.039
98.8
Per-sequence
Video depth
DepthCrafter
37
0.231
69.0
0.112
88.4
scale & shift.
Robust-CVD 73
0.358
49.7
0.182
72.9
Joint vidco depth & pose
CasualSAM 74
0.292
56.9
0.113
88.3
Uni4D [55
0.216
72.5
0.098
89.7
(OnamieGen(Ours)
0.205
72.9
0.091
91.2
shapes and noisy dynamic masks. Uni4D also exhibits mask imprecision. DynamicGen, however, achieves the cleanest dynamic segmentations and the strongest dynamic/static reconstructions.
4.2 Camera Pose Estimation
We evaluate our method against recent dynamic scene pose estimation approaches, including learning- based visual odometry (e.g., LEAP-VO 76). DPVO (77) and joint depth-pose optimization methods (e.g., Robust-CVDD 73.. CasualSAM 74.. MonST3R
63 Experiments are conducted on the Sintel [3) and TUM-dynamics 778 datasets, following LEAP-VO's split for Sintel and subsampling the first 270 frames of TUM-dynamics, as done in MonST3R. Camera trajectories are aligned using 79), and we report Absolute Trajectory Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot). As shown in Tab. DynumicGen consistently achieves state-of-the-ar results across all metrics and datasets, outperforming existing methods in both translation and rotation accuracy.
Umeyama alignment

---

Table 3: Camera Pose Evaluation Sintel and TUM-dynamiic datasets. Bold and underlined values indicate best and second best results.
Sintel
TUM-dynamics
Category
Method
ATEL
RPE trans.)
RPE rot.
ATE]
RPE trans.
RPE rot.
DPVO
0.171
0.063
1.291
0.019
0.014
0.406
Pose only
LEAP-VO
76
0.035
0.065
1.669
0.025
0.031
2.843
Robust-CVy CVD 73
0,368
0.153
3,462
0.096
0.027
2.590
CasualSAM 74
0.137
0.039
0.630
0.036
0.018
0.745
Joint depth
Monstär 63
0.108
0.043
0.729
0.108
0.022
1,371
pose
Uni4D 55
0.1
0.032
0.338
0.012
0.004
0.335
Oynam))Gen(Ours)
0.108
0.029
0.282
0.012
0.004
0.331
4.3
Camera Intrinsics Estimation
Camera intrinsics are typically unavailable for most casual videos, especially those sourced from the Internet. However, accurate intrinsics critical for reliable pose estimation and 3D reconstruction. To assess this, we evaluate focal length estimation accuracy on the Sintel dataset, with results summarized in Tab. UniDepth predicts depth and focal length from single image, while Dust3r processes sequential frames but is trained under classical multi-view settings and fails to generalize well to dynamic scenes. In contrast, DynamicGen demonstrates strong generalization to dynamic content and achieves the best performance in both Absolute Focal Error (AFE) and Relative Focal Error (RFE), setting new state-of-the-art for focal length estimation in unconstrained video scenarios.
Table 5: Dynamic Scene Caption evaluation.
Table 4: Camera intrinsics estimation.
Method
Acc.t
Com.1
Con.t
Rel.1
Avg.
Method
DO
79.28
76.65
73.23
80.33
77.37
UniDepth 33
447.4
0.357
SAKFE
80.23
77.46
74.01
81.45
78.29
Dust3r
434.0
0.364
HP
82.57
81.42
71.17
82.56
79.43
(OnamicGen(Ours)
413.1
0.241
Rephrasing
82.48
80.50
71.86
83.27
79.53
COT
84.38
82.09
75.87
85.56
81.97
4.4
Caption Quality Evaluation
To assess caption quality, we sampled 100 videos from the SA-V dataset 33 As presented in Table 5
experimental results indicate that integrating semantic-aware key frame extraction (SAKFE).
hierarchical prompting (HP), caption rephrasing, and Chain-of- ThhTuughht (CoT) prompting [0) significantly enhances the quality of dynamic scene captions generated by Vision-Language Models (VLMs). We evaluated caption quality using the LLLM-asJJude metric G-VEval [81), conducting ten independent evaluations ensure robust average results. The resulting captions were demonstrably more accurate, complete, concise, and relevant than those produced by direct output (DO), confirming the effectiveness of these strategies for improving caption quality in this task.
Conclusion
In this work, we address key limitations in traditional 4D data curation regarding scalability, physical realism, and modality diversity. We introduce DynamicGen, an automated pipeline leveraging foundation models for video filtering, metric-scale geometry and motion recovery, and hierarchical semantic captioning from raw videos. DynamicGen's capabilities are validated through standard benchmarks on video depth and camera pose/intrinsies estimation, qualitative analyses diverse web videos, and human/LLM based evaluations confirming caption quality. Utilizing DynamicGen, construct Dynamic Verse, large-scale 4D dataset with over 100K dynamic scenes and rich physically grounded multimodal annotations. Together, this work offers a scalable 4D data generation methodology and comprehensive new resource to advance 4D scene understanding.
10

---

References
[1] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 343-352, 2015.
[2] Chao Yu, Zuxin Liu, Xin-Jun Liu, Fugui Xie, Yi Yang, Qi Wei, and Qiao Fei. Ds-slam: semantic visual slam towards dynamic environments. 2018 TEEE/RSJ international conference intelligent robots and systems (IROS), pages 1168-1174, IEEE, 2018.
[3] Berta Bescos, José M Fácil, Javier Civera, and José Neira. Dynaslam: Tracking, mapping, and inpainting in dynamic scenes. IEEE robotics and automation letters, 3(4):4076-4083, 2018.
[4] Linhui Xiao, Jinge Wang, Xiaosong Qiu, Zheng Rong. and Xudong Zou. Dynamic-slame Semantic monocular visual localization and mapping based on deep learning dynamic environment. Robotics and Autonomous Systems, 117:1-16, 2019.
[5] Jiahui Huang, Sheng Yang, Zishuo Zhao, Yu-Kun Lai. and Shi-Min Hu. Clusterslam: A slam backend for simultancous rigid body clustering and motion estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875-5884, 2019.
[6] Jesse Morris, Yiduo Wang, and Viorela lla. The importance coordinate frames in dynamic slam. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 13755-13761. IEFE, 2024.
[7] Haoyu Zhen, Qiao Sun, Hongxin Zhang, Junyan Li, Siyuan Zhou, Yilun Du, and Chuang Gan. Tesseract: Learning 4d embodied world models. arXiv preprint arXiv:2504.20995, 2025.
[8] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1, 2023.
[9] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint 0rXiv:2309..31000 2023.
[10] Panwang Pan, Zhuo Su, Chenguo Lin, Zhen Fan, Yongjie Zhang, Zeming Li, Tingting Shen, Yadong Mu, and Yebin Liu. Humansplat: Generalizable angle-image human gaussian splatting with structure priors. Advances in Neural Information Processing Systems. 37:74383-74410, 2024.
[11] Hezhen Hu, Zhiwen Fan, Tianhao Wu, Yihan Xi, Seoyoung Lee, Georgios Paylakos, and Zhangyang Wang. Expressive gaussian human avatars from monocular rgb video. arXiv preprint arXiv:2407.03204 2024.
[12] Samarth Sinha, Roman Shapovalov, Jeremy Reizenstein, Ignacio Rocco, Natalia Neverova, Andrea Vedaldi, and David Novotny. Common pets 3d: Dynamic new-view synthesis of real-life deformable categories. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 4881-4891, 2023.
[13] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19383-19400, 2024.
[14] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik. Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 2446-24540 2020.
[15] Yang Zheng, Adam W Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J Guibas. Pointodyssey: large-scale synthetic dataset for long-term point tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19855-19865, 2023,
11

---

[16] Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: regh-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[17] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dynamicstereo: Consistent dynamic depth from stereo videos. In Proceedings of the IEEE/CVF Conferencee on Computer Vision and Patlern Recognition, pages 13229-13239, 2023.
[18] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view stereopsis. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2821-2830, 2018.
[19] Guangcong Zheng, Teng Li, Xianpan Zhou, and Xi Li. Realcam-vid: High-resolution video dataset with dynamic scenes and metric-scale camera movements. arXiv preprint arXiv:2504..0 02 2025.
[20] Chris Rockwell, Joseph Tung, Tsung- Yi Lin, Ming- Yu Liu, David F Fouhey, and Chen-Hsuan Lin. Dynamic camera poses and where to find them. arXiv preprint 1xXix:2504.17885 2025.
[21] Hanyu Zhou and Gim Hee Lee. Llava-4d: Embedding spatiotemporal prompt into Imms for 4d scene understanding, 2025.
[22] Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, Wenping Wang, and Yuan Liu. Diffusion as shader: 3d-aware video diffusion for versatile video generation control. arXiv preprini arXiv:2501.0385 2025.
[23] Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, and Hanspeter Pfister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[24] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Dorkine-Homuugg and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 17X4v:704.000 2017.
[25] Linjie Yang. Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5188-5197, 2019.
[26] Weiyao Wang, Matt Feiszli, Heng Wang, and Du Tran. Unidentified video objects: A benchmark for dense, open- world segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10776-10785, 2021.
[27] Pavel Tokmakov, Jie Li, and Adrien Gaidon. Breaking the" object" in video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22836-22845, 2023.
[28] Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1674-1683, 2023.
[29] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2023.
[30] Nikhila Ravi, Valentin Gabeur, Yuan- Mlan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint
2024.
[31] Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. Miradata: A large-scale video dataset with long durations and structured captions. Advances Neural Information Processing Systems, 37:48955-48877 2024.
12

---

[32] Yuheng Qiu, Chen Wang, Wenshan Wang, Mina Henein, and Sebastian Scherer. Airdos: Dynamic slam benefits from articulated objects. In 2022 International Conference on Robotics and Automation (ICRA), pages 8047-8053. IEFF, 2022.
[33] Daniel J Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. Computer Mision-ECCV 2012: 12th European Conference on Computer Vision Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pages 61 1-2... Springer, 2012.
[34] Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040-4048, 2016.
[35] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holyn- ski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint arXiv:2412.0962/ 2024.
[36] Alcksei Bochkovskii, AmaAGI Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. arXiv preprint arXiv:2410.02073, 2024.
[37] Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024.
[38] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9492-9502, 2024.
[39] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-101 16, 2024.
[40] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao.
Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-10381, 2024.
[41] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[42] Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding image matching in 3d with mast3r. In European Conference on Computer Vision, pages 71-91. Springer, 2024.
[43] Jianyuan Wang. Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
[44] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for statc-of-the-arr multimodal models. arXiv preprint arXiv:2409.1 17111116.....6. 2024.
[45] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[46] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. European Conference on Computer Vision, pages 38-55. Springer, 2024.
13

---

[47] Rawal Khirodkar, Timur Bagautdinov. Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, and Shunsuke Saito. Sapiens: Foundation for human vision models. In European Conference on Computer Vision, pages 206-228. Springer, 2024.
[48] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8121-8130, 2022.
[49] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[50] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference Computer Vision, pages 18-35. Springer, 2024.
[51] Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410. 11831,, 2024.
[52] Luigi Piccinelli, Christos Sakaridis, Yung-Hsu Yang, Mattia Segu, Siyuan Li, Wim Abbeloos, and Lue Van Gool. Unidepthv2: Universal monocular metric depth estimation made simpler. arXiv preprint arXiv:2502.20110 10, 2025.
[53] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[54] Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi, Jiashi Feng, and Ming- Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:2501.0400) 2025.
[55] David Yifan Yao, Albert Zhai, and Shenlong Wang. Uni4d: Unifying visual foundation models for 4d modeling from single video. arXiv preprint arXiv:2503.21700 2025.
[56] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 11-25278-25200 2022.
[57] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-fitteed 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
[58] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018
[59] Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-bail bassd image text dataset for multimodal multilingual machine learning. Pro- ceedings of the 44th international ACM SIGIR conference research and development in information retrieval, pages 2449, 2021.
[60] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osear Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 13142-13153. 2023.
[61] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li,
14

---

Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2rXiv:2412.1555 15115, 2024.
[62] Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild. In European Conference Computer Vision, pages 523 223-542. Springer, 2022.
[63] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024.
[64] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung-Yo Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23. 2023.
[65] Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12877 2023.
[66] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. In European Conference on Computer Vision, pages 289-310. Springer, 2024.
[67] Wanhua Li, Renping Zhou, Jiawei Zhou, Yingwei Song, Johannes Herter, Minghan Qin, Gao Huang, and Hanspeter Pfister. 4d langsplat: 4d language gaussian splatting via multimodal large language models. arXiv preprint arXiv:2503. 10437, 2025.
[68] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin- Ying Lee, Jian Ren, Ming- Yang, et Panda-70ml Captioning 70m videos with multiple cross gross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13320-13331, 2024.
[69] Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning. arXiv preprint arXiv:2504. 16072, 2025.
[70] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong. Pan Zhang, Yuhang Zang. Zehui Chen, Haodong Duan, Zhenyu Tang, Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472- 19495, 2024.
[71] Zhiqiu Lin, Siyuan Cen, Daniel Jiang. Jay Karhade, Hewei Wang, Chancharik Mitra, Tiffany Ling, Yuhan Huang, Sifan Liu, Mingyu Chen, Rushikesh Zawar, Xue Bai, Yilun Du, Chuang Gan, and Deva Ramanan. Towards understanding camera motions in any video. arXiv preprint 2504 15376, 2025.
[72] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[73] Johannes Kopf. Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611-1622 2021.
[74] Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William Freeman. Structure and motion from casual videos. In European Conference on Computer Vision, pages 20-37. Springer, 2022.
[75] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(11):1231-123) 2013.
15

---

[76] Weirong Chen, Le Chen, Rui Wang, and Marc Pollefeys. Leap-vo: Long-ten term effective any point tracking for visual odometry. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19844-19853, 2024.
[77] Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry, Advances in Neural Information Processing Systems, 36:39033-333 130031, 2023.
[78] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 20/2 IEEE/RSJ international conference on intelligent robots and systems, pages 573-580. IEEE, 2012.
[79] Shinji Umeyama. Least-squares estimation of transformation parameters between two point patterns. IEEE Transactions on Pattern Analysis & Machine Intelligence, 13(04))376-380, 1991.
[80] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
[81] Tony Cheng Tong. Sirui He, Zhiwen Shao, and Dit- Yan Yeung. G-veval: A versatile metric for evaluating image and video captions using gpt-4o. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7419-7427, 2025.
[82] Olga Sorkine and Marc Alexa. As-rigid-as-possible surface modeling. In Symposium Geometry processing, volume 4, pages 109-116 16. Citeseer, 2007.
[83] Wei-Chiu Ma, Shenlong Wang, Rui Hu, Yuwen Xiong, and Raquel Urtasun. Deep rigid instance scene flow. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 361 3614-322, 2019.
[84] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable neural models from many casual videos. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2022.
[85] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: reconstruction from a single video. arXiv preprint arXiv:2407.13764 2024.
[86] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensionall representation for topologically varying neural radiance fields. arXiv preprint 10x.:2206..3228 2021.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction sections offer comprehensive discussion of the manuscript's context, intuition, and ambitions, as well as its contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed by authors at the end of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

For each theoretical result, the paper provides the full set of assumptions and complete (and correct) proof.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The pipeline the methods and the details experiments are presented with corresponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All utilized data are sourced from open-access platforms. The code, which will be made publicly available, is uploaded as zip file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
coolic//guides/Codetttbm.ss wtw.eeeewuumiisioooooooooooo for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppppclggua.aeaeaeccuuep
Saleee//CoSSSSbmsssS.S.. www.ssssiicgg.oooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The pipeline of the methods and the details of experiments are presented with corresponding reproducible credentials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The results contain the standard deviation of the results over several random
runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details credentials.
of experiments are presented with corresponding reproducible
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. cttpriips.cccpppppeooo / http://wwwpp.p..ppp../.chctcccaa
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original owners of assets, including data and models, used in the paper, are properly credited and are the license and terms of use explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssippsiiiie thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper well documented and provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: as controller
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
23

---

yynamie VERSE
Examples
Annotation
OOO
Dynamic Mask
Video Depth
Camera Pose
Object Mask
Object Caption
Scene Caption
Camera Caption
Intrinsics
Category
Figure 6: Dynamic Verse dataset.
A
Appendix
In the appendix, we provide more results and analysis and summarize them follows:
In Section A.
we introduce the broader impact of our Dynamic Verse framework.
In Section we supplement details of dynamic bundle adjustment.
In Section
we ablate the different components for dynamic bundle adjustment.
In Section
we provide additional experiments on generated hierarchical captions.
In Section we provide more qualitative results of dynamic bundle adjustment.
Section we provide inference speed and computational cost for DynamicGen.
Section we provide the limitation.
A.1 Broader Impact
The introduction of Dynamic Verse, with its large-scale, physically-aware. and multimodally annotated 4D dataset derived from real-world videos, is set to significantly influence several advanced research areas. Our framework's unique ability to capture metric-scale geometry, real-world motion, instance- level semantics, and descriptive captions offers an unparalleled resource that can catalyze progress in the following domains:
Dynamic 4D Scene Generation: Dynamic Verse offers paradigm shift for Dynamic 4D Scene Generation. Current methods often rely on limited simulators or struggle to realistically portray complex real-world physics and motion from internet-sourced content. By accurately interpreting real-world dynamics from monocular videos and integrating window-based Bundle Adjustment with global optimization, Dynamic Verse converts long video sequences into a comprehensive 4D multimodal format, capturing ine-grained dynamic information. This rich, real-world data provides unparalleled training ground for generative models, leading to the creation of highly

---

realistic, physically plausible, and semantically coherent dynamic 4D scenes. This has profound implications for high-fidelity content creation in entertainment (e.g., movies, games), realistic vir- tual environments for training and simulation (e.g., disaster response, architectural visualization), and the synthetic generation of diverse data for further Al research, helping to overcome privacy and data collection limitations.
4D Vision-Language Models (4D-VLM): Dynamic Verse will greatly accelerate the development of sophisticated 4D Vision-Language Models that can reason about space, time, and semantics concurrently. Existing VLMs often operate 2D images or short video clips with limited 3D awareness. Our framework provides a unique combination of metric-scale 4D geometry, real-world dynamic motion, and comprehensive textual descriptions for long video sequences, allowing 4D- 4D-VLMs to learn intricate relationships between evolving 3D scenes and natural language narratives. Such models could enable more advanced human-agent interaction, where agents can provide detailed textual explanations of complex dynamic events they perceive in 4D, understand nuanced, temporally extended instructions involving interactions within a 3D) space. This could revolutionize areas like Al-powered video captioning, temporal question answering in 3D, and the development of embodied AI agents that communicate their understanding of the dynamic world with human-like richness.
4D Language-Gii Groundedd Gaussian Splatting (4D-LangSplat): Dynamic Verse offers founda- tional dataset for advancing 4D-LangSplat methodologies. While current 4D Gaussian Splatting techniques excel at novel view synthesis of dynamic scenes, their integration with language for semantic understanding and manipulation still nascent. Our dataset, rich with 800K+ instance masks and holistic descriptive captions directly linked to evolving 3D structures and motions at physical scale, empowers 4DD-LangSpla models. This will enable the development of systems that can not only reconstruct dynamic scenes with high fidelity but also allow users to query, edit, and interact with these 4D representations using natural language. For instance, users could ask an agent to "track the red car that just turned left" or "remove the person walking in front of the fountain," " with the model understanding both the spatial dynamics and the semantic context. This can significantly enhance applications in robotics, augmented reality, and interactive content creation by bridging the gap between visual perception and linguistic instruction in dynamic 3D environments.
In summary. DynamicVerss Verse is poised to serve crucial catalyst, providing the data and framework necessary bridge the gap between 2D understanding and true 4D world modeling, thereby fostering advancements in semantic scene understanding, dynamic object interaction, multimodal reasoning, and realistic content generation.
A.2 Details of Dynamic Bundle Adjustment
Camera parameterization In Eq. (1), & SE(3) represents the camera poses as rigid transforma- tions. Rotations are parameterized using so(3) rotation vectors, which offer a minimal representation facilitating direct optimization.
Static Area Bundle Adjustment Term In Eq. (2). the bundle adjustment energy CBA Xetatie)
measures the consistency between the pixel-level correspondences and the 3D structure of static scene elements. Given the input pixel tracks Z = {Z k= and video segmentation M = {M'))
we filter all tracks corresponding to statie areas and minimize the distance between the projected pixel location and the observed pixel location:
CBA (P, X M.
Σ Σ Z.EM
uk. ZR.,
TA (Xa, E,)
(5)
where X₁ is the k-th 3D point, ZR is the k-thh 3D point's corresponding pixel track's 's 2D coordinates time f, Wk.e (0, 1) is a visibility indicator and k the perspective projection function.
Camera Smoothness Prior In Eq. (2), given the video input, temporal smoothness prior is imposed on camera poses. This prior penalizes abrupt changes in relative pose, defined as { = STA Er. We adaptively reweight this term based on the magnitude of the relative motion. Specifically, a larger relative motion results in reduced penalty on its change rate, while smaller relative motion
25

---

incurs a higher penalty. Formally, this is expressed as:
Ceam (P)
Σ Coo (R (R, + Σ ( (T(-1,,(41)
where Co
2 Ind [nd] rad RR 1-21l
and (trns(((-11442))
[ld - 1- - - 1
-- rad converts the rotation matrix into absolute radians.
Non-Rigid Bundle Adjustment Term In Eq. (3), for dynamic objects, we impose nonrigid bundle adjustment term, ENK (Xaya), which measures the discrepancy between the dynamic point cloud and pixel tracklets. Here, each pixel tracklet corresponds to dynamic 3D point sequence, (Xg,)), optimized for each observed tracklet:
(NRyddy P. P.
Σ
WkkrllZZ:
TK (Xt,t, Et)1
(6)
MA KcEM
where Xg, E I is the k-th dynamic point's location at t.
Dynamic Motion Prior In Eq. (3), Cmotion (dyn) a regularization term that encodes the charac- teristics of the dynamic structure. It contains two prior terms that are used to regularize the dynamic structure, both of which have demonstrated effectiveness previous work.
mootoo dyn) Carap dyyn)) Cmmooth (Xdya).
Carap represents as- rigid-as-possiieee (ARAP) prior [2 designed to penalize extreme deformations that compromise local rigidity. Specifically, for each dynamic control point its nearest neighbors are identified using k-Nearest Neighbors (KNN) on the remaining tracks. We then enforce that the relative distances among these neighboring pairs remain consistent, preventing sudden changes
Carap
Σ
Wkm
| [d(X,,,, X,,))
(8)
(k,m)
where d(.) is the 1.2 distance and Wkm,t 1 if all relevant points are visible.
Csmooth a simple smoothness term that promotes temporal smoothness for the dynamic point cloud:
Csmoothh ΣED
Σ Wk. Xg, Xx.441122-
X.EX.sn
Despite simplicity, both motion terms are crucial our formulation, as they significantly reduce ambiguities in 4D dynamic structure estimation, which is highly ill- ill-posed. Unlike other methods, do not assume strong modelbased motion priors, such as rigid motion [83), articulated motion [84), linear motion basis [85
Optical Flow Prior In Eq. (4), we also use flow projection loss to encourage the global pointmaps to be consistent with the estimated flow for the confident, static regions of the actual frames. More precisely, given two frames t, t', using their global pointmaps, camera extrinsics and intrinsics, compute the flow fields from taking the global pointmap X,, assuming the scene is static, and then moving the camera from to f'. We denote this value pglobal: t-+t' similar to the term defined
encourage this to be close to the in the confident static region computation above. Then global: c-+1' staic
estimated flow, Ft-t' in the regions which are confidently static parameters:
according to the global
est
Crow (Xstatic)
Σ
Σ
X global: t-+z'
x-ar'
F F
I
(10)
cam
W.EW C'EW*
where- indicates element-wise multiplication. Note that the confident dynamic mask is initialized using the foundation models as described in Sec. 3.3. During the optimization, we use the global static pointmaps and camera parameters to compute gelobal and update the confident dynamic mask.
26

---

A.3 Ablation Study on Different Components for Dynamic Bundle Adjustment
Our dynamic BA pipeline introduces three key components absent in prior work like Uni4D [55]. which systematically improve the decomposition of static/dynamic elements and global consistency:
(a) Epi-Mask-Based Dynamics Filtering: We introduce geometric filtering step using an epipolar-based mask ("Epi-mask") to achieve a cleaner separation between static background and dynamic foreground pixels before bundle adjustment. This leads to more stable camera pose estimation and background reconstruction.
(b) VLM-Based Semantic Dynamics Analysis:
We leverage a Vision-Language Model (VLM) for a high-level, semantic understanding of motion. This enables intelligent, motion-aware keyframe extraction and provides robust masks for dynamic objects, significant improvement over purely geometric or flow-based segmentation.
(c) Optical Flow-Based Sliding Window Global Refinement: To address error accumulation and temporal drift common in long videos, implement global refinement strategy over sliding window. This enforces long-range temporal consistency, correcting errors that frame-by-fin by-ffrmm local BA approach would miss.
Table 6: Components Ablation on Sintel.
Ablations
(a)
(b)
(c)
ATE/
irans
RP
Abs|
01.251
Baseline
0.114694
0.032125
0.347920
0.216433
0.725167
Ablation-1
0.114065
0.032250
0.335198
0.215058
0.726943
Ablation-2
0.11053
0.033122
0.334005
0.210339
0.722999
Ablation-3
0.114694
0.032125
0.347920
0.214282
0.724084
Ablation-4
0.108459
0.028906
0.281979
0.205892
0.727616
Ablation-5
0.114065
0.032250
0.335198
0.214143
0.725534
Ablation-6
0.110530
0.033122
0.334005
0.207329
0.725784
DynamicGen (Ours) v
0.108459
0.028906
0.281979
0.204574
0.728961
A.4 Additional experiments on generated hierarchical captions.
We performed three distinct experiments to validate the high quality of our hierarchical semantic
annotations:
(a) Object-Levvl Semantics via 4D-LangSplat [67] To validate the annotations produced by our DynamicGen framework, we performed time-sensitive querying experiment using 4D-LangSplat model. For this evaluation, we trained the model on the "americano" scene from the HyperNeRF dataset and benchmarked it against a re-implemented 4D-LangSplat baseline. The results, presented in Tab. 7, demonstrate that our approach yields substantial gains in Accuracy and volumetric Intersection over Union (vloU). This superior performance confirms that our precise object masks and labels are highly effective for demanding multi-modal applications.
Table 7: Quantitative comparisons of time-sensitive querying on the HyperNeRR 866 dataset.
Method
americano
A%)
vloU(%)
4D-LangSplat* [67
53.84
27.55
DynamicGen
64.42
51.65
(b) Scene-Level Semanties via G-VEval 81: To rigorously assess our scene-level captions, we moved beyond single-score metrics and employed a more granular evaluation using the ACCR framework in G-VEval benchmark. This approach provides a comprehensive, multi-dimensional assessment of caption quality across four key axes: Accuracy, Completeness, Conciseness, and Relevance. On random sample of 100 videos from SA-V data, our generated captions demonstrated high performance across all four criteria, as detailed in the Tab. The strong
27

---

performance across these metrics confirms that our captions are not only factually accurate and relevant the video content, but also complete in their coverage of events and efficiently concise. This robust, multi-faceted quality makes them highly suitable and reliable for demanding downstream applications.
Table 8: Evaluation of generated captions using the ACCR framework from G- G-VEv.l.
Evaluation Criteria
Accuracyt Completeness Concisenesst
Relevancef
Average?
Scene-Level Captions
84.38
82.09
75.87
85.56
81.97
(c) Camera-Level Semantics via Human Study: We conducted a formal human study to quantitatively analyze the quality of the final camera motion captions. Following prior work (7], we asked human evaluators to rate our captions on three criteria: (1) Clearness (clarity of information). (2) Conciseness (brevity without losing clarity), and (3) Grammar & Fluency. On a sub-sample of 88 videos from our dataset (i.e., filtered DAVIS), our captions performed excellently. The results, presented in Tab. |howwed that over 60.22% of the captions were rated as both clear and fluent, while also receiving high scores for conciseness. This confirms the effectiveness of our generation and quality control process.
Table 9: Human evaluation results for the generated camera captions. Scores indicate the percentage of captions that met each quality criterion.
Human Evaluation
Rated as Clear
Rated as Fluent
Rated Concise
Camera Captions
85.22%
89.77%
67.04%
A.5
More qualitative results of dynamic bundle adjustment
We present additional qualitative reconstruction results in Fig. g. demonstrating the generalizability and performance of our pipeline on real-world data.
A.6 Inference Speed and Computational Cost for DynamicGen
For a reproducible analysis of computational performance, we processed the entire Sintel training set (23 videos) NVIDIA H20 GPUs. A detailed breakdown of the average processing time and peak VRAM consumption for each component of our pipeline is provided in Table I0
Table 10: Computational Cost Analysis.
Module
Hardware Used
Avg. Time / Sintel Video (mins)
Peak VRAM (GB)
Notes
Monion-aware Keylrame Extraction
H20 GPU
representative frames
2. VLM-Based Semanti Analysis (Qwen-Ve VL)
2x 1120 GPU
w1.6
+00
Identifies dymamic elements
Moving Object Segmentation (SA2VA)
1x H2D GPU
~0.8
<3
Per-ohject video segmentation
4. Dynamic Rundle Adjustment
1x CPU Core
1x GPU GPU
~122
<0
Main time bottleneck
Moving Object Captioning
2x 1120 GPU
~20
24
Object level descriptions
Dynamic Scene Captioning
1120
Scanc level descriptions
Camera Motion Captioning
2x 1120 GPU
~20
40
Camera level descriptions
8. Cigeion Rephrasing
1x H20 GPU
~00
~24
L.M refinement for consis-
tency and conciseness
Total (per video)
H20 GPU
-237
Prak VRAM,
A.7 Limitations
Despite its considerable capabilities, Dynamic Verse exhibits several inherent limitations. First, its reliance in-the-wild internet videos introduces significant noise and quality variance. This can compromise the fidelity of metric-scale geometry and motion recovery, particularly in complex, cluttered, or occluded scenes that fall outside the typical distribution of the foundation models' training
28

---

Object Category: Roating
Object Categery: Public Bus
Object Categery: Helling Woren
longo lear with minimal shaling. wiil bucmme larger
Castrea Caprions
carnco smoothle
and wdate publi bas with marra staliing from ligh-
The aroothie para the right, traddigg blue
Caption:
Carnerr Captiem:
canera smoothly trades right, mantakling strady Touus se wansn fullow nigrtight teardiins
sale. Trword
the time
conthnse tradeng her reoversed the caneera movereent vleuing exporionen.
cijject Daption:
lorgo, robust boar with thick choggy cost brown
lrge, gutli bud will slok whitto and thks extoror movss steed aling med alongened body prominant wheditted dide véncies,, providing
with vigg mily herehed baok and a posernt atriks The
Cese newr enenie soata anaangement the loe's
frowed conkdeen cndde She poentell willdin Invesingth havk dinse frrsr geettly with
Objest Caplion:
woman hut
tlcncing Usc dane Au
loge riced is-
the dolboodd surnic ancnnssing spilits, The menosmaail cor th
Litckla ki de cordilantig leftu, Po il progresss, the bux nore the prvement The ture derign drroertlons, with on
of la coel. the boats prus pnoo to coner clig, und
oene purmone atfinn
dreth
roar feceooent through growenos, manserents faad parportts rebhoyying the mntrnnyy mixesitty roperted invegnattionn
cxpectly
yedoow hag the velrep olighity with werhh stee, lee while minvin with purmoss nvurtwiann clieily paano.
Scene Caption:
un streciily with sateral by tiea, arpe fosree bedeppaant vettow store oiile sbricble
bucre Captkere
A wormon a bodk dress and white stoakers waller
tha stik
doually oa
cogetaticn
direct under satumi portioedd digparity Iooggound traawds 37 condont coced
yollon las Apsck moves sluullly from lull rigit with
kerwend lectin oedeaaioon tidi gaiees grass thee pocitoned
slce. The loae ricea el sprouats specal sboip a broar eesiecrie dedtav from the call wii driding wfbeent to olar ents banae stnadtues, which compl metion The some appears senduary, babbut
satn didieen frmm pecserionn bengeeeds, tweccees anal
noar the path rightt bonch within prodiiity
frot, cuch ap cortic 1y rad danns and podesiitnn while baskgrond kembonn cloello, idnne. tecrientoon the enverc smooth, rile complant typical utban muttin adtivity.
spasu for walking neiling. The cnneromment namalits
et centr bed
suited for felsurtly adtvittes.
cxiibes satural crosonor: in contied yot rust
Figure 7: Examples captions on DAVIS dataset.
data. Second, the substantial computational overhead required to process long video sequences with large-scale models presents practical barrier to real-time performance and scalable deployment. Finally, while extensive, the dataset cannot exhaustively capture the long tail of real-world phenomena. Consequently, the model's generalization truly novel environments is fundamentally tethered to the intrinsic biases and capabilities of its underlying foundation models.
These limitations raise Al-safety concerns: (i) privacy and security risks, since metric-scale recon- structions from web videos can expose sensitive interiors or critical infrastructure and facilitate covert mapping surveillance; and (ii) miscalibrated confidence under distribution shift, producing plausible but erroneous geometry and dynamics that misguide downstream robotic or AR planners. Biases and licensing gaps in foundation models and web data may further perpetuate representational harms and legal or IP issues. A practical mitigation is to prefilter ineligible videos using policy rules and automated detectors (e.g., content with PII, sensitive interiors or infrastructure, minors, or restricted licenses).
29

---

Figure 8: Qualitative Results on in-the-wild data. We show qualitatively some of our reconstruction results on in-the-wild data. For full reconstruction, please refer to our attached supplementary webpage.
30

---

Video
SegAnyMo
GPT+DEVA
DynamicGen
MAAF ORG
Canon
SINOR
A
A
A
A A
LAAF.oRo
IAAF.OM ORG
MISRION RACEWAY
BENDOFF
Figure 9: Qualitative Results of moving object Segmentation. We show qualitatively some of our segmentation results on the Youtube-VIS dataset compared with other basclines.
31


Reviews:
[{"review_id": "W8tt5fsMd7", "reviewer": "Reviewer_6NBV", "summary": "To address the challenge of limited 4D multi-modal data, the paper introduces an automated pipeline that generates 4D annotations and corresponding textual descriptions for input videos by leveraging large vision models, optimization techniques, VLMs, and LMMs. Using this pipeline, the authors compile a large-scale video dataset with detailed annotations, including 4D infos and texture captions. To demonstrate the effectiveness of their approach, they conduct experiments on video depth estimation and camera parameter prediction, showcasing the advantages of their pipeline.", "questions": "1. It appears that the authors use VGGT to extract geometric features. Why not compare with more advanced VGGT variants in the camera parameter estimation task?\n2. How can the usefulness of the created dataset be demonstrated?\n3. What's the core difference compared to prior work Uni4D?\n4. How can it be ensured that the video filtering process does not affect the distribution of real-world videos?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The motivation to leverage existing foundation models to overcome the limitations of available 4D data is reasonable\n2. The overall pipeline is well designed, effectively leveraging foundation models to obtain initialization, decompose static and dynamic components during bundle adjustment, and incorporate global optimization for refinement. The engineering effort invested in integrating various models into a unified framework is also commendable, as it demonstrates the potential performance gains achievable by combining foundation models with traditional optimization-based methods.\n3. The experiments effectively demonstrate the versatility and effectiveness of the proposed pipeline across different tasks including depth estimation and camera parameters prediction\n\nWeaknesses:\n1. In the current paper, the goal is to leverage the pipeline to create a large-scale 4D dataset. However, there is no evidence provided to demonstrate that the resulting dataset is actually useful for training models, or that it introduces more challenging data patterns when used for evaluation. The paper does not show whether the annotations generated by the pipeline are of sufficient quality for downstream tasks. Furthermore, it remains unclear whether models trained on this dataset might easily collapse or overfit to certain error patterns.\n2. The proposed pipeline for geometry and correspondence is in fact quite similar to the prior work Uni4D, and the quantitative results show that their performances are also comparable. This somewhat diminishes the technical contribution of the pipeline component.\n3.It's unclear if the video filtering process ensures that the retained videos are not overly simplified, which could reduce their usefulness for dataset creation.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed responses to each point. The additional experiments on downstream tasks and the effective use of data from the constructed dataset help better demonstrate its usefulness. The detailed semantic annotations and improved geometry reconstruction further strengthen the dataset. I believe this dataset can benefit the community across a range of downstream tasks. I would like to raise my score.", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions.\n\n------\n> **Q1: Lack of DynamicVerse Dataset Validation.**\n\nWe thank the reviewer for this critical feedback. To directly demonstrate our dataset's utility, we conducted new experiments which provide strong evidence for the quality and effectiveness of the geometric and semantic annotations in DynamicVerse.\n\n**1. Geometric Validation: Fine-tuning MonST3R**：\n\nTo validate the quality and utility of our geometric annotations (i.e., metric-scale point maps and camera parameters), we used a subset of 1000 scenes from DynamicVerse to fine-tune the prominent model, MonST3R [1]. As shown in the table below, fine-tuning on our data leads to a performance improvement on the Sintel benchmark. This directly confirms that the geometry in DynamicVerse is accurate and beneficial for improving existing models.\n\n| Method                            | ATE↓  | RPE trans↓ | RPE rot↓ | Abs↓  | δ1.25↑ |\n| --------------------------------- | ----- | ---------- | -------- | ----- | ------ |\n| MonST3R (Original)                | 0.108 | 0.043      | 0.729    | 0.358 | 52.1   |\n| MonST3R + sub-DynamicVerse (Ours) | 0.108 | 0.042      | 0.727    | 0.335 | 58.9   |\n\n**2. Semantic Validation: Downstream Tasks and Direct Evaluation**：\n\nWe performed three distinct experiments to validate the high quality of our hierarchical semantic annotations:\n\n**(a) Object-Level Semantics via 4D-LangSplat [2]:** To demonstrate the effectiveness of our dynamic object annotations, we used them to train a 4D-LangSplat model. Our precise object masks and labels enabled the model to achieve superior results in language-grounded 4D scene understanding. This validates that our object-level annotations are highly effective for complex, multi-modal downstream tasks.\n\n| Metrics                                       | Acc(%) ↑ | vIoU(%) ↑ |\n| --------------------------------------------- | :------: | --------- |\n| w/ Original Semantic Annotations for Training |  53.84   | 27.55     |\n| w/ DynamicVerse Semantic Annotations (Ours)   |  64.42   | 51.65     |\n\n**(b) Scene-Level Semantics via G-VEval [3]:** We used the established G-VEval benchmark [3] to quantitatively assess our generated scene-level captions. On a random sample of 100 videos from filtered SA-V data [4], our captions achieved a high score, confirming their coherence, accuracy, and richness.\n\n| Evaluation Criteria   | **Accuracy**↑ | Completeness↑ | Conciseness↑ | Relevance↑ | Average↑ |\n| --------------------- | ------------- | ------------- | ------------ | ---------- | -------- |\n| DynamicVerse Captions | 84.38         | 82.09         | 75.87        | 85.56      | 81.97    |\n\n\nThe strong performance across these metrics confirms that our captions are not only factually accurate and relevant to the video content, but also complete in their coverage of events and efficiently concise. This robust, multi-faceted quality makes them highly suitable and reliable for demanding downstream applications.\n\n**(c) Camera-Level Semantics via Human Study:** We conducted a formal human study on 88 videos from our DAVIS subset to quantitatively analyze our camera motion captions. Following the criteria of Clearness, Conciseness, and Grammar & Fluency from prior work [5], the results were excellent: over **60.22%** of captions were rated as both clear and fluent while also receiving high scores for conciseness, confirming the effectiveness of our generation and quality control process. \n\n| Human Evaluation | Rated as Clear | Rated as Fluent | Rated as Concise |\n| ---------------- | :------------: | :-------------: | :--------------: |\n| Camera Captions  |     85.22%     |     89.77%      |      67.04%      |\n\n\n\n\n> **Q2: Limited Novelty, High Similarity to Uni4D.**\n\nWe thank the reviewer for the comparison to Uni4D. While both works tackle 4D reconstruction, our motivation and scope are fundamentally different. Uni4D's contribution is a geometric reconstruction framework; our goal is to build a scalable pipeline for a rich, multi-modal dataset with comprehensive annotations (geometry, masks, captions, etc.). This required significant new developments.\n\n**1. Core Geometric Contributions Beyond Uni4D** Our BA pipeline introduces three novel components absent in Uni4D for more robust reconstruction:\n\n- `(a) Epi-Mask-Based Dynamics Filtering`: For cleaner static/dynamic separation before BA, leading to more stable pose estimation.\n- `(b) VLM-Based Semantic Dynamics Analysis`: Uses a VLM for semantic motion understanding, enabling smarter keyframe extraction and more robust object masks.\n- `(c) Optical Flow-Based Sliding Window Global Refinement`: A sliding-window global refinement to enforce temporal consistency and correct drift.\n\nThe following ablation study quantifies the significant, measurable impact of each component.\n\n| Components Ablation on Sintel | (a)      | (b)      | (c)      | ATE↓         | RPE trans↓   | RPE rot↓     | Abs↓         | δ1.25↑       |\n| ----------------------------- | -------- | -------- | -------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Baseline                      |          |          |          | 0.114694     | 0.032125     | 0.347920     | 0.216433     | 0.725167     |\n| Ablation-1                    | &#10004; |          |          | 0.114065     | 0.032250     | 0.335198     | 0.215058     | 0.726943     |\n| Ablation-2                    |          | &#10004; |          | 0.11053      | 0.033122     | 0.334005     | 0.210339     | 0.722999     |\n| Ablation-3                    |          |          | &#10004; | 0.114694     | 0.032125     | 0.347920     | 0.214282     | 0.724084     |\n| Ablation-4                    | &#10004; | &#10004; |          | 0.108459     | 0.028906     | 0.281979     | 0.205892     | 0.727616     |\n| Ablation-5                    | &#10004; |          | &#10004; | 0.114065     | 0.032250     | 0.335198     | 0.214143     | 0.725534     |\n| Ablation-6                    |          | &#10004; | &#10004; | 0.110530     | 0.033122     | 0.334005     | 0.207329     | 0.725784     |\n| DynamicGen (Ours)             | &#10004; | &#10004; | &#10004; | **0.108459** | **0.028906** | **0.281979** | **0.204574** | **0.728961** |\n\n**2. A Major Contribution Beyond Geometry: Rich Semantic Annotation** Beyond geometry, our primary novelty is a sophisticated semantic annotation pipeline that sets us far apart from Uni4D. Unlike Uni4D's sparse labels, we generate three levels of rich, hierarchical captions (for objects, scenes, and camera motion). This is achieved with novel techniques like `(a) Semantic-Aware Keyframe Extraction (SAKFE)`, `(b) Hierarchical Prompting (HP)`, `(c) Caption Rephrasing` and `(d) Chain-of-Thought (CoT) Prompting`. Our ablation study below validates the effectiveness of each technique in improving the final caption quality.\n\n| Evaluation Criteria | Accuracy↑ | Completeness↑ | Conciseness↑ | Relevance↑ | Average↑  |\n| ------------------- | --------- | ------------- | ------------ | ---------- | --------- |\n| Direct Output       | 79.28     | 76.65         | 73.23        | 80.33      | 77.37     |\n| + (a) SAKFE         | 80.23     | 77.46         | 74.01        | 81.45      | 78.29     |\n| + (b) HP            | 82.57     | 81.42         | 71.17        | 82.56      | 79.43     |\n| + (c) Rephrasing    | 82.48     | 80.50         | 71.86        | 83.27      | 79.53     |\n| + (d) COT           | **84.38** | **82.09**     | **75.87**    | **85.56**  | **81.97** |\n\nIn summary, these validated contributions in both geometry and semantics result in a more robust and powerful pipeline for creating rich 4D multi-modal data, representing a significant step forward from prior work.\n\n\n\n> **Q3: Potentially Oversimplified Data due to Filtering.**\n\nThank you for this question. Our filtering process is a necessary quality control step, not a means of simplification. Our primary goal with filtering is to ensure annotation quality by removing videos that are intractable for current SOTA foundation models (e.g., those with chaotic non-rigid motion or extreme motion blur). This prevents noisy, unusable annotations from contaminating the dataset.\n\nCrucially, our quantitative analysis in the paper (Table 1) empirically proves that DynamicVerse retains the highest degree of dynamism compared to existing datasets, directly addressing the concern of oversimplification. Furthermore, our pipeline's modularity is a key strength; as better foundation models emerge in the future, researchers can seamlessly integrate them into our pipeline to process the very types of complex videos that are currently intractable, making our framework future-proof.\n\n\n\n> **Q4: Choice of Feature Extractor.**\n\nWe thank the reviewer for the question and clarify that there is a misunderstanding. VGGT is not used for our final, high-fidelity camera parameter estimation.\n\nIts role is strictly limited to an early, fast pre-filtering step to get a rough initial estimate of intrinsics. For the final, accurate results, we use our sophisticated multi-stage dynamic Bundle Adjustment (BA) pipeline. We deliberately do not use models like VGGT for the final, critical step due to their inherent limitations, such as a lack of metric scale and the requirement for input image cropping, which discards valuable information.\n\n------\nReference:\n\n[1] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.\n\n[2] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.\n\n[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025.\n\n[4] SAM 2: Segment Anything in Images and Videos. ICLR 2025.\n\n[5] CameraBench: Towards Understanding Camera Motions in Any Video.  Arxiv 2025."}, {"review_id": "NUqExbnBpN", "reviewer": "Reviewer_6prH", "summary": "This paper presents DynamicVerse, a new large-scale 4D dataset derived from real-world internet videos, and DynamicGen, the automated pipeline built to create it. DynamicGen annotates videos with a rich set of multi-modal data, including metric-scale geometry, consistent camera poses, and dense, hierarchical text descriptions of scene dynamics. The authors validate the high quality of their dataset by using it to train models that achieve state-of-the-art performance on several downstream benchmarks, such as monocular depth and pose estimation.", "questions": "See the weakness section.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "###  **Strengths**\n\n* **Addresses a Significant Problem:** The paper tackles a well-known and critical bottleneck in 4D vision and robotics: the scarcity of large-scale, diverse, and richly-annotated 4D data from real-world scenes. The creation of the DynamicVerse dataset is a valuable and high-impact contribution to the community.\n\n* **Thorough Experimental Validation:** The authors  rigorously validate the quality of their annotation pipeline. By demonstrating state-of-the-art performance on three separate downstream tasks (depth, pose, and intrinsics estimation), they provide strong, quantitative evidence that the generated data is accurate and useful.\n\n* **Novelty in System Integration and Optimization:**  While many components are off-the-shelf, the way they are integrated is novel and non-trivial.\n\n* **Well-Written and Clearly Presented:** The paper is exceptionally well-written and easy to follow. It clearly motivates the problem, logically explains the complex pipeline, and provides detailed appendices that add transparency to the work.\n\n### **Weaknesses**\n\nThe paper's primary contributions are strong, but its claims of generalizability and scalability could be better substantiated by addressing the following key points:\n\n* **Generalizability Beyond \"In-the-Wild\" Internet Videos:** The dataset is sourced from internet videos, which, while diverse, have their own characteristic styles (e.g., camera motions, subjects, video quality). It's worth questioning how well the statistics of this dataset map to other critical domains, such as egocentric robotics or professional film, and consequently, how well models trained on it will generalize.\n\n* **Insufficient Dataset Characterization:** The paper lacks a deep statistical analysis of the DynamicVerse dataset itself, such as detailed distributions of scene/motion types,  clear documentation on data formats and accessibility tools, and a discussion on ethical considerations and potential data source biases.\n\n* **Unreported Inference Speed and Computational Cost:** For a data generation pipeline whose main claim is scalability, the paper provides no analysis of its end-to-end inference time or computational requirements. Understanding the cost to process a single video is crucial for assessing its practical viability and scalability.", "was_revised": true, "final_justification": "I have read the authors' rebuttal and confirm my high rating. The response was exemplary. The authors not only provided a strong justification for the method's generalizability but also went beyond expectations by delivering a full statistical analysis of the dataset and a detailed breakdown of computational costs. These additions significantly improve the paper's transparency and utility, addressing all potential issues. The authors' professionalism and diligence give me full confidence in the final quality of this work.", "rebuttal": "We sincerely thank the reviewer for their positive and encouraging feedback. We are delighted that the reviewer found our paper to be a high-impact contribution with thorough experimental validation and a clear presentation. We address the remaining points below.\n\n------\n\n> **Q1: Questions on Generalizability Beyond Internet Videos.**\n\nWe thank the reviewer for this insightful question regarding generalizability. The robustness of our pipeline on challenging, in-the-wild data is a key prerequisite for its applicability to demanding domains like egocentric robotics and professional film. We demonstrate how our pipeline's qualitative superiority on core challenges translates to value in these areas.\n\nOur pipeline's ability to achieve more robust moving object segmentation and global refinement directly translates to higher-quality results on complex scenes. The full extent of this robustness is best appreciated through video visualizations. While `the NeurIPS policy prohibits providing images, videos, or other rich media during the rebuttal period`, we are fully committed to significantly expanding our supplementary material.\n\n- **Demo Video (suppl. - webpage/index.html):** This showcases our full 4D reconstruction. Specifically, these demos include challenging cases such as significant occlusion (e.g., the \"Public Bus\" scene) and fast motion (e.g., the \"Public Bus\" and \"Walking Woman\" scenes). Handling such dynamic events is a fundamental challenge in both robotic perception, where tracking objects in cluttered environments is key, and in analyzing cinematic action sequences in film.\n- **Figures 2-4 (suppl.):** These figures highlight our ability to produce clean static backgrounds and coherent shapes for dynamic objects, with minimal \"ghosting\" or \"streaking.\" This level of fidelity is critical for applications ranging from creating accurate environment models for robotic navigation to generating clean plates for VFX and compositing in film.\n- **Figure 5 (suppl.):** This figure demonstrates our superior dynamic object segmentation, showing sharper boundaries and greater temporal consistency. Such precise, temporally-stable segmentation is vital for reliable object tracking in robotics and for high-quality rotoscoping and character interaction in film production.\n\nIn summary, by demonstrating robustness on these core 4D challenges, we provide strong evidence that our `DynamicGen` pipeline and the resulting `DynamicVerse` dataset offer a powerful foundation for both general-purpose 4D understanding and specialized applications in robotics and film.\n\n\n\n> **Q2: Insufficient Dataset Characterization and Documentation**.\n\nWe sincerely thank the reviewer for these important suggestions on improving the dataset's characterization and documentation. We agree that providing these details is essential for community adoption and responsible research. We have prepared the following analyses and statements, which will be added to the appendix and our project website in the final version.\n\n**1. Detailed Dataset Characterization**\n\nTo provide a concrete statistical analysis as requested, we have characterized a significant subset of our dataset: the 1,268 high-quality videos sourced and filtered from the original 3,471 videos in the YouTube-VIS dataset. Our analysis reveals a rich diversity of objects and actions:\n\n- **Object Distribution:** This subset contains 2,158 unique moving object instances across 355 distinct categories. The distribution is characterized by a high frequency of common objects followed by a long tail of more specific categories, ensuring a mix of common and rare instances. The top 5 most frequent categories are: person (16.2%), panda (4.4%), rabbit (2.6%), bird (2.5%), and giraffe (2.2%). This demonstrates a rich variety of both human and animal subjects.\n- **Action & Motion Distribution:** To characterize the \"motion types,\" we analyzed the verbs from our generated captions. The distribution highlights a strong focus on dynamic interactions. The top 5 most frequent actions are: moving (276 instances), holding (178), wearing (144), walking (80), and standing (78). The full distribution covers a wide array of fine-grained actions.\n\n**2. Data Format and Accessibility Tools**: To ensure our dataset is transparent and easy to use, we will provide comprehensive documentation detailing our data structure. Each scene will be organized in a self-contained directory with annotations saved in standard, easy-to-parse formats: all camera parameters (intrinsics and poses) will be stored in a single `anno.npz` file for efficient loading; per-frame depth maps will be saved as high-precision 16-bit `.png` files; object segmentation masks will be provided as a sequence of `.png` images; and all hierarchical text captions will be organized in a `captions.json` file. To further improve accessibility, we will release a suite of Python scripts for easy data parsing and visualization. Furthermore, the complete `DynamicVerse` dataset will be publicly hosted on the Hugging Face Hub to ensure broad access for the research community.\n\n**3. Ethical Considerations and Data Bias**: We have also drafted a detailed datasheet and ethics statement that will accompany the dataset release. All videos are sourced from public platforms under permissive, non-commercial licenses (e.g., Creative Commons). We acknowledge that any dataset sourced from the internet may reflect the geographical and cultural biases of its source platforms, and we will encourage users to be aware of this. `DynamicVerse` is intended solely for academic and research purposes and not for surveillance or any application that may infringe on personal privacy. \n\nWe are confident that these additions will make `DynamicVerse` a more transparent, accessible, and responsible resource for the research community.\n\n\n\n> **Q3: Unreported Inference Speed and Computational Cost**.\n\nWe thank the reviewer for highlighting this important point. A clear analysis of the computational cost, including both processing time and memory requirements, is indeed crucial for assessing the practical scalability of our `DynamicGen` pipeline.\n\nTo provide concrete and reproducible figures, we processed all 23 videos from the Sintel training set on a single NVIDIA H20 GPU. The table below presents the average processing time and peak VRAM usage for each major module in our pipeline.\n\n| Module                                   | Hardware Used            | Avg. Time / Sintel Video (mins) | Peak VRAM (GB) | Notes                                                |\n| ---------------------------------------- | ------------------------ | ------------------------------- | -------------- | ---------------------------------------------------- |\n| 1. Motion-aware Keyframe Extraction      | 1x H20 GPU               | ~0.1                            | ~10            | Selects representative frames                        |\n| 2. VLM-Based Semantic Analysis (Qwen-VL) | 2x H20 GPU               | ~1.6                            | ~60            | Identifies dynamic elements                          |\n| 3. Moving Object Segmentation (SA2VA)    | 1x H20 GPU               | ~0.8                            | ~30            | Per-object video segmentation                        |\n| 4. Dynamic Bundle Adjustment             | 1x CPU Core + 1x H20 GPU | ~12.2                           | ~30            | Main time bottleneck                                 |\n| 5. Moving Object Captioning              | 2x H20 GPU               | ~2.0                            | ~24            | Object-level descriptions                            |\n| 6. Dynamic Scene Captioning              | 2x H20 GPU               | ~3.0                            | ~40            | Scene-level descriptions                             |\n| 7. Camera Motion Captioning              | 2x H20 GPU               | ~2.0                            | ~40            | Camera-level descriptions                            |\n| 8. Caption Rephrasing                    | 1x H20 GPU               | ~2.0                            | ~24            | LLM-based refinement for consistency and conciseness |\n| **Total (per video)**                    | H20 GPU                  | **~23.7**                       | **~60**        | Peak VRAM, not sum                                   |\n\n------\n\nWe believe this detailed, per-module cost analysis provides the necessary information for the community to assess the practical viability and scalability of our pipeline. We will add this breakdown to our supplementary material for full transparency."}, {"review_id": "DoMDWyHS9l", "reviewer": "Reviewer_1YD4", "summary": "Official Review of DynamicVerse", "questions": "Besides my concerns mentioned in the Weaknesses, I hope the author report more detailed statistics of DynamicVerse, such as the number/distribution of scenes, moving objects, and body actions.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: This paper introduces an automated data curation pipeline to generate physically-aware multi-modal 4D data at scale. Based on this pipeline, it collects a large-scale dataset named DynamicVerse that contains metric-scale point maps, camera parameters, object masks with corresponding categories, and detailed descriptive captions.\n\nWeakness 1: In this submission, the author did not provide any demo video to intuitively analyze the quality and realism of reconstructed 4D moving objects / dynamic scenes and inferred metric-scale camera trajectories. These screenshots can not comprehensively reflect the realism of the proposed dataset. \n\nWeakness 2: In Sec.3.3, the recovery of fast-moving or obscured objects is challenging. Thus, without visualization experiments, I do not think the technical framework described in Sec. 3.3 is robust and effective for these challenging object samples.\n\nWeakness 3: The weight of each loss item in Eq. 1 is not described in the main paper or supplementary material.\n\nWeakness 4: Besides the three task-specific experiments conducted, I think the author should perform an experiment on a cross-modality generation task. Specifically, since there are many large-scale datasets for the video depth estimation, camera pose estimation, and camera intrinsics estimation tasks, better performance on these tasks solely reflects that some operations used by DynamicGen are effective. However, intrinsically, these operations are developed from existing models and do not have sufficient technical contributions. I suggest the author perform a performance comparison on an under-explored cross-modality generation task with his DynamicVerse dataset. These conducted experiments should verify that the main contribution of DynamicGen is its potential contribution to under-explored cross-modality tasks, as mentioned in line 40~43.\n\n\ncan not reflect the importance and contribution of DynamicVerse.", "was_revised": true, "final_justification": "Thanks to the authors for the detailed responses to each point. These responses have eliminated my main concerns. I would like to raise my score to \"Borderline accept\"", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions.\n\n------\n\n> **Q1: Lack of Video Results / Insufficient Visualization / Robustness in Challenging Scenarios.**\n\nWe completely agree that video demonstrations are the most effective way to intuitively analyze the quality, realism, and robustness of our generated 4D data. While `the NeurIPS policy prohibits providing images, videos, or other rich media during the rebuttal period`, we are fully committed to significantly expanding our supplementary material.\n\n- **Demo Video (suppl. - webpage/index.html)** showcases the reconstructed moving objects, dynamic scenes, and metric-scale camera trajectories. Specifically, these demos include challenging cases such as significant occlusion (e.g., the \"Public Bus\" scene) and fast motion (e.g., the \"Public Bus\" and \"Walking Woman\" scenes). This video evidence directly demonstrates the robustness of our pipeline on the very types of challenging scenarios the reviewer highlighted.\n- **Figures 2-4 (suppl.)** showcase our metric-scale 4D reconstruction results on several challenging in-the-wild videos. These qualitative examples highlight our method's ability to produce cleaner static backgrounds and more coherent shapes for dynamic objects, with significantly fewer artifacts like \"ghosting\" or \"streaking\" compared to baselines. These results serve as compelling evidence that our full pipeline is robust and produces high-fidelity reconstructions even from complex, unconstrained video inputs.\n- **Figure 5 (suppl.)** provides a direct comparison of our dynamic object segmentation against other methods. We urge the reviewer to note how our approach maintains sharper boundaries and greater temporal consistency, especially for fast-moving objects (e.g., \"Sprinting\" and \"Car Racing Competition\" scenes). Crucially, it demonstrates superior robustness to partial occlusions (e.g., \"Figure skating\" scene), correctly segmenting objects even when parts are momentarily hidden. This precise segmentation is the cornerstone of robust downstream 4D reconstruction.\n\n\n\n> **Q2: Missing Loss Weights for Eq. (1).**\n\nThank you for highlighting this. Eq. (1) is a conceptual overview for our multi-stage pipeline, not a single weighted loss. We optimize different subsets of these terms sequentially in distinct stages for optimal performance. Therefore, weights are defined per-stage, as detailed below:\n\n- Stage 3 (Static BA): We set weights  $\\lambda_\\text{BA}=\\text{1.0}$ and $\\lambda_\\text{cam}=\\text{1.0}$.\n- Stage 4 (Non-rigid BA):  We set weights $\\lambda_\\text{NR}=\\text{1.0}$, $\\lambda_\\text{NR}=\\text{100}$ and $\\lambda_\\text{motion}=\\text{10}$.\n\nWe will add this detailed breakdown to the revised paper.\n\n\n\n> **Q3: More Evaluation to Prove Technical & Data Contribution.**\n\nWe thank the reviewer for this excellent suggestion. To demonstrate our work's contribution, we provide new experiments validating our dataset's utility and clarifying our pipeline's technical novelty.\n\n**1. Data Contribution:** Validating DynamicVerse Annotations\n\nTo prove the value of our dataset, we conducted new experiments validating both its **semantic** and **geometric** annotations, which are the core components of DynamicVerse.\n\n**A. Validating Semantic Annotations on a Demanding Cross-Modal Task:** To directly address the reviewer's suggestion, we validated our semantic annotations on a challenging cross-modal task that exemplifies the \"linguistic 4D Gaussian Splatting\" concept from our introduction (lines 40-41): language-grounded 4D scene understanding, using the established 4D-LangSplat [1] model.\n\nTo isolate the impact of our annotations, we replaced the sparse labels in the 4D-LangSplat with our pipeline's richer descriptions. The resulting performance boost (shown below) provides direct quantitative proof of the value of the high-quality annotations that comprise our `DynamicVerse` dataset.\n\n| Metrics                                       | Acc(%) ↑  | vIoU(%) ↑ |\n| --------------------------------------------- | :-------: | --------- |\n| w/ Original Semantic Annotations for Training |   53.84   | 27.55     |\n| w/ DynamicVerse Semantic Annotations (Ours)   | **64.42** | **51.65** |\n\n**B. Validating Geometric Annotations via Fine-tuning:** To demonstrate the quality and utility of our geometric data, we fine-tuned the prominent model MonST3R [2] on a subset (~1000 scene) of DynamicVerse. The model, now enhanced with our data, shows a performance improvement on the standard Sintel benchmark. This confirms that our generated geometry is not only accurate but also provides valuable diversity that improves the robustness and performance of existing models.\n\n| Method                            | ATE↓  | RPE trans↓ | RPE rot↓ | Abs↓  | δ1.25↑ |\n| --------------------------------- | ----- | ---------- | -------- | ----- | ------ |\n| MonST3R (Original)                | 0.108 | 0.043      | 0.729    | 0.358 | 52.1   |\n| MonST3R + sub-DynamicVerse (Ours) | 0.108 | 0.042      | 0.727    | 0.335 | 58.9   |\n\n\n\n**2.Technical Contribution**: Novelty Beyond Integration\n\nThe creation of such high-quality data was only possible due to our pipeline's significant technical contributions, which go far beyond combining existing models.\n\n**A. Geometric Novelty**\n\nOur dynamic BA pipeline introduces three key components absent in prior work: `(a) Epi-Mask Filtering`, `(b) VLM-Based Dynamics Analysis`, and `(c) Sliding Window Global Refinement`. The ablation study below quantifies the significant impact of each component.\n\n| Components Ablation on Sintel | (a)      | (b)      | (c)      | ATE↓         | RPE trans↓   | RPE rot↓     | Abs↓         | δ1.25↑       |\n| ----------------------------- | -------- | -------- | -------- | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Baseline                      |          |          |          | 0.114694     | 0.032125     | 0.347920     | 0.216433     | 0.725167     |\n| Exp-1                         | &#10004; |          |          | 0.114065     | 0.032250     | 0.335198     | 0.215058     | 0.726943     |\n| Exp-2                         |          | &#10004; |          | 0.110530      | 0.033122     | 0.334005     | 0.210339     | 0.722999     |\n| Exp-3                         |          |          | &#10004; | 0.114694     | 0.032125     | 0.347920     | 0.214282     | 0.724084     |\n| Exp-4                         | &#10004; | &#10004; |          | 0.108459     | 0.028906     | 0.281979     | 0.205892     | 0.727616     |\n| Exp-5                         | &#10004; |          | &#10004; | 0.114065     | 0.032250     | 0.335198     | 0.214143     | 0.725534     |\n| Exp-6                         |          | &#10004; | &#10004; | 0.110530     | 0.033122     | 0.334005     | 0.207329     | 0.725784     |\n| DynamicGen (Ours)             | &#10004; | &#10004; | &#10004; | **0.108459** | **0.028906** | **0.281979** | **0.204574** | **0.728961** |\n\n**B. Semantic Novelty**\n\nOur pipeline introduces a sophisticated, multi-level captioning system that moves far beyond the sparse object labels of prior work. This is achieved through several novel techniques:`(a) Semantic-Aware Keyframe Extraction (SAKFE)`, `(b) Hierarchical Prompting (HP)`, `(c) Automated Caption Rephrasing` and `(d) Chain-of-Thought (CoT) Prompting`.\n\n| Evaluation Criteria [3] | **Accuracy**↑ | Completeness↑ | Conciseness↑ | Relevance↑ | Average↑  |\n| ----------------------- | ------------- | ------------- | ------------ | ---------- | --------- |\n| Direct Output           | 79.28         | 76.65         | 73.23        | 80.33      | 77.37     |\n| + (a) SAKFE             | 80.23         | 77.46         | 74.01        | 81.45      | 78.29     |\n| + (b) HP                | 82.57         | 81.42         | 71.17        | 82.56      | 79.43     |\n| + (c) Rephrasing        | 82.48         | 80.50         | 71.86        | 83.27      | 79.53     |\n| + (d) COT               | **84.38**     | **82.09**     | **75.87**    | **85.56**  | **81.97** |\n\nIn summary, by demonstrating significant improvements on both a novel cross-modal task and an established fine-tuning task, and by detailing the novel components of our pipeline, we have provided concrete new evidence for both the data contribution of DynamicVerse and the technical contribution of DynamicGen. We thank the reviewer again for prompting this clarification.\n\n\n\n> **Q4: More Dataset Statistics.**\n\nWe have performed a detailed characterization of the dynamic content within a significant subset of 1,268 videos (filtered from the original 3,471 in YouTube-VIS). We will include the full statistics in our appendix, and a summary of our findings on the distribution of moving objects and actions is provided below.\n\n(1) Moving Object Distribution: This subset contains 2,158 unique moving object instances across 355 distinct categories. The distribution features a high frequency of common objects followed by a long tail of more specific categories, ensuring a mix of both common and rare examples. The top 5 most frequent categories are: person (16.2%), panda (4.4%), rabbit (2.6%), bird (2.5%), and giraffe (2.2%), showcasing a rich variety of both human and animal subjects.\n\n(2) Action and Motion Distribution: To characterize the \"body actions,\" we analyzed the verbs from our generated captions. The distribution highlights a strong focus on dynamic interactions and object manipulation. The top 5 most frequent actions are: moving (276 instances), holding (178), wearing (144), walking (80), and standing (78). The full distribution covers a wide array of fine-grained actions.\n\n------\nReference:\n\n[1] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.\n\n[2] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.\n\n[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025."}, {"review_id": "sjUjUp55XV", "reviewer": "Reviewer_uDiP", "summary": "The paper presents DynamicVerse, a large scale multimodal dataset and pipeline leveraging foundation models for dynamic 4D(3D + time) scene understanding from monocular videos. The proposed system DynamicGen consists of: (1) Data curation, (2) Data filtering with assistant from VLM, motion features, MLLM features, (2) Identify Moving Objects using QwenVL to identify object, SA2VA to segment, and a Physically-aware size extraction using estimated depth and object category,  (3) Dynamic bundle adjustment, (4) Caption Generation. \n\nThe resulting DynamicVerse dataset includes over 100,000 scenes, 800,000 object masks, and 10 million frames. \n\nThey then evaluate their data collection pipeline on existing baselines on video depth estimation, camera pose estimation, and camera intrinsics estimation benchmarks like Sintel and KITTI", "questions": "1. Why use SA2VA for masks but DAM for captions, even though SA2VA can generate captions too? Can you elaborate on this design choice?\n2. Do all camera motion captions come from a VLM? Can you elaborate on which VLM in section Camera motion captioning at line 240? Also, can you clarify how you avoid hallucinations comes from the VLM or is there any analysis on the camera motion caption quality comes from VLM?\n3. The author provide a nice Canonicalization algo, which is very helpful for QA.  The author mentioned in Appendix A.4:\n“When not enough points corresponding to horizontal surfaces are detected, we flag canonicalization as failed and avoid synthesizing questions that depend on canonicalization, such as questions about elevation.”\nCould the authors provide statistics on how often the canonicalization process fails in practice?\n4. Can the authors clarify how the human-in-the-loop review was implemented? Specifically:\n- How many examples were reviewed?\n- What fraction of the dataset was filtered or corrected?\n- Were the annotators domain experts or crowdworkers?\n- Were any inter-annotator agreement metrics used?", "limitations": "The authors mention a few limitations in passing (e.g., canonicalization failure cases, reliance on frozen foundation models), but overall the discussion of limitations is minimal and could be significantly improved.\nA few critical limitations worth addressing more explicitly:\n- Failure modes of the pipeline (e.g., depth estimation under low-light, incorrect motion segmentation) \n- Reliance on proprietary and large-scale foundation models", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:** The paper provides a solid pipeline that integrates models like UniDepthV2 and Qwen2.5-VL to produce precise 4D models from monocular videos. Its experiments on benchmarks like Sintel and KITTI show top results, with clear metrics proving it outperforms methods like MonST3R. The well designed component in the pipeline, like physically-aware size extraction and canonicalization steps, along with LLM-based caption rephrasing, showcase thoughtful engineering. The DynamicVerse dataset offering a massive collection of over 100,000 scenes and 10 million frames. This makes it a valuable resource for fields like robotics and virtual reality. The writing is clear, with detailed figures and equations that explain the approach well.\n\n**Weaknesses:** However, the paper leans heavily on specific foundation models, which might not work as well with different data. \n\nThe paper lacks quantitative and procedural details about the human review phase. It’s unclear how many examples were reviewed, which components were corrected, or whether consistent QA standards were applied.\n\nWhile the authors run strong experiments on existing datasets, they do not show how DynamicVerse might support training or benchmarking new models, nor do they evaluate other methods on DynamicVerse.\n\nThe pipeline depends on closed or heavyweight models like Qwen2.5-VL and UniDepthV2, which raises concerns about accessibility, replicability, and long-term maintenance.\n\nKey operations such as canonicalization, motion segmentation, and VLM-based captioning are susceptible to edge cases (e.g., occlusion, poor lighting), but the paper does not quantify how often such failures occur or how they affect downstream utility.", "was_revised": true, "final_justification": "After reviewing the authors' comprehensive rebuttal, including new experiments (e.g., MonST3R fine-tuning and semantic evaluations), detailed statistics (e.g., failure rates, human review specifics), and qualitative evidence (e.g., demo videos addressing robustness in occlusion/fast motion), all my concerns regarding dataset validation, failure analysis, model reliance, and human-in-the-loop transparency have been resolved. This solidifies my recommendation for acceptance (score 5), as the work now clearly demonstrates value for 4D multimodal research.", "rebuttal": "Thank you for your valuable comments and kind words to our work. Below we address specific questions.\n\n------\n\n\n\n> **Q1: Heavy Reliance on Specific Foundation Models & Reproducibility and Accessibility Concerns**\n\nWe thank the reviewer for these critical points. Our pipeline's core design philosophy of modularity directly addresses these concerns. Instead of a monolithic system, each key component (depth, segmentation, VLM) is a \"plug-and-play\" module. This architecture makes our framework adaptable across domains (e.g., swapping in models for robotics or autonomous driving), future-proof (easily integrating new SOTA models), and accessible (allowing users to substitute heavyweight components with lighter, open-source alternatives). Therefore, our primary technical contribution is the versatile and durable framework itself.\n\n\n\n> **Q2:  Specifics of Human-in-the-Loop Review.**\n\nThank you for these important questions. We provide the details of our review process below.\n\n1.Scale and Purpose: Our human review was a quality audit on a ~10K scene subset, focusing on semantic annotations (object, scene, camera captions)\n\n2.Correction Rate & Impact: Within this subset, ~19% of captions required minor corrections (e.g., for specificity). This feedback was used both to fix samples and to iteratively improve our pipeline's prompting strategies.\n\n3.Annotator Team: The review was conducted in-house by authors and graduate students deeply familiar with the project's quality standards.\n\n4.Consistency Assurance: We ensured consistency not via formal IAA metrics, but through a collaborative review process, where discrepancies in overlapping batches were resolved through group discussion with a senior author.\n\nWe will add a new section to our appendix detailing this entire human-in-the-loop procedure  provide a comprehensive account of our quality assurance procedure.\n\n\n\n> **Q3: DynamicVerse Dataset Validation.**\n\nWe thank the reviewer for this critical feedback. To directly demonstrate our dataset's utility, we conducted new experiments which provide strong evidence for the quality and effectiveness of the geometric and semantic annotations in DynamicVerse.\n\n**1. Geometric Validation: Fine-tuning MonST3R**：\n\nTo validate the quality and utility of our geometric annotations (i.e., metric-scale point maps and camera parameters), we used a subset of 1000 scenes from DynamicVerse to fine-tune the prominent model, MonST3R [1]. As shown in the table below, fine-tuning on our data leads to a performance improvement on the Sintel benchmark. This directly confirms that the geometry in DynamicVerse is accurate and beneficial for improving existing models.\n\n| Method                            | ATE↓  | RPE trans↓ | RPE rot↓ | Abs↓  | δ1.25↑ |\n| --------------------------------- | ----- | ---------- | -------- | ----- | ------ |\n| MonST3R (Original)                | 0.108 | 0.043      | 0.729    | 0.358 | 52.1   |\n| MonST3R + sub-DynamicVerse (Ours) | 0.108 | 0.042      | 0.727    | 0.335 | 58.9   |\n\n**2. Semantic Validation: Downstream Tasks and Direct Evaluation**：\n\nWe performed three distinct experiments to validate the high quality of our hierarchical semantic annotations:\n\n**(a) Object-Level Semantics via 4D-LangSplat [2]:** To demonstrate the effectiveness of our dynamic object annotations, we used them to train a 4D-LangSplat model. Our precise object masks and labels enabled the model to achieve superior results in language-grounded 4D scene understanding. This validates that our object-level annotations are highly effective for complex, multi-modal downstream tasks.\n\n| Metrics                                       | Acc(%) ↑ | vIoU(%) ↑ |\n| --------------------------------------------- | :------: | --------- |\n| w/ Original Semantic Annotations for Training |  53.84   | 27.55     |\n| w/ DynamicVerse Semantic Annotations (Ours)   |  64.42   | 51.65     |\n\n**(b) Scene-Level Semantics via G-VEval [3]:** We used the established G-VEval benchmark [3] to quantitatively assess our generated scene-level captions. On a random sample of 100 videos from filtered SA-V data [4], our captions achieved a high score, confirming their coherence, accuracy, and richness.\n\n| Evaluation Criteria   | **Accuracy**↑ | Completeness↑ | Conciseness↑ | Relevance↑ | Average↑ |\n| --------------------- | ------------- | ------------- | ------------ | ---------- | -------- |\n| DynamicVerse Captions | 84.38         | 82.09         | 75.87        | 85.56      | 81.97    |\n\n\nThe strong performance across these metrics confirms that our captions are not only factually accurate and relevant to the video content, but also complete in their coverage of events and efficiently concise. This robust, multi-faceted quality makes them highly suitable and reliable for demanding downstream applications.\n\n**(c) Camera-Level Semantics via Human Study:** We conducted a formal human study on 88 videos from our DAVIS subset to quantitatively analyze our camera motion captions. Following the criteria of Clearness, Conciseness, and Grammar & Fluency from prior work [5], the results were excellent: over 60.22% of captions were rated as both clear and fluent while also receiving high scores for conciseness, confirming the effectiveness of our generation and quality control process. \n\n| Human Evaluation | Rated as Clear | Rated as Fluent | Rated as Concise |\n| ---------------- | :------------: | :-------------: | :--------------: |\n| Camera Captions  |     85.22%     |     89.77%      |      67.04%      |\n\n\n\n> **Q4: Lack of Failure Case Analysis.**\n\nWe thank the reviewer for this excellent suggestion. We agree that quantifying the frequency and impact of failure cases is crucial for understanding our pipeline's robustness. We have conducted a new analysis to provide these details:\n\n(1) Failure Frequency Analysis\n\nWe analyzed the failure rates of our key components on challenging data to quantify their robustness.\n\n**A. Moving Object Segmentation:** We manually identified challenging subsets within the Sintel training set to analyze our segmentation performance in edge cases. We found failure rates of 28% (3 of 14 scenes) for severe occlusion and 11% (1 of 9 scenes) for low-light conditions.\n\n**B. Canonicalization:** The primary failure mode is the lack of a detectable horizontal surface (e.g., mid-air scenes). On the DAVIS subset, this process fails in approximately 15.9% of the 88 filtered scenes.\n\n(2) Impact on Downstream Utility of Segmentation\n\nTo quantify how segmentation errors affect the final geometric annotations, we performed a controlled experiment on a challenging scene. We compared the final depth and pose metrics when using (1) our pipeline's automated masks vs. (2) manually assited, pseudo ground-truth-level masks. Specifically, we manually identified dynamic objects by placing point prompts on them.\n\n| Mask Type on a Challenging Scene | Low-light: scene='sleeping2' |          | Occlusion: scene='ambush_7' |          |\n| -------------------------------- | ---------------------------- | -------- | --------------------------- | -------- |\n| Metrics                          | Abs Rel↓                     | δ1.25↑   | Abs Rel↓                    | δ1.25↑   |\n| Our Automated Masks              | 0.053944                     | 0.986288 | 0.066431                    | 0.994412 |\n| Manually assited Masks           | 0.053828                     | 0.987431 | 0.066028                    | 0.994031 |\n\nAs shown, while segmentation errors do degrade the final geometric accuracy, the results from our automated pipeline remain reasonable. This indicates a degree of resilience in our downstream bundle adjustment process to minor imperfections in segmentation.\n\n\n\n> **Q5: Design Choice on Model Selection**.\n\nWhile the powerful SA2VA model can perform both segmentation and captioning, we use the specialist Describe Anything Model (DAM) for captioning to achieve superior quality and flexibility. We chose DAM for two advantages over SA2VA: (1) Superior Detail: DAM is optimized for generating richer, more localized descriptions. (2) Mask-Conditioned Input: Crucially, DAM can take GT masks (e.g., from 2D segmentation datasets) as input for precise, targeted captioning, a feature SA2VA lacks but was a requirement for our pipeline.\n\n\n\n> **Q6: Details of Camera Motion Captioning**.\n\nThank you for these detailed questions. We generate camera motion captions using a specialist VLM from CameraBench [5], which was fine-tuned on ~3K expert-annotated videos. To ensure high fidelity and mitigate hallucinations, we employ a two-stage quality control process: first, our upstream data filtering removes videos with erratic motion, and second, a final human review corrects any remaining inaccuracies. The effectiveness of this entire process was confirmed through a human study on 88 DAVIS videos, where our captions demonstrated high quality across the criteria of Clearness, Conciseness, and Fluency [5] (detailed results are in the table for Q3).\n\n\n\n\n> **Q7: Limitation & Typo**.\n\nWe are grateful for the reviewer's guidance and careful reading. We agree that the limitations section can be significantly improved. In the final version, we will expand it to explicitly discuss the pipeline's failure modes and our reliance on large-scale foundation models, as you helpfully pointed out.\n\n\n\nWe would like to sincerely thank you once again for your constructive comments throughout the review process.\n\n------\n\nReferenecs:\n\n[1] MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion. ICLR 2025.\n\n[2] 4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models. CVPR 2025.\n\n[3] G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4o. AAAI 2025.\n\n[4] SAM 2: Segment Anything in Images and Videos. ICLR 2025.\n\n[5] CameraBench: Towards Understanding Camera Motions in Any Video.  Arxiv 2025."}]

Author rebuttal:

Meta-review: The paper addresses the critical issue of the lack of large-scale, in-the-wild 4D datasets and proposes a pipeline which runs foundation models as well as windowed bundle adjustment on in-the-wild videos where dynamic objects are determined through a series of VLM questions. This then produces a large-scale dataset (100k videos) with high quality pseudolabels. 

The paper received positive reviews (3 borderline accept, 1 accept) with reviewers commending the paper's task and approach. The main concerns expressed in the review process was with respect to the verification of the quality of the labels. In particular, that there was no experiment which showed that training with this data could improve prior models. The authors answered this in the rebuttal with 4 new experiments. While the results could be stronger, this does address this issue.

I advocate for acceptance, based on the added verification in rebuttal. I would strongly encourage the authors to rewrite the paper to include these because they substantially improve the paper.
Decision: Accept (poster)