Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Authors: Haoran Zhou
Gim Hee Lee
Venue: NeurIPS 2025 poster ()
Keywords: Video object segmentation
3D point tracking
Monocular dynamic novel view synthesis
4D scene reconstruction
Primary area: applications
TL;DR: Motion4D estimates consistent motion and semantics by integrating 2D models with explicit 4D reconstruction, outperforming SAM2 and state-of-the-art 2D tracking networks.

Abstract: Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.

Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Haoran Zhou
Gim Hee Lee
Department of Computer Science National University Singapore haoran.. zhou@u. nus edu
Department of Computer Science National University of Singapore gimhee lee@nus edu sg
DEVA 5
SAM2 34
Motion4D
GT
Boots 9
CoTracker3
15
Motion4D
Query Points
Figure 1: Visualization of segmentation and tracking results. We compare Motion4D with state- of-the-art 2D foundation models, highlighting their lack of 3D consistency. As shown, existing 2D approaches often suffer from temporal flickering (green box) or spatial misalignment (red box).
Abstract
Recent advancements in foundation models for 2D vision have substantially im- proved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D envi- ronments. In this paper, present Motion4D, novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To en- hance motion accuracy, we introduce 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based per-pixel RGB and semantic errors. Fur- thermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts SAM2. Extensive evaluations demonstrate that our Mo- tion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based track- ing, video object segmentation, and novel view synthesis. Our code is available httts:///rrzonn github. wo/motionoddweett
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
The joint modeling of geometry, semantics and temporal dynamics of 4D scenes is fundamental task in computer vision with broad applications in robotics, autonomous driving, augmented reality, elc. A key aspect of this problem is the estimation of motion-related priors, such as semantic masks, 2D point tracks, and depth maps, which serve as essential cues for reconstructing and interpreting complex scenes. However, despite significant progress, ensuring consistency and accuracy in the predictions remains challenging due to occlusions, view variations, and motion ambiguity.
Recently, the emergence of the Segment Anything Model (SAM) [8]] has revolutionized 2D visual understanding and established itself as a foundational model for image segmentation. Inspired by its success, many related vision models have been developed. For example, Track Any Point [ g for point-based object tracking and Depth Anything 47 48) for monocular depth estimation. Although these models have achieved impressive performance by using large-scale datasets and extensive pre-training, they remain inherently limited in maintaining 3D coherence. In practice, even the state- of-the-art SAM2 34 model still suffers from significant inconsistencies across frames, including spatial misalignment, temporal flickering, and boundary artifacts (cf. Figure T). Such limitations arise from their design, which relies on frame-wise processing and lacks explicit 3D reasoning.
One promising solution is to lift 2D vision models for 3D understanding by leveraging explicit 3D representations such Neural Radiance Fields (NeRFs) 26 or 3D Gaussian Splatting (3DGS) [17. Unfortunately, most existing methods [52 1 37 51 are specifically designed for static scenes, where they improve consistency across the scene by incorporating multi-view segmentation results into an optimized 3D representation. Applying 2D models to dynamic environments introduces new challenges, including motion complexity, occlusions, and spatio-temporal alignment. These challenges, which are already problematic in 2D), remain difficult to resolve through direct integration with 3D representations. Another limitation of recent works 40 14 21
is that they treat 2D priors separately from dynamic 3D representations: they either learn the feature fields independently of the 3D model or decouple semantic understanding from motion estimation. As a result, the limited modeling of appearance and motion ultimately prevents the estimation of coherent predictions.
In this paper, we address the challenges of inconsistent 2D predictions and weak 3D integration by developing a unified dynamic representation that models motion and semantics from casual monocular videos. We introduce Motion4D, a method that refines multiple scene characteristics, including semantic masks, 2D point tracks, and depth estimation, while jointly optimizing a 3D) Gaussian Splatting model augmented with semantic and motion fields. Motion4D consists of two-part iterative optimization framework: 1) Sequential optimization that updates the motion and semantic fields in two consecutive stages within short temporal windows to maintain local consistency, 2) Global optimization that performs a joint optimization of all attributes throughout the sequence to ensure coherence. In the first stage of sequential optimization, we propose iterative motion refinement to enforce 3D motion consistency by correcting accumulated errors across sequences using a tracking loss with learned 3D confidence maps. We further introduce an adaptive resampling module to improve motion consistency by inserting new Gaussians into underrepresented regions identified through per-pixel RGB and semantic errors. In the second stage, we propose an iterative semantic refinement process to iteratively update 2D semantic priors by aligning rendered 3D masks with 2D predictions through bounding boxes and prompt points.
We introduce DyCheck- VOS, a new benchmark for evaluating video object segmentation in real- istic and dynamic scenes with camera and object motion. Extensive evaluations on benchmark datasets show that our Motion4D demonstrates superior performance across various dynamic scene understanding tasks. Our method significantly outperforms both 2D foundation models and 3D representation-basee methods in video object segmentation, point-based tracking, and novel view synthesis. Figure shows an example of our qualitative result compared existing approaches. Our contributions are summarized as:
We propose Motion4D, a model that integrates 2D priors from foundation models into a dynamic 3D Gaussian Splatting representation achieve consistent motion and semantic modeling from monocular videos.
We design a two-part iterative optimization framework comprising sequential optimization, which updates motion and semantic fields consecutive stages to maintain local consistency, and global optimization, which jointly refines all attributes to ensure long-term coherence.

---

We introduce iterative motion refinement using 3D confidence maps and adaptive resampling to enhance dynamic scene reconstruction, and semantic refinement correct 2D semantic inconsistencies through iterative updates with SAM2.
Our Motion4D significantly outperforms both 2D foundation models and existing 3D methods in tasks, including video object segmentation, point-basee tracking, and novel view synthesis.
2 Related Work
2D vision models for scene understanding. As a fundamental aspect of scene understanding, semantic segmentation remains an essential yet challenging task in computer vision. Recently, the Segment Anything Model (SAM) [18] was introduced as a promptable segmentation network trained on billions of masks, enabling strong zero-shot segmentation performance new images. Its successor, SAM2 [34], extends this paradigm to video by using a streaming memory mechanism, and builds the largest video segmentation dataset (SA-V) to date. SAM2 serves foundation model for segmentation across images and videos, achieving state-of-the-art results in video object segmentation [5 30 (88 and offering broad applications across various domains.
Inspired by its success, many related vision models have been developed, further exploring foundation models for vision understanding, Among them, Track Any Point (TAP) 77] introduces the task of point-level tracking for dynamic scenarios, standing out as a key task in scene understanding. Building prior work in 2D optical flow [39], it further extends the definition to capture dense and long-term relationships for tracking arbitrary points in videos. Recently, there has been a rising interest in this problem, with several works demonstrating impressive long-term 2D tracking 20 Another important topic is the Monocular This line of work aims to prediet depth information for any images under any circumstances, demonstrating strong generalization abilities suitable for various downstream scenarios. Recently, Depth Anything [47] has achieved significant progress by greatly scaling up its dataset with large-scale unlabeled data, leading to significant improvements in depth estimation. Building on this progress, several works have been proposed 48 47), with further improvement in video depth estimation and data acquisition. While these 2D foundation models demonstrate superior generalization abilities, they are still intrinsically limited in producing 3D-com 3D-consistent estimations essential for scene understanding. Our work, therefore, builds on the predictions of 2D vision models and seeks to construct a joint and spatio-temporally coherent representation of multiple motion priors.
results on challenging, in-the-wild videos [8] 19
Depth Estimation (MDE) from images and videos 33
3D representation models. In the field of 3D vision, recent advances in 3D representation models, such as Neural Radiance Fields (Nerf) (26] and 3D Gaussian Splatting (3DGS) 17, have achieved impressive results in novel view synthesis. Leveraging the expressive representation of 3DGS, research in scene reconstruction and view synthesis has expanded to dynamic scenes 43 24 49 31
10 27 [1]. allowing for the modeling of complex object geometry and motion. On the other hand, the use of 3D models has extended beyond view synthesis to broader tasks, such generation [ 23) scene understanding [46]]]]]]] 25), and language- related applications 32.. Integrating 2D vision models with 3D representations for scene understanding is also common strategy. typically for segmentation tasks with multi-view inputs in static scenes [5]] 2 52 41
37 D. However, when dealing with dynamic scenes involving complex motion, fewer relevant studies have been proposed. These methods either leverage an existing dynamic 3D representation with an additional semantic field [14] 21], or decouple the modeling of semantics and motion (40) This makes them ineffective in handling complex motion in casual videos, limiting their ability to maintain 3D consistency. Compared to previous works, our method employs an effective iterative optimization strategy jointly model semantics and motion, ensuring 3D consistency in dynamic scenes.
3 Our Method: Motion4D
Problem Definition. The input to our Motion4D is a video sequence of T posed RGB images {Ie), and a set of priors generated by 2D pre-trate preitrained models: 1) The object masks M,; 2) 2D point tracks U from to target frame at t'; 3) Monocular depth D., where t denotes the timestep. Our goal is to estimate spatio-temporally consistent predictions of semantics M, and motion
3

---

Sequential Optmization
Track Any Point & Depth Anything
SAM2 Segmentation Model
Stipe
Lest Borot
Updale Confdonce
Gonge 2
Seraec Selamnet
Securic Refrener
tracka
Masiss
Optimization
Cartanonn
Fancared H6ldd Traces & Degti
New
Ocah
toution
Treske
Motion Adeptive Sampling
Optimization Semant
(8) lterative Motion Refinement
(b) Iterative Semantic Refinement
(c) Optimization Pipeline
Figure 2: Overview of Motion4D. Our Motion4D introduces iterative refinement framework, consisting of (c) sequential optimization and global optimization stages. We develop (a) an iterative motion refinement module that uses 3D confidence maps and adaptive resampling to improve motion accuracy, and (b) an iterative semantic refinement module to refine the semantic field.
components (Ü.-ve, D.} while simultaneously reconstructing a coherent scene representation that models appearance and dynamics of the scene.
Overview. Figure 2 shows overview of our Motion4D pipeline. We first define the 3DGS with motion and semantic fields as our 4D scene understanding representation in Section [.1] followed by the introduction of Motion4D. Conceptually, our method aims to iteratively refine both the 2D priors and 3D) scene representation throughout the optimization process of dynamic scene reconstruction. For the motion field, update the 2D tracking priors by learning confidence weight to control the supervision and introduce an adaptive resampling strategy to further enhance the modeling of motion (ef. Section 3.2). For the semantic field, we leverage the promptable SAM2 model [34] as a component in our training pipeline. After each iteration, we render semantic views from the GS model and feed them as additional prompts the SAM2 model, which directly refines the 2D semantic priors (cf. Section 3.3). Furthermore, we adopt a sequential optimization strategy for motion and semantic fields effectively mitigate error accumulations over time (cf. Section 3.4). Finally, we further improve reconstruction accuracy via global optimization stage to ensure consistency and coherence across all fields (cf. Section 3.4). By combining the strong temporal consistency of explicit 3D representations and the rich semantic priors provided by 2D networks, our Motion4D achieves improved motion and semantic coherence across space and time.
3.1 4D Scene Understanding Representation
represent the scene using 3DGS [17] and further extend it with motion and semantic fields to model dynamic 3D environments. Traditionally, 3DGS defines set of N static Gaussians in the canonical frame as gf = {4p RQ 8,, os, J. where = 1,.., .N. de E R3 and R E SO(3) denote the 3D position and orientation in the canonical frame, and E R is the scale, o; R is the opacity, and o E 3 the color. During rendering, given pixel p in view with extrinsic matrix and intrinsic matrix K, its color I(p) be computed by blending intersected 3D Gaussians:
I(p) Σ Ga (1-aa))
(1)
(EH(p)
where (p) is the set of Gaussians intersecting at pixel and a; exp(- - (p 2) nd and 222 are the projected 2D Gaussian mean and covariance, respectively.
To model the motion of dynamic objects, we define a deformation field (motion field) which adjusts positions and orientations each frame via rigid transformations [42). Specifically, the pose parameters (u², R.) are rigidly transformed from the canonical frame to via T [AR oo 0-- e SEE(3 for dynamic 3D Gaussian time t, i.e.:
/ 1 #1 t th
= R²
Instead of defining per-frame motion parameters for all Gaussians, we use set of global motion bases B Tb=1 and assign coefficients wi to each Gaussian. As a result, the per-frame transformation is obtained by weighted combination T =E w. wf.g-+

---

Due to the explicit representation of 3DGS, we follow 51 2 21 to directly embed a semantic field onto each Gaussian for scene semantics. This is analogous to the role of the color field during rendering. As result, we can render the semantic view in a similar way Equation | by:
(pm(p))
((--a))
(EH(P)
j=1
This gives the per-pixel semantic features, which can be converted to object masks M, at time t.
3.2 Iterative Motion Refinement
Figure aa) illustrates the motion field refinement process, where we focus on improving the motion (deformation field) of dynamic Gaussian Splatting that directly impacts the accuracy of 3D point tracking estimation. The initial 3D point tracks are formulated by the 2D point tracks U,-r' and monocular depth D,. Following [42), we then compute the pixel-wise 3D motion trajectory of the query frame by rendering the 3D positions of Gaussian points at the target time L':
(-)
E 1 oi II 11-ay ag),
(EH(p)
where ut is position of the Gaussian at the target frame. X4-s1- therefore estimates the 3D position of the pixel at time f', which can be projected into 2D tracks U (p) and depth De-re (p). We now consider supervising these estimates using the corresponding 2D priors.
Unlike the refinement of semantics (cf. Section B.3). 2D tracking networks do not support prompt- based interactions such those in SAM2, iee. we cannot directly update 2D track and depth priors to ensure consistency. Instead, we seek to control the supervision loss by assigning a pixel-wise confidence weight w(p) reduce the influence of erroneous estimations in the input priors:
Tarack =
Σ pEt.
w(p) (p)
(p)Il, Ledepth =
Σ PEL.
w(p) DD - Dz(p')l (5)
I
[
where p/ X (p) is the position at L'. The confidence weight w(p) estimates the probability that the input priors inconsistent with the ground truth. We then add a new uncertainty field u, E IR to each Gaussian and obtain the weight by rendering the 3D uncertainty logits into 2D:
w(p) = Σ aaa II (1-a)) j=1
(6)
(EH(P)
We define the weight by evaluating the self-consistency of the pixel across time in terms of the color and semantic estimations:
Th = (C)(()))), w())), where (p)) = {
Ix), < and [[M.()), <8,
otherwise.
(7)
where BCB(.) is the binary cross entropy, and ö and 8' are the distance thresholds.
Adaptive Resampling, We introduce
3DGS densification strategy further enhance motion
modeling through an adaptive resampling process. As discussed in prior works 35] 38) the Adaptive Density Control of 3DGS [17] relies heavily on gradient magnitude. This makes it sensitive the choice of loss functions and often fails to identify the underfitting regions. Consequently, we extend the idea of error-based densification from static scenes (35) dynamic environments and develop sampling strategy to insert new Gaussians into the under-represented regions of the moving objects. In each iteration, we first compute per-pixel errors (p) and evem for the RGB and semantic views, respectively. We then select regions where Eigh (p) > Oigb or Csem (p) sample 2D points from these error-prone regions, project them into 3D using the rendered depth, and initialize new Gaussians based on their nearest dynamic Gaussians. By progressively refining and filling sparsely reconstructed regions, our Motion4D effectively recovers blurry or missing parts of foreground targets that are typically caused by inaccurate initial motion estimates tracking failures.
6xem- Finally, we
5

---

(1) loper 20 Modis
(3 Rendered Madic
Time
TAPIR
Refined
Frome00
Frame1000
Frame30
Frame100
Framcll Input
Framed Inpur
Metionll
MetionD
(a) Itcrative Refinement
Sequential Optimization (Semantic)
(c) Sequential Optimization (Motion)
Figure 3: Illustration of (a) iterative semantic refinement and sequential optimization processes for (b) the semantic field and (c) the motion field. The proposed strategies effectively update 2D input priors to achieve consistent results across space and time.
3.3 Iterative Semantic Refinement
Our Motion4D iteratively refines scene semantics by alternating between updating the 3D rep- resentation and semantic field, and refining the 2D semantic priors using the reconstructed 3D scene representation. Specifically, at the s-th iteration step, we first obtain 2D segmentation masks (M generated by SAM2 in the previous iteration to supervise the semantic field during its optimization. We then render 3D masks MC and compare each rendered mask with the corresponding 2D mask M to identify mismatched regions that require additional prompts. For each object, m prompts generated align the 2D predictions with the rendered 3D masks as follows: 1) exact bounding box the 3D mask is provided, and 2) we place positive or negative prompt points at the center of the most prominent regions, determined by the maximum value of the distance transform. Note that, we avoid using the exact 3D mask as a prompt since that SAM2 tends to strictly follow the mask input, limiting its flexibility in refining 2D semantic priors. Figure 3aa provides visual example of the iterative refinement on semantics. We show that rendered 3D masks offer greater consistency inherent to the 3D scene representation, while SAM2 excels at preserving fine-grained high-resolution details. To this end, our method effectively updates inconsistent 2D masks through additional prompts that resolve semantic ambiguity.
3.4 Optimization Pipeline
Figure illustrates the overall optimization pipeline, which consists of the sequential and global optimization in three stages.
The sequential optimization consists of two stages. In Stage 1, our Motion4D optimizes the motion field across video sequences, where each sequence is defined as chunk of consecutive frames S, I, t [iL, (i + 1). To enhance short-term tracking and dynamics, we apply the iterative motion refinement module within each sequence. We proceed to Stage upon completion of the motion field optimization, which focuses on optimizing the semantic field. During this stage, we ensure more stable semantic refinement by keeping the motion field fixed to prevent inaccurate semantic priors from affecting the Gaussian parameters.
Sequential Optimization.
Sequential optimization is crucial for maintaining long-term consistency in both motion and semantic estimations. As shown in Figure 3)), 2D priors are prone to eπror accumulation over time since 2D networks typically rely on the short-term memory of recent predictions. We observe that the SAM2 result aligns well with objects in the initial frames but gradually loses track as the video progresses. The accumulation of these inconsistencies corrupt the semantic representation and dominate the supervision when optimizing over the entire sequence space. To address this issue, our Motion4D uses sequential optimization steps to produce consistent results over short temporal windows and progressively extending them across the entire video. To ensure consistency between sequences, the semantic refinement step explicitly updates SAM2 semantic priors to enhance temporal consistency

---

Table Comparisons of segmentation performance on DyCheck- VOS and DAVIS. We compare the results of 2D networks (2D) and 3D methods (3D + SAM2 masks).
DyCheck-Ve VOS
DAVIS 2017
Method
Representation
J&FF
J
F
J&F
J
F
XMem
2D
83.5
81.0
86.0
86.2
82.9
89.5
DEVA 5
2D
84.5
81.7
87.4
87.0
83.6
90.4
SAM2 1
2D)
89.4
88.3
90.5
90.7
89.4
92.0
Semantic Flow
40
3D SAM2 masks
76.9
74.4
79.3
72.2
69.3
75.2
SADG
21
+ SAM2 masks
81.8
79.2
84.3
75.0
71.9
78.1
Motion4D
+ SAM2 masks
91.0
89.6
92.4
89.7
86.1
90.3
Motion4D SAM2
3D + SAM2 masks
91.7
90.4
93.0
90.8
89.6
92.0
(a) SADG 21
(b) DEVA
(c) SAM2 RY
(d) Motion4D
(c) MotiondD*
(f) GT
Figure 4: Visualization of segmentation results DyCheck- VOS (our proposed VOS benchmark). We provide our results both rendered masks (Motion4D) and refined SAM2 masks (Motion4D*).. As shown, the 2D predictions lack 3D consistency, which leads to misaligned spatial structures.
and correct accumulated errors. The motion refinement step also corrects accumulated errors by implicitly aligning the motion trajectories through the tracking loss (Equation 5) across sequences. Concurrently, these complementary refinements ensure robust scene understanding over time.
Global Optimization. Finally, we perform a global optimization in Stage 3 to jointly train all fields the full sequences of video frames. The global stage integrates information across all fields to achieve a coherent 4D scene representation. We show in Section that it is beneficial to learn motion and semantic representations together via scene reconstruction. The overall loss defined as L = Aigh Asem Lsem + Ausek Lrrck + Adepth + Ln where each is hyperparameter to balance the loss terms.
4 Experiments
We evaluate Motion4D across diverse tasks, including video object segmentation, point-based tracking. and novel view synthesis, demonstrate its ability model motion and semantics in dynamic scenes.
4.1 Segmentation Results
Introducing DyCheck- VOS for Video Object Segmentation. We create a new VOS benchmark to evaluate segmentation performance in realistic and dynamic scenes by manually annotating the DyCheck dataset [12] with high-quality per-frame object masks. The DyCheck dataset was originally designed for scene reconstruction and novel view synthesis. It contains 14 sequences, each with
7

---

(a) Query Points
(b) BootsTAPIR 9
(c) CoTracker3
(d) Motion4D
Figure 5: Visualization of 2D point tracking results. Motion4D maintains stable and accurate point trajectories even under severe occlusions and drastic object or camera motion.
200-500 frames, featuring diverse types of real-world motions. We annotate selected foreground objects of interest, with focus on partial regions instead of full-object masks, which involve more challenging motion patterns and frequent occlusions. As shown in Figure 4 2D networks often produce inconsistent predictions on object subparts, making DyCheck- VOS a strong benchmark for
assessing semantic consistency. We follow the standard VOS settings 30 45 and report the J&F
which is the average of Jaccard index (J), and boundary Fi-score (JF).
We also evaluate method the DAVIS dataset [30]. We use the DAVIS 2017 validation set and follow the standard setting of semi-supervised video object segmentation by providing the ground-truth object masks the first frame input.
Results. Table I summarizes the results. We report results for both the rendered masks (Motion4D)
and the refined SAM2 outputs by our method (Motion4D SAM2) in the comparisons. For DyCheck- VOS, we show that Motion4D significantly outperforms all compared methods, including both segmentation models and approaches based on 3D representations. For the compared 2D models 4 [5] we follow standard VOS settings by using the ground-truth masks of the first frame as input. For the 3D-based approaches [4 (1), we provide masks generated by SAM2 to match the input setting used our method. Unfortunately, the full training code for SADG 21 is not publicly available, and we implement the training pipeline based their released model code. Figure 4 shows 4sours visual comparisons of the segmentation results. We show that 2D models often struggle to maintain consistent object masks, and 3D-based methods, when given inconsistent masks input, also fail to improve the quality, leading to degraded performance. In contrast, our method is able to produce consistent estimations by effectively combining the 2D masks and 3D representation.
We also observe strong performance on the DAVIS dataset, demonstrating the generalization of our method. As shown in Figure [ Motion4D effectively improves the segmentation results of SAM2 by correcting temporally inconsistent regions. While the directly rendered masks from Motion4D achieve slightly lower scores, this is primarily due to limitations in reconstruction quality, which can affect -raiine mask boundaries.
4.2
2D Point Tracking Results
We next evaluate our method on the task of 2D point tracking. The DAVIS dataset, introduced for 2D point tracking by TAP-Vid [7], provides sparsely annotated point trajectories across real-war world

---

Table Comparison of 2D point tracking performance on DAVIS
7 and DyCheck [2] datasets.
DAVIS 2D Tracking
DyCheck 2D Tracking
Method
AJ
OA
AJ
8avg
OA
TAPIR 8
56.2
70.0
86.5
27.8
41.5
67.4
LocoTrack
62.9
75.3
87.2
26.1
40.5
66.9
CoTracker
16
76.1
88.3
33.9
73.0
BootsTAPIR
61.4
73.6
88.7
30.1
42.8
78.5
CoTracker3 [1
64.4
76,9
91.2
31.0
44.4
79,9
HyperNeRF 22
50.6
63.6
81.0
10.1
19.3
52.0
Deformable-3DD
55.6
69,4
82.0
20.9
63.9
Shape Motion 42
62.3
76.2
87.1
34.4
47.0
86.6
Motion4D
64.4
77.7
90.4
37.3
50.4
87.1
Table Comparison of 3D point tracking and novel view synthesis results on DyCheck [2] dataset.
3D Point Tracking
Novel View Synthesis
Method
EPE 6:00
63p
PSNR
SSIM
LPIPS
T-NeRF 12
15.60
0.55
0.55
HyPerNeRF 29
0.182
28.4
45.8
15.99
0.59
0.51
DynlBaR 22
0.252
11.4
24.6
13.41
0.48
0.55
Deformable-3D-cm
50
0.151
33.4
55.3
11.92
0.49
0.66
CoTracker [6
Depth Anything 47
0,202
34.3
57.9
TAPIR + Depth Anything 44
0.114
63.2
Shape of Motion 42
0.082
43.0
73.3
16.72
0.63
0.45
Motion4D
0.072
46.7
75.9
17.91
0.69
0.42
video sequences. It serves as a benchmark for assessing a model's ability to track arbitrary points under occlusion, deformation, and complex motion. Similarly, the DyCheck [12] dataset provides annotations of to 1S keypoints sampled at equally spaced time steps for each sequence. The dataset contains longer video sequences, making it well suited for evaluating long-ten nerg-term point tracking performance. We therefore evaluate the tracking performance terms of both position accuracy and occlusion accuracy, reporting the Average Jaccard (AJ), average position accuracy (< davg), and Occlusion Accuracy (OA).
Results. Table 2 summarizes the quantitative results for point-based tracking. Motion4D achieves superior accuracy compared both 2D tracking models and 3D>-based methods across various challenging scenarios. Moreover, Figure presents visual comparisons with strong 2D tracking baselines. As the ground-truth annotations in the DAVIS dataset are available only for sparsely sampled points, we visualize dense query point tracking results for qualitative comparison. Motion4D demonstrates clear advantages particularly in handling severe occlusions (first and second rows) and drastic motion (third and fourth rows), maintaining consistent and accurate trajectories throughout the sequences. In contrast, 2D tracking models tend to lose track of points under such challenging conditions, resulting in noticeable drift and fragmented tracks.
4.3
3D Point Tracking and Novel View Synthesis
Finally, we evaluate our method on the DyCheck [12] dataset for the tasks of 3D point tracking and novel view synthesis. Following the experimental setting of [42,
camera poses estimated by COLMAP [36] and initial monocular depths from Depth Anything [7]. For 3D point tracking, we generate ground-truth 3D trajectories by lifting the 2D keypoint annotations into 3D using LiDAR depth, and evaluate the tracking performance using the 3D end end-point-error (EPE) and the percentage of points that fall within given threshold 5:05 5cm and 5:16 10 10em. For novel view synthesis, we assess reconstruction quality using PSNR, SSIM, and LPIPS scores.
Results. We report quantitative results for both tasks in Table 3 Motion4D consistently outperforms state-of-the-art methods, including both 2D- and 3D-based approaches, across the tracking and novel view synthesis tasks. For 3D point tracking, method achieves lower 3D end-point error and

---

Table 4: Ablation studies on the DyCheck- VOS and DyCheck dataset.
Method
2D Update Sampling Seq. Opt. Global Opt. J&&F 1
AJ f
<
Gayz f
OA f
Ours (Full)
91.7
37.3
50.4
87.1
w/o Iterative Refinement
v
87.6
34.6
47.2
86.5
w/o Adaptive Sampling
V
88.9
35.1
47.7
84.2
Full Initialization
88.0
34.9
47.5
87.0
w/o Global Optimization
/
90.3
36.5
49.4
86.6
higher accuracy at both 3D and bD thresholds, demonstrating more precise and temporally stable motion estimation. For novel view synthesis, Motion4D produces sharper and more geometrically consistent renderings, achieving higher PSNR and SSIM scores and lower LPIPS values, indicating improved visual fidelity.
4.4 Ablation Studies
We ablate key components of our method on the DyCheck dataset, with results summarized in Table First, we validate the effectiveness of the iterative refinement strategy. In the ablation of "w/o Iterative Refinement", we remove the iterative update mechanism used to refine 2D priors during training, including both the update of 2D semantic masks and the confidence-based refinement of point tracks. In this setting, only the 3D scene representation is optimized, while the 2D priors remain fixed throughout training. In addition, the "w/o Adaptive Sampling" variant disables the densification step based on error-driven sampling. The results highlight the importance of both iterative refinement and adaptive sampling for achieving robust motion and segmentation performance.
Next, we validate the optimization strategies including sequential and global optimization stages. In the "Full Initialization" variant, we apply straightforward initialization over the entire video sequence without dividing into temporal chunks. This setting suffers from inconsistent supervision due to unreliable 2D priors, often leading to unstable training and degraded performance. In the "w/o Global Optimization" setting, we skip the final stage joint optimization over the full sequence and instead rely solely sequential updates. This results in local consistency but introduces slight temporal drift across chunks. As shown Table 4 both optimization stages are critical for achieving accurate and temporally coherent motion and segmentation results.
5 Conclusion
present Motion4D, a unified framework dynamic 3D scene reconstruction that jointly models motion and semantics via 4D scene reconstruction. By leveraging iterative refinement of 2D priors and a multi-stage optimization strategy, Motion4D achieves robust and temporally consistent performance across multiple tasks, including video object segmentation, point-based tracking, and novel view synthesis. We also introduce DyCheck- VOS, an annotated benchmark for segmentation in dynamic reconstruction scenes, and demonstrate the effectiveness of method on various tasks. While Motion4D achieves strong performance in dynamic scene understanding, still relies heavily on the quality of the underlying 3D reconstruction. In scenes with severe occlusions, low-texture regions, or inaccurate depth estimation, the reconstruction quality can degrade, which in turn affects the accuracy of motion and semantic predictions.
Acknowledgment
This research project is supported by the National Research Foundation (NRF) Singapore, under its NRF-Innestigatorshii Programme (Award ID. NRF-NRFF19900009) and the Tier grant MOE- T2EP20124-0015 from the Singapore Ministry of Education.
References
[1] Yash Bhalgat, Iro Laina, João F Henriques, Andrew Zisserman, and Andrea Vedaldi. Contrastive lift: 3d object instance segmentation by slow-fast contrastive fusion. In shirit- Conference on Neural Information Processing Systems, 2023, URL htttpp////ppeneeee.eeeeeeerrrrrr
10

---

[2] Jiazhong Cen, Jiemin Fang. Chen Yang. Lingxi Xie. Xiaopeng Zhang, Wei Shen, and Qi Tian. Segment any 3d gaussians. arXiv preprini urXiv:2312.00860, 2023.
[3] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception in the wild. Advances in neural information processing systems, 29, 2016.
[4] Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022.
[5] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Joon-Young Lee. Tracking anything with decoupled video segmentation. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1316-1326, 2023.
[6] Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon-Young Lee. Local all-pair correspondence for point tracking. In European Conference on Computer Vision, pages 306-325. Springer, 2024.
[7] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: benchmark for tracking any point in a video. Advances in Neural Information Processing Systems, 35: 1310013666, 2022.
[8] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vixion, pages 10061-10072, 2023.
[9] Carl Doersch, Pauline Lue, Yi Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco. Ross Goroshin, Joao Carreira, al. Bootstap: Bootstrapped training for tracking-an any- any-point. In Proceedings of the Asian Conference on Computer Vision, pages 3257-3274 2024.
[10] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479 12488, 2023.
[11] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1112-5721,20 2021.
[12] Hang Gao. Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances Neural Information Processing Systems, 35:33768-33780, 2022.
[13] Ruiqi Gao*, Aleksander Holynski*, Philipp Henzler, Arthur Brussee, Ricardo Martin-Brualla, Brualla, Pratul P. Srinivasan. Jonathan Barron, and Ben Poole*. Cat3d: Create anything in with multi-view diffusion models. Advances in Neural Information Processing Systems, 2024.
[14] Shengxiang Ji, Guanjun Wa, Jiemin Fang. Jiazhong Cen, Taoran Yi. Wenyu Liu, Qi Tian, and Xinggang Wang. Segment any 4d gaussians. arXiv preprint arXiv:2407.0m 045544 2024.
[15] Nikita Karaev, lurii Makarov, Jianyuan Wang. Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.118111 2024.
[16] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It better to track together. In European Conference Computer Vision, pages 08-3. Springer, 2024.
[17] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1 2023.
[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead. Alexander C Berg. Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pages 4015 4026, 2023.
[19] Skanda Koppula, Ignacio Rocco, Yi Yang, Joe Heyward, João Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. TAPVid-3D: A benchmark for tracking any point in 3D. Advances in Neural Information Processing Systems, 2024.
[20] Hongyang Li, Hao Zhang. Shilong Liu, Zhaoyang Zeng. Tianhe Ren, Feng Li, and Lei Zhang. TAPTR: Tracking Any Point with Transformers as Detection. In European Conference on Computer Vision, pages 57-75. Springer, 2024.
11

---

[21] Yun-Jin Li, Mariia Gladkova, Yan Xia, and Daniel Cremers. Sadg; Segment any dynamic gaussian without object trackers. arXiv preprint S.Kiv:4411.192990 2024.
[22] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[23] Fangfu Liu, Wengiang Sun, Hanyang Wang. Yikai Wang. Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan, Reconx: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767, 2024.
[24] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809. IEEE, 2024.
[25] Hidenobu Matsuki, Riku Murai, Paul HJ Kelly, and Andrew J Davison. Gaussian splatting slam. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 18039- 18048, 2024.
[26] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[27] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding, ACM transactions on graphics (TOG), (1()::1-15, 2022.
[28] Scoung Wug Oh, Joon- Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space- time memory networks. In Proceedings of the IEEE/CVF international conference on computer vision. pages 9226-922333 2019.
[29] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla Brualla, and Steven M Seitz. Hypernerf: A dimees-ai representation for topologi- cally varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
[30] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Homung, A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
[31] Albert Pumarola, Euric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 10318-10327, 2021.
[32] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. arXiv preprint arXiv:2312.16084, 2023.
[33] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3)1162311677, 2020.
[34] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam Segment anything in images and videos. arXiv preprint arXiv:2408.007/ 2024.
[35] Samuel Rota Buld, Lorenzo Porzi, and Peter Kontschieder. Revising densification in gaussian splatting, In European Conference on Computer Vision pages 347-362. Springer, 2024.
[36] Johannes L Schonberger and Jan Man-Michael Frahm. Structure- motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104 4113, 2016.
[37] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Buló, Norman Müller, Matthiss Nießner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d seene understanding with neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9043-9052, 2023.
[38] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjic Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20675-20685, 2024.
[39] Zachary Teed and Jia Deng. Raft: Recurrent pairs field transforms for optical flow. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part II 16. pages 402 419. Springer, 2020.
12

---

[40] Fengrui Tian, Yueqi Duan, Angtian Wang, Jianfei Guo, and Shaoyi Du. Semantic flow: Learning semantic field of dynamic scenes from monocular videos. arXiv preprint arXiv:2404.05163, 2024.
[41] Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi SM Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of scenes. arXiv preprint arXiv:2111.13260, 2021.
[42] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407. 13764, 2024.
[43] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang. Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 20310-20320, 2024.
[44] Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception with web stereo data supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 311-320, 2018.
[45] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. vos:: A large-scale vidco object segmentation benchmark. arXiv preprint 180.:1800.033227 2018.
[46] Chi Yan, Delin Qu Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19604, 2024.
[47] Lihe Yang, Bingyi Kang, Zilong Huang. Xiaogang Xu. Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10371-10381, 2024.
[48] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875 21911, 2025.
[49] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang, Real-time photorealistic dynamic representation and rendering with 4d gaussian splatting. International Conference on Learning Representations (ICLR). 2024.
[50] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, Xiaogang Jin. Deformable 3d gaussians for nigh-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20331 00331-20341, 2024.
[51] Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lci Ke. Gaussian grouping: Springer, 2024.
Segment and edit anything in
3d scenes. In European Conference on Computer Vision, pages
[52] Shuaifeng Zhi, Tristan Laidlow. Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838-15847, 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions of this paper are clearly stated in both the abstract and introduction, including the Motion4D iterative optimization framework. The claims are aligned with the methods and results presented, accurately reflecting the paper's scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
Justification:
The paper does not include theoretical proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the framework, optimization procedure, datasets, and evaluation metrics, ensuring that the main experimental results can be reproduced independently.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The code not included in the submission to preserve anonymity, but we plan
to release it publicly upon acceptance to enable full reproducibility of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
www.c//wweeeeeeee Cooeee/mddeeSbbiiic SowpoddeummsssiooPP... www.amisssiooooiiyy... for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides overview of the experimental settings in the main text, with full training and testing details provided in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the standard evaluation protocol used in prior work and report single-run results in tabular format without error bars. The compared methods and datasets do not typically include statistical variation measures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide details about the compute resources used in the supplementary material alongside other implementation details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The method has potential positive impacts in areas like robotics and AR/VR. but it could also be misused in surveillance applications. We encourage responsible and ethical use of the technology.
Guidelines:
17

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper does not release models or datasets with high risk of misuse, so no specific safeguards are necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external datasets and models used in this work are properly cited, and we ensured that their licenses and terms of use were respected as specified by their original authors.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
18

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pa se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce benchmark built top of existing dataset. Details of the benchmark, including evaluation protocol and annotations, are documented and will be released with the asset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects or crowdsourcing, so IRB approval is not applicable.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This research does not use LLMs as part of its core methods. Any use LLMs was limited to writing or editing support and does not impact the scientific content of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https://neure) /otee//eeeeeaeeeeeee////00055 for what should or should be described.
20


Reviews:
[{"review_id": "tFDlTYwfIs", "reviewer": "Reviewer_iUVL", "summary": "This submission presents a method for simultaneously reconstructing and tracking semantics and points in a 4D scene given input posed RGB imagery. The method leverages pretrained SAM2, Track Any Point, and Depth Anything models for segmentation, depth, and tracking priors. A 3DGS model with temporally varying gaussians is optimized via a series of optimization stages (first sequential motion refinement, then sequential semantic refinement, then a global optimization where all frames and layers are optimized together. The method demonstrates impressive results.", "questions": "1. Please further explain the relationship between the confidence and uncertainty maps. Please visualize if possible.\n\n2. Please further explain the global motion bases.\n\n3. Please add visualizations if possible to help clarify how the optimization progresses over time (from a novel view).", "limitations": "The limitations section could definitely be more extensive. What are some failure cases? Do any of those occur in the DyCheck-VOS dataset?", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "This is a compelling submission with strong results (on a modest number of scenes) and relatively thorough ablations. The proposed system is complex but manages to simultaneously optimize multiple per-frame priors into a coherent 4D whole in a way that I have not seen before. I don't personally know of anything immediately comparable, though other reviewers with more experience in this area may.\n\nI do have some clarity and experimental questions/concerns, but I do not think they are severe enough to warrant rejection.\n\nFirst, I do not fully understanding the confidence and uncertainty maps nor their exact relationship. I think it would be very helpful to see some video results that show the optimization stages progressing along with the underlying 4DGS representation shown from multiple views and colored by different fields. Without that, even a good text description makes it hard to really grasp the method.\n\nAdditionally, the deformation field is a little confusing as well-- the notation doesn't really make it easy to understand the relationship between # of bases, # of gaussians, and # of timestamps. There is also no ablation regarding this step, nor an easy intuition for why it is required.\n\nThird, the proposed evaluation is quite small-- the DyCheck-VOS dataset is only 14 sequences. Only a handful of stills from those are shown, making it hard to understand the breadth of the data and how challenging it is (or isn't).", "was_revised": true, "final_justification": "All other reviews concur with initial accept recommendation. Author rebuttal is reasonable and no other major concerns were raised during rebuttal process.", "rebuttal": "**Responses to Reviewer iUVL**\n\n**Q1: Lack of visualization of confidence and uncertainty maps.**\n\n**A1**: We will include these visualizations in the revised version, but cannot provide them here due to the rebuttal’s format constraints. The \"uncertainty logits\" refer to the learnable attributes attached to the 3D Gaussians (one uncertainty logit per 3D Gaussian). We use this 3D field (the learnable attributes) to render 2D views of the confidence maps, which is similar to how 3D Gaussian colors are rendered into images. This approach allows us to optimize a 3D-consistent uncertainty field that reflects the underlying scene geometry, and use the rendered confidence maps to update 2D priors.\n\n**Q2: Lack of visualization of the optimization progress.**\n\n**A2**: We will include this visualization in the final paper. Nonetheless, we note that Figure 3 has provided a related illustration of the optimization progress. The figure shows our Motion4D progressively refines the inconsistent 2D priors and produces temporally consistent results during the sequential optimization stage.\n\n**Q3: The deformation field is a bit confusing.**\n\n**A3**: The deformation field follows the formulation introduced in prior work [1]. Instead of defining $N$ bases, where $N$ is the number of Gaussians, we define $B$ global bases ($B \\ll N$), where each base contains $T$ elements in $\\mathbb{SE}(3)$. We assign a coefficient vector of length $B$ for each 3D Gaussian, whose per-frame transformation is then computed as a weighted sum of the $B$ elements in $\\mathbb{SE}(3)$ at that time step. This design significantly reduces the number of parameters and enables smooth temporal deformation. Although we adopt the same deformation field formulation, we extend it to integrate with the adaptive resampling module in Motion4D. This is why we kept its description concise in the method section. We also evaluated a variant using per-Gaussian transformation. While this increases flexibility, it results in slightly worse performance ($<\\delta_{avg}$ drops from 50.4 to 49.6 on DyCheck) due to overfitting and unstable tracking estimation. In the final paper, we will expand the explanation and add this ablation to clarify its effect.\n\n\n**Q4: The proposed evaluation is quite small.**\n\n**A4**: The proposed DyCheck-VOS benchmark consists of long video sequences, each with 200–500 frames. For comparison, the DAVIS 2017 validation set for segmentation and tracking includes 30 short videos, each with fewer than 100 frames. As a result, DyCheck-VOS introduces greater challenges in long-term tracking, especially in handling frequent occlusions, spatial alignment, and object re-identification over extended sequences. Due to the difficulty of annotating dense point trajectories for Track-Any-Point, a larger benchmark is currently unavailable for evaluation. We will include additional examples to better showcase the complexity and challenges posed by Dycheck-VOS.\n\n**Q5: Failure cases on DyCheck-VOS.**\n\n**A5**: Our method may fail in scenarios where the underlying 3D reconstruction is unreliable, typically due to inaccurate SfM camera poses or extremely challenging scene geometry. Nonetheless, our proposed iterative refinement framework can address several failure cases when the 2D priors are noisy and unreliable. These cases are shown in Figure 3, where the input segmentation and tracking results are poorly estimated. In the final paper, we will elaborate on the causes of such failure cases and discuss potential strategies for mitigating them in future work.\n\n[1] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint arXiv:2407.13764, 2024."}, {"review_id": "PzuZ9F11IT", "reviewer": "Reviewer_D7AY", "summary": "In this paper, the authors propose an iterative optimization framework to jointly refine the 4D motion prediction and segmentation results of videos. They directly start with 2D tracks, depth prediction, and segmentation maps from well-established pre-trained models. Then they propose to construct the scene with 3DGS representation, which is enriched with semantic features and motion deformation field. The paper then proposes an iterative motion refinement technique with an adaptive resampling strategy and an interactive iterative semantic refinement module. For optimization, they develop sequential optimization and global optimization to build both short and long temporal relations.", "questions": "1. It would be better if the authors could provide visualization comparisons of ablation settings (just like column (d)(e) in Figure 4 clearly showing the improvement from rendered masks to refined SAM2 masks). For example, the authors mentioned that without global optimization, there is temporal drift across chunks. However, this is not evident from mere numerical results. A visualization comparison would be more intuitive. \n\n2. I am curious about why not optimizing depth estimation. Depth is a crucial part of 3DGS scene reconstruction and point tracking.\n\n3. Since point track and depths are both available, it would be better if the authors could compare their results with 3D point tracking benchmarks.\n\n4. I'm not very familiar with video segmentation, but I'm curious about why not evaluate the mIoU metric for segmentation results.\n\n5. What is the computational cost (time, gpu, params) of each optimization process? Since this is a refinment method, high fps and lightweight are crucial.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths:\n\n1. The paper is overall well-written and easy to follow. The motivation is clearly stated.\n\n2. The idea of jointly optimizing 4D motion and semantic segmentation via 3DGS representation is interesting, novel, and technically sound.\n\n3. The quantitative comparisons with the previous method and the visualization results strongly convince the effectiveness of the proposed method.\n\n#### Weaknesses:\n\n1. The overall concept of the paper is rather incremental. Currently, the community is expecting systematic design with a unified framework and joint optimization of multiple tasks. I acknowledge the efforts made by the authors to jointly optimize low-level 3D motion prediction and high-level semantic segmentation. Their technique is also effective. However, the paper still seems like an auxiliary refinement module to be inserted in pre-trained models. I like the idea of unifying with 3DGS representation, but it would be better if the authors could draw a bigger picture with this concept, rather than just implement it as a refinement module.\n\n2. The experiment part is not thorough enough. It would be better if more visualization comparisons about each ablation setting could be provided to vividly show their effectiveness.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns, and as a result, I maintain my original positive score. I hope the authors could incorporate the points discussed in the rebuttal into the revised version of their paper.", "rebuttal": "**Responses to Reviewer D7AY**\n\n**Q1: The overall concept of the paper is rather incremental.**\n\n**A1:** We thank the reviewer for the valuable feedback. While Motion4D leverages existing 2D and 3D components, its contribution goes beyond serving as an auxiliary refinement module. Motion4D constructs a joint 4D representation that integrates motion, depth, and semantic cues. This directly addresses the reviewer’s call for “a unified framework and joint optimization of multiple tasks”. Crucially, the joint optimization of 2D priors and an explicit 3D representation is not only complementary but fundamental to the effectiveness of our approach. We will revise the final paper to clarify and emphasize that Motion4D is a carefully designed joint optimization framework rather than a simple refinement pipeline.\n\n**Q2: Visualization of ablation settings.**\n\n**A2:** We appreciate the suggestion to include additional visualizations and will incorporate them in the final paper. Although figures cannot be attached in this rebuttal, we highlight the key findings from our ablation studies. Particularly, removal of the adaptive resampling module leads to unreliable reconstruction in fast-moving or occluded regions such as the legs of dynamic objects. Figure 3 of the supplementary material shows the importance of this component, where the baseline without resampling exhibits noticeable artifacts and even missing regions in the challenging regions. In addition, without global optimization, we observed that the Gaussians may be shifted abruptly between chunks, leading to clear motion drift.\n\n**Q3: Optimizing depth estimation.**\n\n**A3:** Depth is optimized jointly with 2d tracks as part of the iterative motion refinement process. Specifically, Equation (5) defines the depth loss as the discrepancy between the 3D-rendered depth and the 2D depth priors, formulated in a manner consistent with the loss for 2D tracks.\n\n**Q4: Comparison on 3D point tracking benchmarks.**\n\n**A4**: We have evaluated 3D point tracking performance on the DyCheck benchmark, which includes tracking annotations and LiDAR depth. The results are reported in Table 1 of the supplementary material. Our Motion4D demonstrates its ability to maintain accurate long-term 3D correspondences by achieving consistent improvements over existing 3D methods. We will emphasize these results and provide more detailed discussion in the final paper.  \n\n**Q5: Evaluating the mIoU metric for segmentation results.**\n\n**A5:** Our evaluation follows the standard $\\mathcal{J}$ & $\\mathcal{F}$ metric widely adopted in video object segmentation. Here, the $\\mathcal{J}$ score corresponds to the mean Intersection over Union (mIoU) across frames, and the $\\mathcal{F}$ score measures contour accuracy. This means that our reported $\\mathcal{J}$ & $\\mathcal{F}$ results have included mIoU which ensures comparability with other VOS benchmarks.\n\n**Q6: Computational cost of the optimization process.**\n\n**A6:** Our model is trained and evaluated entirely on a single RTX 4090 GPU and contains approximately 150M parameters, depending on the scope of the scene. The training phase of our Motion4D is not real-time since it relies on predictions from multiple pre-trained networks and follows the per-scene optimization paradigm of 3DGS-based methods. Despite this, our runtime is comparable to other 3D-based pipelines: optimizing a scene with 100 frames typically takes approximately 30 minutes on a single 4090 GPU, close to the baseline of 4DGS (~24min). The rendering of segmentation and tracking results is fast, at about 75 fps, after optimization since the 4DGS representation can be rendered efficiently. We will clarify in the final paper that real-time performance during optimization is beyond our current scope. The primary goal of our paper is to deliver accurate and temporally consistent results over long sequences, which is challenging for lightweight pre-trained 2D models."}, {"review_id": "uVdOnwtBtf", "reviewer": "Reviewer_XuWq", "summary": "This paper proposes Motion4D, a motion and semantics understanding framework. It lifts priors from 2D foundation models to 4D space with dynamic Gaussians. To alleviate the spatial misalignment and temporal flickering issues in 2D models, Motion4D proposes a set of technologies including iterative gt refinement, dual phases optimization and error-induced Gaussian densitification. To validate the effectiveness of the proposed method, this paper further introduces the DyCheck-VOS dataset. Extensive experiments are conducted.", "questions": "1. The iterative motion refinement module: the confidence $\\omega(p)$ is learned adaptively, but its gt is manually designed. Can we replace it with a hand-crafted value? Like $\\delta - ||I_t(p), I_{t'}(p) ||$\n2. The iterative pipeline shows similar mechanism with SA3D [1]. More discussion about it is encouraged.\n\n[1] Cen, J., Zhou, Z., Fang, J., Shen, W., Xie, L., Jiang, D., ... & Tian, Q. (2023). Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems, 36, 25971-25990.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The method seems solid. Experimental results are good.\n2. A new dataset is provided, providing good for following studies.\n3. Writing is clear.\n\nWeaknesses:\n\n1. The method is complex, making it hard for reproducing.\n2. Also, the complexity of the method makes me doubt about the robustness of it. Though the ablation study demonstrate the effectiveness of each component independently, it can not provide any evidence for the generalization ability of the proposed method. Can the method be simplified?\n3. Figure 2 is important but unclear. The figure looks good but cost me for some time to totally understand. Some more annotations can be added to enhance its readability.\n4. The proposed method requires per-scene optimization, which brings time overhead compared with e2e methods. It is essential to analyze its time consumption.\n5. Table 3 is unclear. I think there is a typo. Check the check mark of the row of w/o global optimization.", "was_revised": true, "final_justification": "The response addresses most of my concern thus I keep the score.", "rebuttal": "**Responses to Reviewer XuWq**\n\n**Q1: The method is complex and hard to reproduce.**\n\n**A1:** Our design is intentionally modular to ensure simplicity and ease of replication. Each component (SAM2, Track Any Point, Depth Anything, and 4DGS) is an independent and standalone module, which makes the overall pipeline straightforward to reproduce. Our primary contribution is the unified optimization framework that connects these modules. This framework is conceptually simple, effective, and clearly detailed in Sections 3.2–3.4. We have also indicated in Line 20 of the paper that we will release the complete codebase including training and evaluation scripts upon acceptance to ensure full reproducibility.\n\n**Q2: Robustness and generalization of the method.**\n\n**A2:** The modular design of our method enhances robustness. Large pre-trained networks such as SAM2 and Track Any Point are known for their strong generalization across diverse datasets. By using these models, we improve the robustness of the 4DGS-based scene representation. As shown in Tables 1 and 2, previous 3D representation methods often fail to generalize to casual dynamic scenes in the DAVIS and DyCheck datasets, where uncontrolled motion and noise violate their assumptions. Although 2D networks generalize well in many scenarios, Figures 1 and 4 demonstrate their limitations under severe occlusions and complex dynamics. In such cases, 3D reasoning is essential to achieve consistent and reliable results.\n\n\n**Q3: Figure 2 is unclear.**\n\n**A3:** We appreciate the reviewer’s feedback regarding Figure 2. In the revised version, we will improve its readability by adding annotations to key components and refining the caption to better explain the pipeline.\n\n**Q4: Time consumption analysis.**\n\n**A4:** Our method differs from single-task end-to-end 2D models by building a full 4D representation that jointly produces segmentation masks, tracking, depth and rendered images for both training and novel views. This richer capability inherently requires per-scene optimization, as we model the scene’s 3D geometry and dynamics jointly rather than processing frames independently. Despite the joint optimization, our runtime remains comparable to other 3D-based pipelines. Optimizing a scene with 100 frames typically takes about 30 mins on a single 4090 GPU, which is close to the 4DGS baseline (~24mins). After optimization, the rendering is highly efficient, at about 75 fps, which enables real-time rendering and querying for segmentation and tracking results. We will clarify by adding a detailed runtime analysis in our final paper. \n\n**Q5: Table 3 is unclear.**\n\n**A5:** We thank the reviewer for pointing this out. There is indeed a typo in Table 3, which we will correct.\n\n**Q6: Replacing the confidence $w(p)$ with a hand-crafted value.**\n\n**A6:** The confidence $w(p)$ is an adaptively learned logit, with its ground-truth defined based on the rendered result. This is similar to TAP model designs, where the predicted confidence logit is supervised by a binary signal, which is set to 1 when the prediction is within a predefined threshold and 0 otherwise. This binary signal serves only as a training target and enables the model to learn a smooth confidence field in 3D space. In contrast, hand-crafted confidence measures often yield brittle and noisy weights, particularly in regions with similar textural structures.\n\n**Q7: Comparison with SA3D.**\n\n**A7:** Our iterative pipeline shares a similar prompting mechanism with SA3D, but differs substantially in design and scope: \n\n- **(a) Target setting:** While SA3D is designed for static scenes and focuses solely on propagating a single-view mask to multi-view masks, our method targets monocular videos in dynamic scenes and refines SAM masks across the entire sequence to construct a coherent semantic representation. \n\n- **(b) Temporal consistency:** In addition to multi-view consistency, our method explicitly enforces temporal consistency by combining SAM prompting with sequential optimization of semantics as shown in Figure 3(b). This enables masks to propagate smoothly across frames and align segmentation with dynamic motion.\n\n- **(c) Iterative refinement:** Our approach jointly refines both the 2D priors and the 3D semantic field, instead of only updating a semantic voxel grid as in SA3D. This design allows us to progressively improve mask quality across frames and views by re-prompting SAM using the corrected inputs. In comparison, SA3D queries SAM only once per frame with no feedback loop. \n\nWe will include this discussion and compare our method with SA3D in the final paper."}, {"review_id": "UaNlbDWNCS", "reviewer": "Reviewer_8CeR", "summary": "The paper addresses a critical challenge in dynamic scene understanding: the lack of 3D consistency in 2D foundation models (e.g., temporal flickering and spatial misalignment). \nTo this end, this paper presents a framework that integrate 2D priors from foundation models into a unified 4D scene representation. \nA two-part iterative optimization framework is designed to updates motion and semantic fields in consecutive stages and jointly refines all attributes for long-term coherence.\nThe paper evaluates on multiple tasks (segmentation, point tracking, novel view synthesis) and benchmarks (DyCheck-VOS, DAVIS). The results show clear improvements over existing 2D and 3D methods.", "questions": "See the weaknesses above.\n\nHow does the adaptive resampling strategy handle regions that are under-represented due to occlusion? Does it rely on the depth estimates, which might be inaccurate in occluded regions?", "limitations": "Sequential-global optimization is resource-heavy and time-consuming. The 2D foundation models can be used directly without any additional iteratively optimization, even though the proposed framework improves the 2D priors with more spatial consistent results, it might be hard to support tasks requires faster inference.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1: The integration of 2D priors (SAM2, TAP, Depth Anything) into a dynamic 3DGS representation with iterative refinement is novel.\n\nS2: The two-part optimization (sequential and global) is a well-motivated approach to correct 2D priors using 3D reconstruction feedback and handle error accumulation.\n\nS3: The use of 3D confidence maps for motion refinement and adaptive resampling based on per-pixel errors is novel.\n\nS4: A new benchmark is provided. DyCheck-VOS addresses the lack of dynamic-scene segmentation datasets.\n\n\nWeaknesses:\n\n\nW1: 2D Prior Dependency: Performance degrades with low-quality inputs.\n\nW2: In Table 1, the improvement on DAVIS 2017 val is trivial, or no improvement is observed (SAM2 vs. Motion4D + SAM2).\n\nW3: Missing ablation on optimization tasks: like only using part of the 2D tasks and foundation models, e.g., one or two for the two tasks (tracking, depth, and segmentation).\n\nW4: The contribution of each component (3D confidence map, adaptive resampling, iterative semantic refinement) could be isolated more clearly.", "was_revised": true, "final_justification": "While I still think the improvement on DAVIS 2017 val is trivial, the rebuttal addressed most of my concerns. So I keep my rating as Borderline Accept.", "rebuttal": "**Responses to Reviewer 8CeR**\n\n**Q1: 2D prior dependency.**\n\n**A1:** Although our method uses multiple 2D priors, our objective is to ***refine and correct these priors*** through 3D consistency and temporal coherency. As demonstrated by the results in Figure 1 and 3, our iterative optimization substantially improves the input 2D priors even when they are noisy. Our design offers a distinct advantage over both pure 2D and 3D methods. Pure 2D networks often produce inconsistent outputs, particularly in regions with severe occlusions and motion ambiguity. Furthermore, 3D baselines struggle to leverage priors as effectively as our approach. Consequently, our refinement process is essential to achieve stable and robust results despite inheriting some limitations of the 2D priors. \n\n**Q2: Improvement on DAVIS 2017 val is trivial.**\n\n**A2:** The modest gain on DAVIS 2017 val arises because the baseline (SAM2) already achieves near-saturated performance (over 90%) on this split. While diverse in content, the DAVIS dataset contains short video clips (fewer than 100 frames) with salient foreground targets and limited occlusion. In contrast, our method shows clear improvements on more challenging scenes (over 400 frames) in the DyCheck dataset (Tables 1 and 2), which require long-term tracking across extended sequences. Even on DAVIS val, our method enhances temporal consistency across frames (Figure 1), which is not fully captured by raw performance scores.\n\n**Q3: Missing ablation on optimization tasks.**\n\n**A3:** Tracking and depth cannot be treated as independent tasks as each track corresponds to a 3D trajectory. Since 3D tracks are not directly observable, we supervise the motion field using both 2D tracking and depth cues. Ablating either signal leads to a significant performance drop: using only tracking reduces $<\\delta_{avg}$ on DyCheck from 50.4 to 41.6, and using only depth yields 25.9. This highlights that both signals are essential and complementary. Similarly, training solely on segmentation is not viable since the semantic field relies on the underlying 3D geometry. Without the motion field (tracking and depth), the model fails to converge and produces unusable results. We will include these additional ablations in the final paper to clarify the necessity of the full design.\n\n**Q4: The contribution of each component.**\n\n**A4:** The effectiveness of our adaptive resampling module has been shown in the ablation study in Table 3 (“w/o Adaptive Sampling”). An ablation for iterative semantic refinement, which is applied during  the sequential optimization stage, has also been presented in Table 3 (“Full Initialization”). In addition, we show the ablation results without confidence map here, which achieves 89.4 $\\mathcal{J}$ & $\\mathcal{F}$ and 48.3 $<\\delta_{avg}$. This shows that our confidence map is effective in improving the performance. We will make the contribution of each component more clearly in the final paper. \n\n\n**Q5: How does the adaptive resampling strategy handle regions that are under-represented due to occlusion?**\n\n**A5:** Our adaptive resampling strategy operates only on visible regions. Fully occluded areas cannot be directly reconstructed since their true geometry is inherently unobservable from the available data. Nevertheless, as shown by the tracking results in Figures 1 and 5, our 3D representation remains more stable in under-represented regions compared to the 2D networks. This suggests that our 3D representation is capable of producing plausible and coherent results by enforcing spatial and temporal consistency, even when direct observations are unavailable and 2D predictions are drifting. \n\n**Q6: Sequential-global optimization is resource-heavy and time-consuming.**\n\n**A6:** We acknowledge that achieving real-time performance during optimization is beyond our current scope. However, our primary goal is to produce accurate and temporally consistent results over long sequences, which is challenging for lightweight pre-trained 2D models. Despite the computational overhead, our runtime is comparable to other 3D-based pipelines: optimizing a scene with 100 frames typically takes approximately 30 minutes on a single 4090 GPU, close to the baseline of 4DGS (~24min). After optimization, the rendering is highly efficient, at about 75 fps, which enables real-time rendering and querying for segmentation and tracking results."}]

Author rebuttal:

Meta-review: This paper tackles the problem of dynamic 3D reconstruction and proposed a framework which integrates 2D priors into a 4D representation in the form of a dynamic 3dgs. This final representation is optimized through a two stage process which jointly refines all attributes for extended time frame. This representation is then tested on several different downstream tasks with promising results.

The reviewers were positive (3 borderline accept, 1 accept) with the reviewers commending the overall framework as well as results. Reviewers were concerned about the technical novelty of the paper as well clean ablations on the component parts of the framework. However, all were positive in the end.

I also advocate for acceptance.
Decision: Accept (poster)