Title: Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video object segmentation

3D point tracking

Monocular dynamic novel view synthesis

4D scene reconstruction
Abstract: Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses a critical challenge in dynamic scene understanding: the lack of 3D consistency in 2D foundation models (e.g., temporal flickering and spatial misalignment). \nTo this end, this paper presents a framework that integrate 2D priors from foundation models into a unified 4D scene representation. \nA two-part iterative optimization framework is designed to updates motion and semantic fields in consecutive stages and jointly refines all attributes for long-term coherence.\nThe paper evaluates on multiple tasks (segmentation, point tracking, novel view synthesis) and benchmarks (DyCheck-VOS, DAVIS). The results show clear improvements over existing 2D and 3D methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nS1: The integration of 2D priors (SAM2, TAP, Depth Anything) into a dynamic 3DGS representation with iterative refinement is novel.\n\nS2: The two-part optimization (sequential and global) is a well-motivated approach to correct 2D priors using 3D reconstruction feedback and handle error accumulation.\n\nS3: The use of 3D confidence maps for motion refinement and adaptive resampling based on per-pixel errors is novel.\n\nS4: A new benchmark is provided. DyCheck-VOS addresses the lack of dynamic-scene segmentation datasets.\n\n\nWeaknesses:\n\n\nW1: 2D Prior Dependency: Performance degrades with low-quality inputs.\n\nW2: In Table 1, the improvement on DAVIS 2017 val is trivial, or no improvement is observed (SAM2 vs. Motion4D + SAM2).\n\nW3: Missing ablation on optimization tasks: like only using part of the 2D tasks and foundation models, e.g., one or two for the two tasks (tracking, depth, and segmentation).\n\nW4: The contribution of each component (3D confidence map, adaptive resampling, iterative semantic refinement) could be isolated more clearly.\n\nQUESTIONS:\nSee the weaknesses above.\n\nHow does the adaptive resampling strategy handle regions that are under-represented due to occlusion? Does it rely on the depth estimates, which might be inaccurate in occluded regions?\n\nLIMITATIONS:\nSequential-global optimization is resource-heavy and time-consuming. The 2D foundation models can be used directly without any additional iteratively optimization, even though the proposed framework improves the 2D priors with more spatial consistent results, it might be hard to support tasks requires faster inference.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Motion4D, a motion and semantics understanding framework. It lifts priors from 2D foundation models to 4D space with dynamic Gaussians. To alleviate the spatial misalignment and temporal flickering issues in 2D models, Motion4D proposes a set of technologies including iterative gt refinement, dual phases optimization and error-induced Gaussian densitification. To validate the effectiveness of the proposed method, this paper further introduces the DyCheck-VOS dataset. Extensive experiments are conducted.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The method seems solid. Experimental results are good.\n2. A new dataset is provided, providing good for following studies.\n3. Writing is clear.\n\nWeaknesses:\n\n1. The method is complex, making it hard for reproducing.\n2. Also, the complexity of the method makes me doubt about the robustness of it. Though the ablation study demonstrate the effectiveness of each component independently, it can not provide any evidence for the generalization ability of the proposed method. Can the method be simplified?\n3. Figure 2 is important but unclear. The figure looks good but cost me for some time to totally understand. Some more annotations can be added to enhance its readability.\n4. The proposed method requires per-scene optimization, which brings time overhead compared with e2e methods. It is essential to analyze its time consumption.\n5. Table 3 is unclear. I think there is a typo. Check the check mark of the row of w/o global optimization.\n\nQUESTIONS:\n1. The iterative motion refinement module: the confidence $\\omega(p)$ is learned adaptively, but its gt is manually designed. Can we replace it with a hand-crafted value? Like $\\delta - ||I_t(p), I_{t'}(p) ||$\n2. The iterative pipeline shows similar mechanism with SA3D [1]. More discussion about it is encouraged.\n\n[1] Cen, J., Zhou, Z., Fang, J., Shen, W., Xie, L., Jiang, D., ... & Tian, Q. (2023). Segment anything in 3d with nerfs. Advances in Neural Information Processing Systems, 36, 25971-25990.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose an iterative optimization framework to jointly refine the 4D motion prediction and segmentation results of videos. They directly start with 2D tracks, depth prediction, and segmentation maps from well-established pre-trained models. Then they propose to construct the scene with 3DGS representation, which is enriched with semantic features and motion deformation field. The paper then proposes an iterative motion refinement technique with an adaptive resampling strategy and an interactive iterative semantic refinement module. For optimization, they develop sequential optimization and global optimization to build both short and long temporal relations.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths:\n\n1. The paper is overall well-written and easy to follow. The motivation is clearly stated.\n\n2. The idea of jointly optimizing 4D motion and semantic segmentation via 3DGS representation is interesting, novel, and technically sound.\n\n3. The quantitative comparisons with the previous method and the visualization results strongly convince the effectiveness of the proposed method.\n\n#### Weaknesses:\n\n1. The overall concept of the paper is rather incremental. Currently, the community is expecting systematic design with a unified framework and joint optimization of multiple tasks. I acknowledge the efforts made by the authors to jointly optimize low-level 3D motion prediction and high-level semantic segmentation. Their technique is also effective. However, the paper still seems like an auxiliary refinement module to be inserted in pre-trained models. I like the idea of unifying with 3DGS representation, but it would be better if the authors could draw a bigger picture with this concept, rather than just implement it as a refinement module.\n\n2. The experiment part is not thorough enough. It would be better if more visualization comparisons about each ablation setting could be provided to vividly show their effectiveness.\n\nQUESTIONS:\n1. It would be better if the authors could provide visualization comparisons of ablation settings (just like column (d)(e) in Figure 4 clearly showing the improvement from rendered masks to refined SAM2 masks). For example, the authors mentioned that without global optimization, there is temporal drift across chunks. However, this is not evident from mere numerical results. A visualization comparison would be more intuitive. \n\n2. I am curious about why not optimizing depth estimation. Depth is a crucial part of 3DGS scene reconstruction and point tracking.\n\n3. Since point track and depths are both available, it would be better if the authors could compare their results with 3D point tracking benchmarks.\n\n4. I'm not very familiar with video segmentation, but I'm curious about why not evaluate the mIoU metric for segmentation results.\n\n5. What is the computational cost (time, gpu, params) of each optimization process? Since this is a refinment method, high fps and lightweight are crucial.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission presents a method for simultaneously reconstructing and tracking semantics and points in a 4D scene given input posed RGB imagery. The method leverages pretrained SAM2, Track Any Point, and Depth Anything models for segmentation, depth, and tracking priors. A 3DGS model with temporally varying gaussians is optimized via a series of optimization stages (first sequential motion refinement, then sequential semantic refinement, then a global optimization where all frames and layers are optimized together. The method demonstrates impressive results.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a compelling submission with strong results (on a modest number of scenes) and relatively thorough ablations. The proposed system is complex but manages to simultaneously optimize multiple per-frame priors into a coherent 4D whole in a way that I have not seen before. I don't personally know of anything immediately comparable, though other reviewers with more experience in this area may.\n\nI do have some clarity and experimental questions/concerns, but I do not think they are severe enough to warrant rejection.\n\nFirst, I do not fully understanding the confidence and uncertainty maps nor their exact relationship. I think it would be very helpful to see some video results that show the optimization stages progressing along with the underlying 4DGS representation shown from multiple views and colored by different fields. Without that, even a good text description makes it hard to really grasp the method.\n\nAdditionally, the deformation field is a little confusing as well-- the notation doesn't really make it easy to understand the relationship between # of bases, # of gaussians, and # of timestamps. There is also no ablation regarding this step, nor an easy intuition for why it is required.\n\nThird, the proposed evaluation is quite small-- the DyCheck-VOS dataset is only 14 sequences. Only a handful of stills from those are shown, making it hard to understand the breadth of the data and how challenging it is (or isn't).\n\nQUESTIONS:\n1. Please further explain the relationship between the confidence and uncertainty maps. Please visualize if possible.\n\n2. Please further explain the global motion bases.\n\n3. Please add visualizations if possible to help clarify how the optimization progresses over time (from a novel view).\n\nLIMITATIONS:\nThe limitations section could definitely be more extensive. What are some failure cases? Do any of those occur in the DyCheck-VOS dataset?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}