Title: Enhancing 3D Reconstruction for Dynamic Scenes
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconstruction

4D Reconstruction
Abstract: In this work, we address the task of 3D reconstruction in dynamic scenes, where object motions frequently degrade the quality of previous 3D pointmap regression methods, such as DUSt3R, that are originally designed for static 3D scene reconstruction. Although these methods provide an elegant and powerful solution in static settings, they struggle in the presence of dynamic motions that disrupt alignment based solely on camera poses. To overcome this, we propose D$^2$USt3R that directly regresses Static-Dynamic Aligned Pointmaps (SDAP) that simultaneiously capture both static and dynamic 3D scene geometry. By explicitly incorporating both spatial and temporal aspects, our approach successfully encapsulates 3D dense correspondence to the proposed pointmaps, enhancing downstream tasks. Extensive experimental evaluations demonstrate that our proposed approach consistently achieves superior 3D reconstruction performance across various datasets featuring complex motions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for robust 3D reconstruction in dynamic scenes. It addresses the limitations of prior pointmap regression methods that struggle with object motion by proposing Static-Dynamic Aligned Pointmaps (SDAP). SDAP simultaneously accounts for both spatial structures and temporal motions, providing a more reliable 3D reconstruction of both static and dynamic regions. The method integrates occlusion and dynamic masks, derived from optical flow, to guide the network and stabilize learning. With a confidence-aware 3D regression loss that combines separate objectives for static and dynamic areas,this paper demonstrates superior 3D reconstruction performance across various complex motion datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrenth\n\n\n1) Distinguishing camera-induced motion from genuine object-specific motion is challenging, as it requires precisely separating the camera’s ego-motion from the independent movements of objects in the scene.\n\n2) It introduces a unique representation called Static-Dynamic Aligned Pointmaps, which can simultaneously account for both static spatial structures and dynamic temporal motions in a unified framework.\n\n3) The paper is clearly written and easy to follow.\n\n\n\nWeakness\n\n\n1)The authors describe their dynamic mask generation as relying primarily on optical flow analysis, comparing the observed optical flow with that induced purely by camera motion. However, this approach does not inherently leverage high-level semantic or shape information. Modern state-of-the-art object tracking methods, such as SAMURAI, can effectively track dynamic objects in videos and could potentially address the challenges of dynamic mask generation; however, the authors did not conduct any study to evaluate this possibility.\n\n\n2)The results in Table 2 show no improvement on the KITTI dataset, highlighting poor generalization. Despite KITTI’s mostly static scenes, the method still degrades, suggesting it fails to adapt when dynamic motion is absent.\n\n\n3) There is a contradiction in the results: while Table 2 shows no improvement on the KITTI dataset, Table 5 reports a significant boost in pointmap alignment accuracy for dynamic objects. This suggests that the new method may actually degrade overall reconstruction quality.\n\n\n4) The camera pose results in Table 4 do not show any improvement over the state of the art, and the authors fail to clearly explain this outcome.\n\n\n5) The state-of-the-art methods considered are insufficient, as they do not include a comparison with 'Shape of Motion: 4D Reconstruction from a Single Video,' which performs reconstruction for long-range 3D tracking\n\nQUESTIONS:\nRefer weaknesses\n\nLIMITATIONS:\nNo. Integrate semantic segmentation models or object detection networks to provide high-level semantic and shape information, can make the dynamic mask generation more robust and accurate, especially for complex or ambiguous motions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper trains a neural network to estimate a point map for each frame from two given video frames that capture dynamic objects. The network learns how to estimate depth for different types of motion. To this end, the static background and the dynamic object regions are separated and are trained with sophisticated objectives that address occlusion and dynamic regions. The qualitative and quantitative comparisons demonstrate its performance across various dynamic scene datasets.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper’s strengths are as follows;\n\n- The performance outperforms other learning-based stereo models, such as Dust3r and Monst3r, for monocular and binocular depth estimation. \n- The occlusion mask and dynamic mask are utilized to train the network without false guidance. For a dynamic region, the objective encourages the network to estimate a point map at the time as its counterpart, shown in Equation 8.\n- Supplemental shows the outperformance of the proposed method against Monst3r.\n\nThere are the following weak points:\n\n- Term X^{1,2} in Equation 8 is used, but there is no explicit definition of it. This makes me confused about understanding the training objectives. Specifically, the first term in Equation 6 and the second term in Equation 8 could conflict with each other if X_1,2 and X_1,1 are from the same estimation. \n- Term X has different meanings at Line 89 and Line 92. It should be consistent throughout the paper context.\n- The optical flow supervision is available for synthetic datasets. This may lead to a domain gap when the trained model is applied to a real-world dataset.\n- The proposed model doesn’t utilize the predicted optical flow for depth estimation. Plus, optical flow and depth estimation are computed independently. This may cause the inconsistency between two different types of predictions: optical flow and 3d points. However, it has not been discussed in the paper.\n- The evaluation doesn't separate the dynamic and the static regions.\n- Figure 7 is too small to see the correspondence matches.\n- There is no video demo that can verify the temporal consistency of the proposed method.\n- The comparison for camera pose estimation is not plausibly discussed.\n\nQUESTIONS:\nI have the following questions:\n\n- Please clarify the notation of a predicted point map X^{n,m} and explain whether there is a conflict between the two objective terms mentioned above.\n\n- In the supplemental, the reconstruction in the background suffers from wavy artifacts and a multiview inconsistency problem. Does the proposed method show degraded performance for the static dataset? Figure 5 also shows that the Dust3r achieves reasonable relative scene scales and a field of view on the static region comparable to Monst3r and the proposed method. It would be great to conduct the evaluation for the static region and the dynamic region individually. \n\n- Do the predicted optical flow and depth agree with each other? \n\n- Does the proposed method produce temporal consistency and multiview consistency? Since the video is not produced, those components cannot be evaluated.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes $D^2UST3R$, an approach to improve upon previous 3D aware pointmap regression approaches such as Dust3r and Monst3r for dynamic scenes. They propose SDAP: Static-Dynamic Aligned Pointmaps, which capture both static and dynamics elements of the 3d geometry. They also propose leveraging an alignment loss that takes into account missing direct 3d correspondences due to occlusion or motion. Finally, they benchmark across a variety of tasks such as camera pose estimation, multi-frame depth, etc.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is written well and easy to follow\n2. Unified modelling of motion and static elements(SDAP) is an interesting approach to handle dynamic scenes\n\n\nWeaknesses\n1. While the method improves upon 3D aware regression, it still seems imperative to compare to methods such as Align3R[1], MegaSAM[2], etc.\n2. Quality of Flow: How robust is the approach towards noisy flow? Currently the method relies on ground truth flow and off-the-shelf state of the art flow estimators.\n3. All five training datasets are synthetic. No real-capture dynamic-scene sequence seems to be used for training so generalizability to real photometrics, motion-blur, sensor noise is unproven.\n\n[1]Lu, Jiahao, et al. \"Align3r: Aligned monocular depth estimation for dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Li, Zhengqi, et al. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nQUESTIONS:\n1. I would like to see comparisons to some more recent methods for dynamic scene reconstruction as stated in the weaknesses.\n2. How does inaccurate flow impact depth or pose estimation, for eg if you were to use a weaker model than the current one your approach is using?\n3. Have you tested on real dynamic-scene datasets with hand-held cameras such as DAVIS or Waymo? How does the model perform in those cases?\n\nLIMITATIONS:\n1. Please address the point about not using real-world dataset in training. Is that by design or some other reason?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an enhanced 3D reconstruction method for dynamic scenes, addressing limitations in existing approaches like DUS3R and MonST3R, which struggle with dynamic object motion due to their reliance on static scene assumptions. The authors introduce Static-Dynamic Aligned Pointmaps (SDAP), a novel representation that integrates camera pose information for static regions and optical flow for dynamic regions to achieve consistent 3D point correspondences across frames. By incorporating motion-aware training objectives, the method improves depth estimation and pointmap alignment in dynamic scenes. The approach is evaluated on datasets like TUM-Dynamics, Sintel, and KITTI, demonstrating superior performance in multi-frame depth estimation and pointmap alignment compared to baselines. Training details, including datasets and hyperparameters, are provided, though pretrained weights are not released due to their size.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The introduction of Static-Dynamic Aligned Pointmaps is a significant contribution, effectively addressing the misalignment issues in dynamic scenes by combining camera pose-based alignment for static regions with optical flow-based warping for dynamic regions. This dual approach is a clever solution to the temporal inconsistency problem.\n- The paper conducts thorough evaluations across multiple datasets (TUM-Dynamics, Sintel, KITTI) with varying temporal strides, providing robust evidence of the method’s effectiveness in both static and dynamic regions. The inclusion of qualitative visualizations (Figures 5-7) strengthens the interpretability of results.\n- By augmenting the feed-forward framework with explicit motion-aware constraints, the method tackles the limitations of per-frame training paradigms in prior work, leading to improved consistency in depth estimation and pointmap reconstruction.\n- The paper provides sufficient details on dataset construction, loss functions, and training procedures (Sections 4.2, 4.3, 4.4, 5.1), facilitating potential reproducibility despite the lack of code release.\n\nWeaknesses\n- The paper provides an \"end-to-end\" solution for dynamic scene reconstruction. However, the core component—Static-Dynamic Aligned Pointmap (SDAP)—relies heavily on external optical flow estimators (RAFT or precomputed ground-truth flow). This design introduces additional computational overhead and potential inconsistencies, and ultimately constrains the upper bound of model performance due to its dependence on the quality of the external flow. Moreover, the paper does not provide an in-depth analysis of the robustness of the reconstruction to flow errors, nor does it evaluate system performance under scenarios with missing or inaccurate optical flow.\n- The authors introduce an occlusion mask to filter out unalignable regions. However, the results lack any quantitative evaluation of reconstruction quality specifically within occluded areas. Given that occlusion is often the most challenging aspect in dynamic scene reconstruction, this omission limits the assessment of the method’s robustness and generalization to boundary conditions.\n- The authors design separate losses for static and dynamic regions (L_static and L_dyn), yet the alignment loss for dynamic regions depends on warping via optical flow indices. In practice, this approach is prone to sparsity and misalignment issues, especially in occluded regions, at depth discontinuities, or near object boundaries. The resulting loss becomes unstable and may introduce noise into the overall reconstruction. Furthermore, this split-loss design fails to consider continuity and smoothness across static-dynamic region boundaries, which can lead to “crack-like” artifacts in the reconstruction.\n- The authors introduce additional heads—Dynamic Mask Head and Optical Flow Head—aiming to support dynamic region recognition and auxiliary supervision. However, these modules are trained independently and are not integrated into the SDAP optimization process in a closed-loop manner. This lack of interaction limits their effectiveness and wastes the potential of mutual reinforcement.\n- The loss function in D2USt3R is formulated purely based on local frame-pair consistency, without incorporating longer-term temporal coherence or dense path modeling across multiple frames. This absence of long-range temporal constraints makes the method prone to inconsistency or drift, particularly in fast motion or non-rigid object scenarios.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}