Title: TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep learning; computer vision; 3D point tracking
Abstract: Monocular 3D tracking aims to capture the long-term motion of pixels in 3D space from a single monocular video and has witnessed rapid progress in recent years. However, we argue that the existing monocular 3D tracking methods still fall short in separating the camera motion from foreground dynamic motion and cannot densely track newly emerging dynamic subjects in the videos. To address these two limitations, we propose TrackingWorld, a novel pipeline for dense 3D tracking of almost all pixels within a world-centric 3D coordinate system. First, we introduce a tracking upsampler that efficiently lifts the arbitrary sparse 2D tracks into dense 2D tracks. Then, to generalize the current tracking methods to newly emerging objects, we apply the upsampler to all frames and reduce the redundancy of 2D tracks by eliminating the tracks in overlapped regions. Finally, we present an efficient optimization-based framework to back-project dense 2D tracks into world-centric 3D trajectories by estimating the camera poses and the 3D coordinates of these 2D tracks. Extensive evaluations on both synthetic and real-world datasets demonstrate that our system achieves accurate and dense 3D tracking in a world-centric coordinate frame.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes TrackingWorld, a novel pipeline for dense world-coordinate 3D tracking that estimates world-centric trajectories of almost all pixels from monocular video. The method introduces two key contributions:\n* All-frame Dense 2D Tracking: The approach leverages sparse 2D tracking [1] and the upsampling layer from DELTA [2] to enable tracking of all pixels across all frames, including newly emerging objects, while eliminating redundant tracks.\n\n* Optimization-based Camera Pose Estimation and World-centric 3D Tracking: A three-phase optimization strategy is proposed:\n  * (1) Initial camera pose estimation using monocular depths and static 2D tracking.\n  * (2) Joint refinement of camera poses and static 3D tracks.\n  * (3) Estimation of dynamic 3D tracks.\n\nThe proposed method is evaluated on several benchmarks, including Sintel, Bonn, TUM-D, ADT, and PStudio, and demonstrates improvements over prior methods in pose estimation accuracy, 3D tracking quality, and depth consistency.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n* The paper is clearly written and easy to follow, with rigorous and well-presented mathematical formulations.\n* Each design choice is well-motivated and thoroughly justified, supported by comprehensive ablation studies that validate their contributions.\n* The proposed method outperforms prior approaches in dynamic pose estimation, demonstrating its effectiveness in challenging tracking scenarios.\n\n**Weaknesses**:\n* Monocular Depth Inconsistency: The method relies on monocular depth estimators, which are known to produce temporally inconsistent depth, even on static objects. Although the proposed residual term for static 3D tracks and the dynamic 3D track optimization can partially address this issue, I am concerned that depth inconsistency may still significantly affect the final results. Prior works like Mega-SAM and Uni-4D explicitly refine video depth for temporal consistency. It would strengthen this paper if the authors also refined monocular depth across time and potentially provided consistent video depth as an additional output.\n* Questionable Depth Evaluation Setup: The evaluation protocol for depth accuracy of dense 3D tracks (Sec. 4.2.2) is unclear. The reported depth accuracy depends on both: (1) The quality of the 2D tracking to correctly localize the corresponding pixel at the target frame; (2) the accuracy of the 3D tracking depth itself. Without guaranteeing perfect 2D correspondence, the comparison between the ground-truth depth at the target pixel and the estimated depth can be misleading. The current evaluation does not disentangle these two sources of error, which undermines the validity of this metric.\n* Sparse 3D Tracking Evaluation Details: In Sec. 4.2.3, the authors mention randomly selecting video subsets for evaluation but do not explain the selection criteria or whether this ensures a fair comparison to prior work. A more standardized setup, such as using the TAPVid3D mini-split (50 videos per dataset), would strengthen the evaluation. Additionally, it is unclear whether the sparse 3D tracking is evaluated in world or camera coordinates—this should be specified.\n* Lack of Runtime Baseline: The reported runtime of 10 minutes per 30-frame video is considerably slower than methods like Mega-SAM or Uni-4D, though those methods do not output world-centric trajectories. A simple baseline would be to combine camera poses and consistent depths from Mega-SAM [3] or Uni-4D [4] with DELTA’s camera-centric 3D tracking (or dense 2d tracking from Cotracker3 + DELTA’s upsampler) and project them into the world frame. It would be helpful to see how this simpler and potentially faster alternative compares in both accuracy and runtime.\n\nQUESTIONS:\n**Questions**:\n* L188–190: How are the camera poses from individual clips combined into a unified, globally consistent pose sequence for the entire video?\n* The current approach uses dense 2D tracks and monocular depth for the optimization step. Could the method alternatively leverage dense 3D tracks (e.g., from DELTA) to directly estimate camera poses and refine world-centric 3D trajectories?\n\n**Conclusion**:\n\nAlthough the paper addresses an interesting and valuable task, with a well-motivated pipeline and strong performance across several benchmarks, I find the technical novelty to be somewhat limited. The use of decomposed static and dynamic components for camera pose estimation, dynamic refinement, tracking-based correspondence for pose optimization, and the associated loss functions are largely based on existing methods. However, I appreciate the thoughtful integration of these components into a unified pipeline that jointly estimates camera poses and world-centric 3D trajectories, which adds practical value to the field. However, there are several issues in the evaluation protocol (see weaknesses above) that need to be addressed for the results to be fully convincing. I would be happy to increase my score if the authors can address these concerns during the rebuttal.\n\n[1] Karaev, Nikita, et al. \"CoTracker3: Simpler and better point tracking by pseudo-labelling real videos.\" arXiv preprint arXiv:2410.11831 (2024).\n\n[2] Ngo, Tuan Duc, et al. \"DELTA: Dense Efficient Long-range 3D Tracking for any video.\" arXiv preprint arXiv:2410.24211 (2024).\n\n[3] Li, Zhengqi, et al. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[4] Yao, David Yifan, Albert J. Zhai, and Shenlong Wang. \"Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper address the problem of extracting dense 3D point tracking from monocular videos in a world-centric 3D coordinate.  It use pretrained visual foundation models to extract sparse (later upsampled to dense) 2D pixel tracks( CoTracker or DELTA for), monocular depth(Unidepth) and dynamics masks (VLM followed by Grounding-Sam). Then it performs stages of optimization to minimize 2D projection loss and some regularization loss as ARAP to get the camera pose, and dense 3D tracks. The author report evaluation metrics like depth accuracy,3D point tracking accuracy  and optical flow accuracy, showing state-of-the-art results compared with both feedforward methods and optimization-based methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nThis paper tackles a very important problem, 3D dynamic tracking in a world-coordinates, and deliveres good results, especially quantitative results.  The writting is also in general clear. \n\n\n\nWeakness:\n\n1. I think the inherent weakness is about the complexity of the pipeline, making it hard to understand the current bottleneck and failure mode. I encourage the author providing more insight to it (mentioned in the question section as well)\n2. Another inherent weakness is that the proposed method is not a end-to-end feedforward model. It's an adhoc optimization methods, which makes it hard to scale (if we have more data, the performance of the optimization based methods might grow slower than end-to-end methods)\n\nQUESTIONS:\nWhich part is more fragile. This paper proposed an multi-stage optimization that chains the predictions of multiple foundation model together. It would be great to analyze common failure cases, e.g. which part of the stage is less reliable under some cases. I could imagein the part of dynamic mask, since it's generated with VLM and grounded SAM, it might miss some objects with semantic feature of static objects but are moving in the video. \n\n\nNovel view visualization will be better.  In the supp, I found videos of lifted 3D dynamic pointclouds at a static viewpoint. It would be very beneficial to include pointclouds visualizations of novel views, since novel views give better visualization of the 3D geometry.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an optimization-based method to estimate camera poses and 3D dense tracking of dynamic objects in the world-coordinate system from monocular RGB inputs. This is achieved mainly by building on top of several vision foundation models such as UniDepth for depth prior, DELTA for 2D tracks, Uni4D for dynamic object masks. The sparse 2D tracks are lifted to dense 2D tracks using DELTA uplifting module. Then, tracking is performed on every frame and the overlapping tracks are removed. After the final optimization, the method achieves new state-of-the-art performance on several benchmarks, including camera pose estimation, depth accuracy, and 3D tracking performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This is the first method to estimate camera poses and 3D dynamic point tracking in world-coordinate system from monocular RGB streams. This is an important task, and the proposed solution will benefit many other downstream tasks.\n* A wide variety of tasks are also solved by the proposed method (camera pose, depth, 3D tracking), and new state-of-the-art results are achieved on several datasets.\n* The method is implemented in such a way that it runs in some reasonable time, by means such as camera pose estimation at a clip level.\n* The paper is well-written and seems to be straightforward to reproduce.\n\n\nWeaknesses:\n* St4RTrack is mentioned and it seems to be addressing a similar setup. Since their code is not released yet, would it be possible to follow some of their evaluation protocols to see how the proposed method compares to St4RTrack?\n* Dynamic masks are essentially also optimized by allowing other parts of the scene become dynamic. Why is the accuracy of those refined masks not evaluated?\n* It is mentioned that the monocular priors are not required to be accurate. However, this is not so well evaluated. What happens when there are inaccuracies in the depth predictions? What if other depth predictions are used? L139-147 are confusing because they mention that CoTracker or DELTA can be used (also, GroundingSAM or SegmentAnyMotion can be used, etc). However, this is not ablated. If I understood it correctly, only one type of each monocular prior was tried. It is important to ablate this as well.\n* As mentioned on L284, the performance boost is mainly visible for dynamic camera scenes. When the camera is static, the performance improvement is quite small.\n\nQUESTIONS:\n* It is mentioned that \"almost all\" means that the method filters some noisy and outlier tracks. However, I haven't found information about how this filtering is done. How much it affects accuracy? Is the threshold selection robust? \n* It is also mentioned (P3L120) that VGGT and other methods enable full 4D reconstruction. However, this is not true as VGGT works only for static scenes. Could you please elaborate?\n* L192 mentions that dynamic object masks are not accurate, and some dynamic objects can still exist in \"static\" regions. Can the error come also from dynamic object masks containing some actually static regions? Can this be fixed by the optimization?\n\nLIMITATIONS:\nYes, the limitations are adequately discussed in the supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a dense 3d tracking method for monocular vidoes, named TrackingWorld. Given a video consisting of T frames, TrackingWorld will output 3D trajectories which are defined in world-centric 3D space for almost all pixels. Firstly, TrackingWorld adopt a preprocess procedure which output 2D sparse tracking results, dynamic masks and depth maps for all frames. Then, a upsampler module is used to output dense 2D tracks. After obtaining dense 2D tracks, points in static regions are used to estimate the initial camera pose and a dynamic background refinement module is used for accurate camera estimation. Finally, camera pose and depth are used to obtain 3D tracks of dynamic regions with depth consistency, as-rigid-as-possible and temporal smoothness constraints. The authors conduct experiments on different benchmarks including camera pose estimation, depth estimation, sparse 3d tracking and dense 2d tracking tasks. The proposed methods achieve superior performance compared to other SOTA methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper proposed a pipeline without training to obtain dense 3d tracking results for a monocular video. The pipeline adopt multiple vison foundation models such as 2d tracking, segmentation, depth estimation and use a optimization-based framework to obtain results. \n2. The authors conduct experiments on multiple benchmarks and show the proposed method can achieve superior performances across all metrics.\n\nWeaknesses\n1. The proposed method relies on depth estimation models for the 2D-to-3D lifting process heavily. The authors should conduct ablation studies to evaluate the pipeline's robustness to different depth estimation models.\n2. What is the necessity of employing an upsampler module for 2D track upsampling? The paper lacks ablation studies to explain.\n\nQUESTIONS:\n1. Please add ablation study to evaluate the robustness of the proposed method with different depth estimation methods.\n2. Please explain the necessity of the upsampler module.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-09"}