TrackingWorld: World-centric Monocular 3D Tracking of Almost All Pixels
Authors: Jiahao Lu
Weitao Xiong
Jiacheng Deng
Peng Li
Tianyu Huang
Zhiyang Dou
Cheng Lin
Sai-Kit Yeung
Yuan Liu
Venue: NeurIPS 2025 poster ()
Keywords: deep learning; computer vision; 3D point tracking
Primary area: applications

Abstract: Monocular 3D tracking aims to capture the long-term motion of pixels in 3D space from a single monocular video and has witnessed rapid progress in recent years. However, we argue that the existing monocular 3D tracking methods still fall short in separating the camera motion from foreground dynamic motion and cannot densely track newly emerging dynamic subjects in the videos. To address these two limitations, we propose TrackingWorld, a novel pipeline for dense 3D tracking of almost all pixels within a world-centric 3D coordinate system. First, we introduce a tracking upsampler that efficiently lifts the arbitrary sparse 2D tracks into dense 2D tracks. Then, to generalize the current tracking methods to newly emerging objects, we apply the upsampler to all frames and reduce the redundancy of 2D tracks by eliminating the tracks in overlapped regions. Finally, we present an efficient optimization-based framework to back-project dense 2D tracks into world-centric 3D trajectories by estimating the camera poses and the 3D coordinates of these 2D tracks. Extensive evaluations on both synthetic and real-world datasets demonstrate that our system achieves accurate and dense 3D tracking in a world-centric coordinate frame.

Tracking World: World-centric Monocular 3D Tracking of Almost All Pixels
Jiahao Lu
Weitao Xiong
Jiacheng Deng?
Peng Li
Tianyu Huang?
Zhiyang Dou"
Cheng Lin
Sai-Kit Yeung'
Yuan Liu't
HKUST USTC
CUHK
4HKU
XMU
MUUST
https://github. www. HO.
Abstract
Monocular 3D tracking aims capture the long-term motion of pixels in 3D space from a single monocular video and has witnessed rapid progress in recent years. However, we argue that the existing monocular 3D tracking methods still fall short in separating the camera motion from foreground dynamic motion and cannot densely track newly emerging dynamic subjects in the videos. To address these two limitations, we propose Tracking World, a novel pipeline for dense 3D tracking of almost all pixels within world-centric 3D coordinate system. First, we introduce a tracking upsampler that efficiently lifts the arbitrary sparse 2D tracks into dense 2D tracks. Then, to generalize the current tracking methods to newly emerging objects, we apply the upsampler to all frames and reduce the redundancy of 2D tracks by eliminating the tracks in overlapped regions. Finally, present efficient optimization-batt based framework to back- pock-project dense 2D tracks into world-centric 3D trajectories by estimating the camera poses and the 3D coordinates of these 2D tracks. Extensive evaluations on both synthetic and real-world datasets demonstrate that our system achieves accurate and dense 3D tracking in world-centric coordinate frame.
Introduction
Estimating long-term motion in dynamic videos remains persistent challenge in computer vision T 2 4]. Fine-grained motion tracking is crucial for understanding object dynamics, modeling camera motion, and facilitating the generation of temporally and geometrically consistent videos [5 6
16] has emerged as an active 1 which employs transformers to iteratively update 2D tracks and has driven progress in 2D motion analysis. This development also motivates many recent works for 3D tracking. Early 3D tracking works like OmniMotion 2 [8] adopt optimization-aann based approaches to estimate 3D motion, while subsequent feedforward methods such SpatialTracker 3 and DELTA 4 leverage extracted features directly estimate the 31) tracking in a feedforward manner without per-sequence optimization. These 3DD tracking methods demonstrate substantial potential for downstream applications, including detailed 3D motion analysis and high-fidelity novel view synthesis, highlighting the growing importance of monocular 3D tracking critical research frontier.
In recent years, dense 2D pixel tracking g 10
13
14
research topic, with notable advancements such CoTrackers 17
Upon analyzing all existing 3D tracking methods, we observe that these existing methods still suffer from two noticeable shortcomings. First, these methods 4 B cannot distinguish the camera
Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Tracking World
Monocular video
Reconstructed camera poses and dense tracking
Tracking World estimates world-centric dense tracking results from monocular videos. Our model can accurately estimate camera poses and achieve disentangled 3D track modeling of static and dynamic components, not just limited to one foreground dynamic object. We only visualize subset of foreground dynamic point trajectories and apply fading color background static points.
Figure 1:
motion and the dynamic object motion. All these methods assume a static camera and just model the 3D flow within the camera coordinate system. However, many downstream tasks like motion analysis or novel-view-synthesis require distinguishing camera motion from the dynamic object motion. Moreover, some recent works [19] also show that explicitly considering camera poses in motion estimation improves the 3D tracking quality. Only some very recent works (20] 21 22] try to estimate the 3D tracks in the world-centric coordinate system and enable distinguishing camera motions from dynamic object motions. Estimating camera motion is still challenging for monocular video containing dynamic objects because only static scenes provide cues for camera pose estimation.
The second shortcoming is that existing methods are mostly limited to tracking sparse pixels in the first frame of the video and cannot track all pixels in all frames (e.g., new objects emerging in the intermediate frames). Tracking all pixels brings a huge computational complexity to all tracking methods. Recent works like DELTA [ propose to upsample the sparse tracking points with neural networks to produce dense 3D tracks. However, DELTA is still limited to tracking the first frame of the video, and how estimate the dense 3D tracks for all pixels of all frames still remains an unexplored problem.
In this paper, propose TrackingWorld, 3D tracking method that enables dense 3D tracking of almost all pixels of all frames from a monocular video within a world-centric coordinate system. "almost all" means we filter some noisy and outlier tracks ensure robustness and accuracy. Specifi- cally, BaockingWorld takes a monocular video and the monocular estimation from foundation models as input, including sparse tracks [4] I1 depth maps (23] and coarse foreground dynamic masks (24. Then, TrackingWort World produces high quality dense 3D tracks for almost all pixels of the monocular video and the camera poses for every frame. Tracking World addresses the above shortcomings with the following strategies.
First, to enable the dense tracking of almost all pixels, utilize the track upsampler of DELTA 4 and track every frame iteratively. We find that the tracking upsampler module of DELTA [4] is applicable to arbitrary 2D tracks, which are utilized by TrackingWorld to upsample the input sparse 2D tracks to dense 2D tracks. Then, we not only track the pixels of the first frame but also repeat all subsequent frames. To reduce computational complexity, we observe that many regions of subsequent frames have already been seen in the first or previous frames. Therefore, we delete the redundant tracks corresponding to these overlapping regions.
Second, to accurately separate the camera motion from the dynamic object motion, we estimate the 3D tracks and the camera poses from the upsampled dense 2D tracks and the input estimated depth maps. A key challenge lies in the inaccuracy of the estimated dynamic masks, which often fail to capture dynamic background objects. This limitation leads to suboptimal bundle adjustment interfered by dynamic background objects, ultimately compromising the accuracy of both camera pose estimation and object motion tracking. Thus, we treat all points in the initial static regions as potentially dynamic but impose an cestaaticasspossiiii which effectively helps rule out the dynamic background points for an accurate camera pose estimation. Finally, we utilize the estimated camera poses along with the depth maps to convert all the 2D tracks into 3D tracks in the world coordinate.
constraint for the camera pose estimation,
To comprehensively evaluate whether our proposed method can effectively achieve dense 3D tracking of almost all pixels across all frames within a world-centric coordinate system, we conduct evaluations

---

from multiple perspectives: 1. Camera pose estimation accuracy; 2. Depth accuracy of the dense 3D tracks; 3. Sparse 3D tracking performance; 4. Accuracy of the dense 2D tracking results. Our empirical analysis demonstrates that the proposed method yields superior performance across all metries, confirming its effectiveness in establishing accurate and consistent 3D tracks over time.
2 Related Work
2.1 2D Point Tracking
The task of tracking arbitrary points 88 10
14 13 16 across video frames is first
introduced by PIPs 8. which leverages deep learning to tackle point tracking based on optical flow. Built upon RAFT
decoder iteratively refine tracking results. TAP-Vid [9] further improves the problem formulation, introducing three
25 J. PIPs computes inter-frame correlation maps and
standardized benchmarks along with TAP-Net, dedicated model for point tracking. TAPIR (IO] advances performance by combining a matching stage with a refinement stage, enhancing tracking accuracy. CoTrackers [17 | observe that strong correlations exist across different point trajectories, and exploit this insight by training on unrolled sequences over long videos, which significantly improve long-tern tracking performance. Drawing inspiration from DETR [26]. TAPTR [12] proposes an end-to-end transformer- based architecture, where each point is represented as query token in the decoder, enabling direct modeling of point dynamics. LocoTrack [T]] extends traditional 2D correlation features to 4D correlation volumes and introduces a lightweight correlation encoder, achieving better efficiency while preserving accuracy.
2.2 3D Point Tracking
While previous works have primarily focused on 2D point tracking, recent research has increasingly focused on 3D point tracking (2) [8] 4 27 28 29 30 31
22 20 21]. Early 3D tracking methods, such as OmniMotion [2]. adopt optimization based approaches to estimate 3D motion. Subsequent work like OmniTrackFast [18] aims to reduce the optimization time and enhance robustness. More recently, increasing attention has shifted toward feedforward
methods. For example, Spatial- Tracker (3) represents points in (u, V, coordinate system, combining image-plane coordinates with depth information. It incorporates depth priors and uses a triplane representation to enable effective 3D tracking. Building upon this idea, DELTA 4 also adopts the UVD coordinate system, but takes a different approach by decoupling appearance and depth correlations. DELTA introduces a coarse-to- to-fine trajectory estimation strategy, allowing for efficient dense tracking across the entire frame rather than being limited to sparse set of locations. In contrast to the aforementioned methods that focus on UVD (2.5D) representations, several concurrent works have recently explored 3D tracking in world-centric coordinate system. St4RTrack (20) adopts DUSt3R (32 like framework to establish pairwise correspondences, but this approach may suffer from drift during long-term tracking. TAPIP3D (21 primarily focuses on sparse tracking and is inherently unable to recover camera motion. comparison, our method introduces comprehensive pipeline for dense 3D tracking that can robustly capture newly emerging objects within world-centric coordinate system.
2.3 4D Reconstruction
4D reconstruction object motion within scene. The problem of non-rigid structure from motion is highly ill-posed. To overcome this limitation, variety of approaches have been proposed. RobustCVD 33
33 34 35
36
19
37
38
39
40 24 aims recover both camera motion and
refines depth
estimation using 3D geometric constraints, while CasualS AM 34 finetunes a depth network guided by predicted motion masks. MegaSaM (35) integrates monocular depth priors and motion probability maps into differentiable SLAM paradigm. Inspired by DUSt3R [32), several data-driven methods such as MonST3R (37]. Align3R (38) and Cut3r 40 adopt 3D point cloud representations to enable full 4D reconstruction. In addition, Uni4D 24 multi-stage optimization framework, leverages multiple pretrained models to improve reconstruction in dynamic scenes. Its core contribution lies in the use of foundation models to achieve effective separation of static and dynamic elements within the scene. Our method also adopts an optimization-bast based framework to decouple camera motion and object motion. However, unlike prior works that primarily focus on 4D reconstruction, our approach targets a bigher-level task-dense tracking every wixcl-which enables fine-grained correspondence estimation across time. By focusing on dense pixel-level tracking, our method
3

---

Preprocessing
Lifting 2D Tracks to World-centric 3D Tracks
Ltatic
Cllp
Mask Edimatior
Edinstto
Intra Clip
Pose Estimation
Sjarse Tracks
Depth Mags
Dynanmie Masks
Clip Extimation
Clip b+
Dynamie Background Refinement
Dense 2D Tracking for Every Pixel
New Tracks
DELTA
Intra Clip
Pose Estimation
Tracking
Point
Tracking Every
Lifting
Features
Frame
:
pecmmper
Traclls
Initial Pase Estimation
Dynamic Object Tracking
Figure Overview. Given video sequence, Tracking World first generates dense tracking results that are capable of capturing newly emerging objects in the scene. These 2D trajectories are then fed space. Specifically,
into an optimization-baser framework to transform them into a world-centric we begin by estimating the initial camera poses for each frame at the clip level. We then perform dynamic background refinement to exclude potentially dynamic regions and refine the camera poses. Based on the optimized poses, we finally reconstruct the trajectories of all dynamic regions.
provides more detailed and temporally consistent understanding of dynamic scenes, making it well-suited for applications such as motion analysis, scene understanding, and video editing 6 77.
3 Method
3.1
Overview
Given video consisting of T frames [I, Hx 1,...,,; T). the goal of Tracking World to estimate the corresponding dense 3D trajectories (3I) tracks) (T, E RRMx33 | [= 1,...,77 .T of almost all pixels, where M, denotes the number of tracked points timestep along with the camera poses { E 1x34 = 1, ...,, T). Our proposed Tracking World framework, illustrated in main components: first, generating dense 2D tracking results that are capable of following nearly every object in the scene; second, back-projecting these dense 2D tracking results into world-centric 3D space.
Fig. [ achieves this through
Preprocessing with vision foundation models. For the monocular video, we first preprocess it with a 2D tracking model, a foreground dynamic mask estimation module, and a monocular depth estimation module to get set of 2D tracks, dynamic masks, and depth maps for all frames. For the 2D tracking model, we choose the CoTrackerV3 [ or the 2D tracking part of DELTA [4) For the dynamic mask estimation method,
follow Uni4D (24] to apply VLM [4] and Grounding- SAM [42 43 to segment out foreground dynamic objects. Alternatively, we could also choose the SegmentA AaanmentAnMMotionn Motion [44] get dynamic masks. For the depth estimation, choose UniDepth (2] Note that all these predictions are not required to be accurate, and we may also adopt other foundation models for this purpose.
3.2 Dense 2D Tracking for Every Pixel
In this section, our target is to achieve dense 2D tracking of almost any pixels in the video. We achieve this through two modules: First, we lift the input sparse 2D tracks for a frame to dense tracks; Second, repeat tracking every frame and eliminate the overlapped redundant 2D tracks.

---

Sparse to dense tracks. Given the sparse 2D tracks P sparse E ( ")xXX2 for specific frame, this module aims to lift the sparse 2D tracks to dense 2D tracks P donse E RX means the downsampled factor. We achieve this by utilizing the upsampler module of DELTA 4.. The upsampler module takes the sparse tracks P sparse and features defined on the sparse 2D tracks sparse E R x W. xTxC, where C' is the feature dimension, as inputs and predicts weight matrix W E R x (HHW)) Then, the upsampled dense 2D tracks are
P Peuss = W sparse:
(I)
where W actually only correlates a dense track P dense with its neighboring 2D tracks in P sparse. We find that this upsampler module is not only compatible with DELTA's 2D tracks but also generalizes arbitrary 2D tracks, so we adopt it here to upsample the arbitrary input sparse 2D tracks into dense 2D tracks for a specific frame.
Tracking every frame. Based on the above upsampler. we further enable tracking of almost all pixels of all frames. To achieve this, we conduct 2D tracking and the sparse-to tarse-to-dense upsampling on all frames in the video. However, this leads to a large redundancy on the tracking points because most regions are already seen in the previous frames, while only a few regions are new. To avoid wasting computation on these redundant 2D tracks the subsequent computation, if pixel resides near the tracking trajectory of arbitrary visible previous 2D tracks, then we discard the pixel. More details can found in A.6 of the supplementary material.
3.3 Lifting 2D Tracks World-centric 3D Tracks
In this module, we will estimate the camera poses of all frames and lift the dense 2D tracks estimated by the previous section to 3D tracks in the world-centric coordinate system. We achieve this through the following three steps: First, we utilize the input estimated coarse dynamic masks and estimate the camera poses using only the coarse static regions. However, the dynamic masks are usually not accurate enough, and some dynamic objects in the background still remain. Second, we utilize an as-static-as- asosssibll constraint to further improve the camera pose estimation and find out the dynamic objects in the background. Finally, we transform all 2D tracks within the dynamic regions into 3D tracks in the world-centric coordinate system.
Initial camera pose estimation. In this step, we want to estimate per-frame camera poses { E SE(3) from the 2D tracks on static regions and the estimated depth maps. We first utilize the input dynamic foreground masks to select 2D tracks P static E NemN on these static regions. Then, for each static 2D track, we unproject its location at timestep ( into the 3D space using the monocular depth map D static E Ne xT x1, The resulting 3D points are subsequently reprojected into the image plane at timestep 12 using the camera poses. Then, we define the projection loss to optimize the camera poses
Nusius
T
T Σ f2
ppoj
(P Ptatiic 41), D (i, 41)) P sstatic (i, 12)
where (-) means project with the camera pose on timestep and P sataii (i, IR2 means the position of the i-th static 2D track on timestep f, D statii t) means the depth value for the i-th static point on time step and Ninierss denotes the number of static 2D tracks whose projection errors fall within the threshold т.
To further improve the computational efficiency for camera pose estimation, first divide the entire video into C clips and estimate camera poses within each clip in parallel. After estimating camera poses within each clip, estimate the pose between clips to merge the camera poses together.
Dynamic background refinement. The foreground dynamic object masks are usually not accurate enough, some dynamic objects in the background still exist in the assumed "static" regions and pre- vent us from accurately estimating camera poses. Thus, we further refine the camera pose estimation by treating these static regions dynamic and introducing
aastattc-as-pssiiball constraint.
Specifically, each static 2D track corresponds to unique 3D point in the world-centrie coordinate system, denoted as Tstatic E F. Nuii x3. We initialize T Tstatic by back-projecting the static 2D tracks
5

---

using the depth estimated by UniDepth and the camera poses obtained from the previous stage: Initial camera pose estimation. Notably, for each 2D track RT*2, we only back-project the visible timesteps and take the average of the resulting 3D points. To better model the potentially dynamic regions that are not accurately segmented, we introduce an additional object motion term static Ra xT'x3 which captures residual object motions over time. With this term, the time- dependent world-centric static tracking becomes
T Ta (2, (i, T stmiic + O static (tatic(2, (2,t). t),
(3)
where TV (i, RD means the 3D coordinate of the i-th static point at timestep t and O suati (i, the corresponding dimensional offset. We then jointly optimize the camera poses Ty and the static 3D coordinates T static using bundle adjustment loss:
Navi T
Lba
Σ
( staric (t, P P (tatic f))2
t=1
where P (s t) is the observed 2D projection of the i-th track at timestep t. In addition to the bundle adjustment loss, also compute depth consistency loss Lae to enforce the consistency between the projected depth maps from static and the estimated monocular depth maps, as introduced in the supplementary material. To ensure that residual motion remains minimal for genuinely static regions, we regularize the offset O static with an as as-static-ass as- aaspossible constraint
Lusap Σ
O (i, 1)
(5)
where we minimize the LI norms of offsets to make all points as static as possible. This Casap enables the accurate camera estimation and also models the dynamics of background objects.
Dynamic object tracking. In this step, our target to lift the 2D tracks of dynamic regions to 3D tracks. We also include the dynamic background points with staie (i, >)l 2 e here as the dynamic 3D tracks. For these dynamic 3D tracks, we directly represent their 3D coordinates by T dyynamic E R Mosuni x7x33 Similar to the 3D static tracks, we initialize the dynamic 3D tracks by back brojeprooe them using the depths predicted by UniDepth and the camera poses refined in the second stage. Based Taymaiic, we also compute the projection loss in Eq. 4 the depth consistency loss Lax as-rigid-as-pont posssibee loss Lump [45 24], and a temporal smoothness loss Lis (24). All the details of these loss terms are included in the supplementary material. The final outputs the dynamic 3D tracks T Taynamic. static 3D tracks T Tamee and the camera poses Te.
Discussion.
The tracking module TrackingWorld differs from previous 3D tracking methods, DELTA and SpatialTracker 33 by explicitly estimating the camera poses, which enables the estimation of 3D tracks in the world-centric coordinate system. The explicit separation between camera motion and object motion also improves the quality of 3D tracking because of the better decomposition, demonstrated by experimental results in Tab. In comparison with the cxisting dynamic video camera pose estimation methods, like Uni4D (24, we do not just assume a single dy- namie foreground object but also model the background object motion in the camera pose estimation for a better performance. Instead of simply discarding these dynamic background objects, we also track their 3D points in the world-centric coordinate system, enabling tracking almost all pixels.
4 Experiment
4.1 Implementation details
All experiments are conducted on an RTX 4090 GPU. We use CoTrackerV3 [I and DELTA 4 to obtain dense tracking results, and adopt UniDepth (24] the depth prior. The entire framework takes ~0 minutes to produce dense world-centric 3D tracking results for a 30-frame video. All baseline methods are run on the datasets using their official implementations and default hyperparameters. More details about hyperparameters can be found in the supplementary materials.
4.2 Quantitative comparisons
To demonstrate the capability of our method in dense 3D tracking within world-centric coordinate system, we evaluate the following performance: 1. Camera pose estimation accuracy; Depth accuracy of dense 3D tracks; 3. Sparse 3D tracking performance; 4. Dense 2D tracking performance.

---

4.2.1 Camera pose estimation results
Benchmarks and metrics. We evaluate camera pose estimation performance on three dynamic datasets: Sintel [46] Bonn 47 and TUM-D 48 For all three datasets, we adopt the same settings as MonST3R [37] Following 49 50 51 we report three ATE J. (Absolute Trajectory Error), RTE & (Relative Translation Error), and RRE & (Relative Rotation Error). ATE measures the deviation between estimated and ground truth trajectories after alignment. RTE and RRE evaluate the average local translation and rotation errors over consecutive pose pairs, respectively.
Comparison with existing methods. Tab. ! presents the quantitative comparison between our method and existing approaches. To recover the camera pose, we first obtain dense tracking results, followed by optimization process that refines the camera pose and world-centric dense tracking. As shown in the table, regardless of whether the dense tracking is derived from DELTA [4] or CoTrackerV3 D1 our method consistently achieves more accurate pose estimation than previous approaches across all three datasets.
Sintel RTEL
Bonn RTE,
TUM-D ATE
Category
Method
ATE
RREI
ATE +
RREJ
RTEL
RRE
DROID-SLAM;
52
0.175
0.084
1,912
Pose only
DPVO [51
0.115
0.072
1.975
1
COLMAP 53
0.559
0.325
7.302
/
0.076
0.059
7.689
Robust-CVD 33
0.360
0.154
3,443
0.153
0.026
3.528
DUSt3R (32
0.601
0.214 11.43
0.046
0.014
1.836
0.083
0.017
3.567
Joint depth
MonST3R 337
0.111
0.044 0.780
0.029
0.007
0.612
0.063
0.009
1.217
& pose
Align3R (38 (Depth Pro 54
0.128 0.042 0.432
0.023
0.007
0.620
0.027
0.018
0.446
Uni4D* 24
0.116 0.046 0.603
0.017
0.006
0.561
0.039
0.007 0.434
Ours (Co'TrackeV33 11
0.103 0.039 0.439
0.016
0.005
0.561
0.014
0.005 0.338
Ours (DELTA 44)
0.088 0.035 0.410
0.016
0.005
0.564
0.016
0.005 0.333
Table I: Camera pose estimation results. We evaluate our model on three datasets: Sintel, Bonn, and TUM-D. Best results are highlighted. means using ground truth camera intrinsics as input. *
means reproduced by 2D tracks from DELTA, the same as "Ours(DELTA)".
4.2.2 Depth accuracy of the dense 3D tracks
Benchmarks and metrics. Since our method does not aim to optimize 2D tracking accuracy directly, but rather focuses how transform 2D tracking into dense world-centric tracking, evaluates the accuracy of the camera-centric depth for each tracked point. Specifically, we compare the predicted depth with the ground-truth depth only for tracked points that lie within the image bounds. As multiple tracked points may track the same pixel, we retain the one with the smaller depth value for evaluation, assuming more likely corresponds the visible surface. Similar the camera pose benchmark, we evaluate on the same datasets and under identical settings: Sintel, Bonn, and TUM-D. Following prior works [55 37), we align the estimated dense tracking depth with the ground truth using single scale and shift before computing the evaluation metrics. We primarily report two metries: Abs Rel I (absolute relative error) and the percentage of inlier points with 1.25
Comparison with existing methods. Tab. reports the results dense tracking depth estimation. Thanks optimization-basee bundle adjustment, which enforces strong 3D geometric consistency, the estimated tracking depth is significantly improved across all datasets.
Sintel
Bonn
TUM-D
Method
Depth Prior
Abs Rel + 5 < 1.25 1 Abs Rel 4 5 < 1.25 t Abs Rel 4 8 < 1.25
DELTA
ZoeDepth [66
0.814
46.1
0.168
88.5
0,239
70.5
DELTA
4
Depth Pro 54
0.813
50.7
0.160
90.6
0.222
78,4
DELTA 4
Unidepth 23
0.636
63.1
0.153
90.5
0.178
85.6
Ours (CoTraceerV3 I1)
Unidepth
0.219
73.1
0.054
97.2
0.089
91.5
Ours (DELTA 4
Unidepth 23
0.218
73.3
0.058
97.3
0.084
92.3
Table 2: Depth accuracy of the dense 3D tracks. Best results are highlighted.
7

---

Calegory Method
APDso
ADT
AJ
PStudio APDaa 1OA
Method
EPE]
CVO-Clean
loUt
EPE4
CVO-Final
lol?
CoTrackorV3
U-Uii
13.6
21.3
88.5
14.1
22.8
87.7
RAFFT 25
2.48
57.6
2.63
56.7
Food.
SpatiaFracker
14.3
22.3
91.5
13.8
23.7
79.5
Colracker
DELTA
15.1
Spatial Tracker 3
1.84
68.5
1.88
68.
Omnn/TaccFang
18
8.6
18.2
63.9
6.4
12.2
81.8
DOT-3D 59
1.33
79.0
1.38
78.8
Optin.
Ours (CoTracker33
22.5
31.5
88.5
14.2
24.0
87.7
DELTA
1.14
78.9
1.39
78.2
(DELTA
CoTrackerV3 0 Up
1.24
80.9
1.35
80.6
Table 3: Sparse 3D tracking results. "Feed." means feedforward methods while "Optim" means optimization-b ased method.
Table 4: Long-range optical flow results.
4.2.3 Sparse 3D tracking results
Benchmarks and Metrics. To evaluate the performance of 3D sparse tracks, we conduct experi- ments two datasets, ADT [57] with moving cameras, and PStudio (58) with static cameras. For each dataset, the video subsets for evaluation are selected at fixed intervals: for ADT, we sample one video every 100 videos, and for PStudio,
video every 20 videos. The sparse 3D tracking result evaluated in camera coordinates. As for evaluation metrics, we adopt Average Jaccard (AJ), which jointly evaluates the accuracy of both spatial position and occlusion estimation, serving prehensive indicator tracking quality; ADD3D (< ōavg) which measures the average percentage of tracked points whose errors fall within given threshold 6, reflecting geometric accuracy, Occlusion Accuracy (OA) which evaluates the precision of occlusion state prediction across frames.
Comparison with existing methods. Since our method primarily focuses dense tracking, maintain the optimization of dense tracking results even when evaluating sparse tracking performance. To this end, sample evaluation points from the optimized dense tracking set. As shown in Tab. B. our method achieves higher 3D geometric consistency in tracking. For scenes with camera motion (ADT), the explicit separation between camera motion and object motion leads to significant improvements in both AJ and APDan. In contrast, for scenes with static cameras (PStudio), the benefits from geometric optimization are relatively limited, resulting in smaller performance gains. is worth noting that OA mainly evaluates the visibility accuracy of tracking points. Since we directly adopt the visibility maps predicted by DELTA/CoTrrccen the OA scores remain consistent with those of DELTA/CoTraccerV3.
4.2.4 Accuracy of dense 2D tracks
Benchmarks and Metrics. We evaluate the dense 2D tracking performance the CVO 60) test set, which consists two subsets: CVO-Clean and CVO-Final, with the latter incorporating motion blur. Each subset contains approximately 500 videos with 7 frames. For evaluation, we adopt the following metries: the end-point error (EPE) between the predicted and ground-truth optical flows for points, and the intersection-over-ee regions in visible masks.
(IoU) between the predicted and ground-truth occluded
Comparison with existing methods. To verify the accuracy of the 2D dense tracks generated by the upsampler module (Up) introduced in Sec. [3.2] we conduct additional long-range optical flow experiments, as shown in Tab. 4 The results demonstrate that the upsampler module generalizes well to other 2D trackers, such as CoTrackerV3, achieving comparable performance with DELTA.
4.3 Qualitative results
Fig. 3 Equalitatively visualizes the world-centric dense tracking results produced by method on the DAVIS [61] dataset. For each video sequence, the second row displays 3D tracking results on temporally spaced keyframes, making the changes in object trajectories more perceptible while avoiding visual clutter. The third row presents continuous 3D tracks across all frames, offering a comprehensive view of motion consistency and trajectory completeness. As discussed in Sec. B.3 by separating dynamic and static elements, we can generate stable tracking results for both the static background and dynamic objects.
4.4 Ablation study
Ablation study on the different components. shown in Tab. we conduct ablation studies to validate our major design choices. Specifically, the different configurations are as follows: 1) without

---

Monocular video
Reconstructed World-centriy Dense Tracking
Figure Qualitative results on DAVIS dataset. Our method can output both reliable camera trajectories and world centric dense tracking. The second row visualizes 3D tracking results on temporally spaced keyframes, while the third row shows complete tracks across continuous frames.
Input
w/o 0 static
Sintel
Setting
ATE RTEU RREJ|Abs Rel
< 1.25
w/o T.E.F
0.171 0.047 0.748
/
w/o pose-init.
0.659 0.153 1.382
0.230
72.4
w/o D.O.T
0.088 0.035 0.410
0.468
73.0
w/o Niniiess
0.089 0.035 0.414
0.220
72.9
w/o Ouniic
0.092 0.036 0.459
0.224
72.6
w/o Ca:
0.093 0.036 0.441
0.234
71.2
Full
0.088 0.035 0.410
0.218
73.3
Otaaiic
GT
Table 5: Ablation study on Sintel dataset.
Figure 4: Effectiveness of Ostatic- Key regions are highlighted in red.
tracking every frame (w/o T.E.F); In this setting, we only track from the first frame, which leads to the loss of many critical cues for pose estimation, thereby resulting in a sigmificant performance drop. 2) without initial camera pose estimation (w/o pose-init.): We observe that under this setting, it becomes difficult to jointly optimize both camera poses and 3D tracks effectively - good initialization of the camera poses is necessary to achieve satisfactory results. 3) without dynamic object tracking (w/o D.O.T): this setting, the depths of dynamic tracks are directly obtained from UniDepth predictions without further refinement. As shown in the table, optimizing dynamic tracks is crucial for achieving better performance. 4) without selecting the inliers whose reprojection errors are within threshold (w/o Niniier): By filtering all static points and optimizing with nearly static points, we can effectively reduce the influence of outlier trajectories and obtain more accurate camera poses. 5) without the object motion term O static (w/o O static:: We do not consider the dynamic objects in the assumed "static" background and directly optimize the camera poses with "static" background. We show the projected background "static" points (in green and red dots) Fig. [ As can see, the "apple" (in the red dots) considered a background static region but actually dynamic. Without modeling dynamic points in the background, the points of the apple are incorrectly projected onto incorrect regions. 6) without the depth consistency loss (w/o Lac): Lae can enforce the consistency between the projected depths and the estimated monocular depths, which helps suppress abnormal depth estimations to some extent.
Ablation on different depth estimation models. We conducted an ablation study using three com- monly used monocular depth estimation models: ZoeDepth (56), Depth Pro [54 and UniDepth (23]. For all experiments, we fixed the tracking component to DELTA [ and evaluated both the camera pose estimation accuracy and the depth accuracy of the dense 3D tracks on the Sintel dataset. As shown in Tab, 6 our method consistently improves over raw depth predictions across all depth models, especially in downstream tasks such as camera pose estimation. This demonstrates that our pipeline robust to different depth estimation backbones.

---

Method
ATE 4
RTE 4
RPE
Abs Rel +
< 1.25 f
ZoeDepth
0.814
1 50.7
Depth Pro
0.813
UniDepth
/
0,636
63.1
Ours (ZoeDepth)
0.093
0.038
0,418
0.236
72.1
Ours (Depth Pro)
0.101
0.036
0,434
0.228
72.6
Ours (UniDepth)
0.088
0.035
0.410
0.218
73.3
Table 6: Ablation study on different depth estimatiin models.
Ablation on dynamic mask segmentators. As shown in Tab. 7 we also evaluate different sources of dynamic mask segmentations and observe comparable performance, further demonstrating the robustness of our pipeline.
Method
ATE
RTE +
RPE +
Abs Rel +
8 < 1.25
Ours + VLM + GroundingSA.M
0.088
0.035
0.410
0.218 0.224
73.3
Ours + Segment Any Motion
0.093
0.041
0.379
73.3
Table 7: Ablation study on different dynamic mask segmentators.
Necessity of the 2D upsampler module. The 2D upsampler is crucial for achieving efficient dense tracking. Directly predicting dense 2D correspondences (e.g.. using CoTrackerV3 ( is computationally expensive and memory-intensive, with no clear accuracy gain. To validate this, compare CoTrackerV3 with and without our upsampler on the CVO-Clean dataset (7-frame sequences). As shown in Tab. 8 the upsampler improves both accuracy (lower EPE, higher IoU) and drastically reduces runtime (approximately 12x speed-up). This supports our design choice.
Method
EPE
IoU
Avg. Time (min)
CoTrackerV3
1.45
76.8
3.00
CoTrackerV3 Up
1.24
80.9
0.25
Table 8: Ablation on the 2D upsampler module.
5
Conclusion
In this paper, we propose TrackingWorld, novel method for dense 3D tracking of almost all pixels of all frames from monocular video within a world-centric coordinate system. The key idea of Tracking World is to explicitly disentangle camera motion from foreground dynamic motion while densely tracking newly emerging objects. We first introduce tracking upsampler to densify sparse 2D tracks and apply it to capture newly emerging objects. Finally, we design an efficient optimization- based framework to lift dense 2D tracks into consistent 3D world-centric trajectories. Extensive evaluations across multiple dimensions demonstrate the effectiveness of our system.
References
[1] Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.11831, 2024.
[2] Qianqian Wang. Yen- Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19795-19806 2023.
10

---

[3] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20406-20417. 2024.
[4] Tuan Due Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin-Ying Lee, and Chaoyang Wang. Delta: Dense efficient long-range 3d tracking for any video. arXiv preprint arXiv:2410.42211.200 2024.
[5] Ryan Burgert, Yuancheng Xu, Wenqi Xian, Oliver Pilarski, Pascal Clausen, Mingming He, Li Ma, Yitong Deng, Lingxiao Li, Mohsen Mousavi, Michael Ryoo, Paul Debevec, and Ning Yu. Go-widh-the-flow: contoll-e video diffusion models using real-time warped noise. In CVPR, 2025.
[6] Zekai Gu, Rui Yan, Jiahao Lu, Peng Li, Zhiyang Dou, Chenyang Si, Zhen Dong, Qifeng Liu, Cheng Lin, Ziwei Liu, et al. Diffusion as shader: 3d-aware video diffiusion for versatile video generation control. arXiv preprint arXiv:2501.05 03847, 2025.
[7] Haonan Qiu, Zhaoxi Chen, Zhouxia Wang, Yingging He, Menghan Xia, and Ziwei Liu. Freetraj: Tuning- free trajectory control in video diffusion models. arXiv preprint arXiv:2406. 16863, 2024.
[8] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. European Conference on Computer Vision, pages 59-75. Springer, 2022.
[9] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. Advances in Neural Information Processing Systems, 35: 1331101601662 2022.
[10] Carl Doersch, Yi Yang. Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10061-10072, 2023.
[11] Scokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Scungryong Kim, and Joon- Young Lec. Local pal- correspondence for point tracking. In European Conference on Computer Vision, pages 306-325. Springer, 2024.
[12] Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers as detection. European Conference on Computer Vision, pages 57-75. Springer, 2024.
[13] Artem Zholus, Carl Doersch, Yi Yang, Skanda Koppula, Viorica Patraucean, Xu Owen Hc, Ignacio Rocco, Mehdi SM Sajjadi, Sarath Chandar, and Ross Goroshin. Tapnext: Tracking any point (tap) as next token prediction. arXiv preprint MXXv:25504005599 2025.
[14] Jinyuan Qu, Hongyang Li, Shilong Liu, Tianhe Ren, Zhaoyang Zeng, and Lei Zhang. Taptrv3: Spatial and temporal context foster robust tracking of any point long video. arXiv preprint arXiv:2411.1867 2024.
[15] Tingyang Zhang, Chen Wang, Zhiyang Dou, Qingzhe Gao, Jiahui Lei, Baoquan Chen, and Lingjic Liu. Pro- tracker: Probabilistic integration for robust and accurate point tracking. arXiv preprint 03X202250.,03220, 2025.
[16] Qiaole Dong and Yanwei Fu. Online dense point tracking with streaming memory. arXiv preprint 2rXiv:2503.06471, 2025.
[17] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In European Conference on Computer Vision, pages 18-35.55 Springer, 2024.
[18] Yunzhou Song, Jiahui Lei, Ziyun Wang. Lingjie Liu, and Kostas Daniilidis. Track everything everywhere fast and robustly. In European Conference on Computer Vision, pages 343-359. Springer, 2024.
[19] Ruijie Zhu, Yanzhe Liang, Hanzhi Chang, Jiacheng Deng, Jiahao Lu, Wenfei Yang. Tianzhu Zhang, and Yongdong Zhang. Motiongs: Exploring explicit motion guidance for deformable 3d gaussian splatting. arXiv preprint arXiv:2410.07707, 2024.
[20] Haiwen Feng. Junyi Zhang, Qianqian Wang. Yufei Ye, Pengcheng Yu, Michael Black, Trevor Darrell, and Angjoo Kanazawa. St4rtrack: Simultaneous 4d reconstruction and tracking in the world. arXiv preprint arXiv:2504. 131522 2025.
11

---

[21] Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry. arXiv preprint urXiv:2504.14717 2025.
[22] Linyi Jin, Richard Tucker, Zhengqi Li. David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos. CVPR. 2025.
[23] Luigi Piccinelli, Yung-Hsy Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10106-10116 16, 2024.
[24] David Yifan Yao, Albert J Zhai, and Shenlong Wang. Uni4d: Unifying visual foundation models for 4d modeling from single video. arXiv preprini M.Xx:2503.21761,, 2025.
[25] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECC ECC 2020: 16th European Conference, Glasgow. UK, August 23-28, 2020, Proceedings, Part II 16. pages 402-419. Springer, 2020.
[26] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[27] Bo Wang, Jian Li, Yang Yu, Li Liu, Zhenping Sun, and Dewen Hu. Scenetracker: Long-term scene flow estimation network. arXiv preprint ar Xiv: 19924, 2024.
[28] Weirong Chen, Ganlin Zhang, Felix Wimbauer, Rui Wang, Nikita Araslanov, Andrea Vedaldi, and Daniel Cremers. Back on track: Bundle adjustment for dynamic scene reconstruction. arXiv preprint 12X1::2504.14516, 2025.
[29] Jenny Seidenschwarz. Qunjie Zhou, Bardienus Duisterhof, Deva Ramanan, and Laura -aixe Dynomo: Online point tracking by dynamic online monocular gaussian reconstruction. arXiv preprint arXiv:2409.02104 2024.
[30] Scokju Cho, Jiahui Huang. Seungryong Kim, and Young Lec. Scurat: From moving points to depth. arXiv preprint arXiv:2504. /4687, 2025.
[31] Yoni Kasten, Wuyue Lu,, and Haggai Maron. Fast encoder- based 3d from casual vidcos via point track processing. arXiv preprint arXiv:2404 2024.
[32] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20697-20700 20709, 2024.
[33] Johannes Kopf, Xuejian Rong, and Jia-Bin Bin Huang. Robust consistent vidco depth estimation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1611-1621, 2021.
[34] Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In European Conference on Computer Vision, pages 20-37. Springer, 2022.
[35] Zhengqi L.,, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. Megasam: Accurate, fast, and robust structure and motion from casual dynamic videos. arXiv preprint arXiv:2412.04463, 2024.
[36] Jiahao Lu, Jiacheng Deng. Ruijie Zhu, Yanzhe Liang, Wenfei Yang, Tianzhu Zhang, and Xu Zhou. Dn-4dgs: Denoised deformable network with temporal -patial aggregation for dynamic scene rendering. arXiv preprint Xixv:22400.1 36007 2024.
[37] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Doqing Sun, and Ming-Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint arXiv:2410.03825, 2024. 2024.
[38] Jiahao Lu, Tianyu Huang, Peng Li, Zhiyang Dou, Cheng Lin, Zhiming Cui, Zhen Dong, Sai-Kit Yeung, Wenping Wang, and Yuan Liu. Align3r: Aligned monocular depth estimation for dynamic videos. arXiv preprint 4rXiv:2412.03073 2024.
[39] Yukang Cao, Jiahao Lu, Zhisheng Huang. Zhuowen Shen, Chengfeng Zhao, Fangzhou Hong, Zhaoxi Chen, Xin Li, Wenping Wang, Yuan Liu, et al. Reconstructing 4d spatial intelligence: A survey, arXiv preprint arXiv:2507.1105, 2025.
12

---

[40] Qianqian Wang, Yifei Zhang, Aleksander Holynski, Alexei A Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. arXiv preprini urXiv:2501.1238% 2025.
[41] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, Igee Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 2303:2303.08774, 2023.
[42] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026, 2023.
[43] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38-55. Springer, 2024.
[44] Nan Huang. Wenzhao Zheng. Chenfeng Xu, Kurt Keutzer, Shanghang Zhang, Angjoo Kanazawa, and Qianqian Wang. Segment any motion in videos. arXiv preprint arXiv:2503.2223 2025.
[45] Qiangian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa, Shape of motion: 4d reconstruction from a single video. arXiv preprint ar arXiv:2407.1370 13764, 2024.
[46] Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. naturalistic open souree movic for optical flow evaluation. In Computer Vision ECCV 2012: 12th European Conference on Computer Vision, Florence, Ital, October 7-13, 2012, Proceedings, Part VI 12, pages 611-625. Springer, 2012.
[47] Emanuele Palazzolo, Jens Behley, Philipp Lottes. Philippe Giguere, and Cyrill Stachniss. Refusion:
3d reconstruction in dynamic euvironments for rgb-d cameras exploiting residuals. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Syxtems (IROS), pages 7855-7862. IEEE, 2019.
[48] Jürgen Sturm, Nikolas Engelhard. Felix Endres, Wolfram Burgard, and Daniel Cremers. A benchmark for the evaluation of rgb-d slam systems. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 573-580. IEFF, 2012.
[49] Weirong Chen, Le Chen, Rui Wang, and Mare Pollefeys. Leap-vo: Long-term effective any point tracking for visual odometry. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 19844-19853, 2024.
[50] Wang Zhao, Shaohui Liu. Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras the wild. In European Conference on Computer Vision. pages 523-542. Springer, 2022.
[51] Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. Advances in Neural Information Processing Systems, 36, 2024.
[52] Zachary Teed and Jia Deng. Droid Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558- 16569, 2021.
[53] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure- frotmmmmooo revisited. In Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[54] Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. arXiv preprint arXiv:2410.22073 2024.
[55] Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open- world videos. arXiv preprint arXiv:2409.00 022.4009.0009,0000 2024.
[56] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
[57] Xiaging Pan, Nicholas Charron, Yongqian Yang, Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: new benchmark dataset for egocentric 3d machine perception. In Proceedings of the IEEE/CVF International Conference on Compuder Vision, pages 20133-20144 2023.
[58] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In Proceedings of the IEEE international conference on computer vision, pages 2015.
13

---

[59] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187- 19197, 2024.
[60] Guangyang Wu, Xiaohong Liu, Kunming Luo, Xi Liu, Qingging Zheng, Shuaicheng Liu, Xinyang Jiang, Guangtao Zhai, Wenyi Wang. Aceflow: Backward accumulation for long-range optical flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12119-12128 12128, 2023.
[61] Federico Perazzi, Jordi Tont-T Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Homung. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately reflect the contributions and scope in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer the supplemental material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Sec. Joutines the structure, while See. Sec.Adee delves into the specific experimental settings. For additional details, please refer to the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[No]
Justification:
The data are public datasets and we commit to open-sourcing our code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com / ssiniiiiccypyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provide the specific experimental settings in Sec. More details can be acquired in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification:
We compare methods under the same setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide them Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our work solely intended for academic research purposes.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We properly credit the related assets and explicitly mention the license and terms of use. Please refer to the supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21


Reviews:
[{"review_id": "eY1e6VT4ar", "reviewer": "Reviewer_tFqV", "summary": "This paper proposes a dense 3d tracking method for monocular vidoes, named TrackingWorld. Given a video consisting of T frames, TrackingWorld will output 3D trajectories which are defined in world-centric 3D space for almost all pixels. Firstly, TrackingWorld adopt a preprocess procedure which output 2D sparse tracking results, dynamic masks and depth maps for all frames. Then, a upsampler module is used to output dense 2D tracks. After obtaining dense 2D tracks, points in static regions are used to estimate the initial camera pose and a dynamic background refinement module is used for accurate camera estimation. Finally, camera pose and depth are used to obtain 3D tracks of dynamic regions with depth consistency, as-rigid-as-possible and temporal smoothness constraints. The authors conduct experiments on different benchmarks including camera pose estimation, depth estimation, sparse 3d tracking and dense 2d tracking tasks. The proposed methods achieve superior performance compared to other SOTA methods.", "questions": "1. Please add ablation study to evaluate the robustness of the proposed method with different depth estimation methods.\n2. Please explain the necessity of the upsampler module.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper proposed a pipeline without training to obtain dense 3d tracking results for a monocular video. The pipeline adopt multiple vison foundation models such as 2d tracking, segmentation, depth estimation and use a optimization-based framework to obtain results. \n2. The authors conduct experiments on multiple benchmarks and show the proposed method can achieve superior performances across all metrics.\n\nWeaknesses\n1. The proposed method relies on depth estimation models for the 2D-to-3D lifting process heavily. The authors should conduct ablation studies to evaluate the pipeline's robustness to different depth estimation models.\n2. What is the necessity of employing an upsampler module for 2D track upsampling? The paper lacks ablation studies to explain.", "was_revised": true, "final_justification": "The author's rebuttal addressed my concerns. I raise my score to accept.", "rebuttal": "1. **Ablation Study on Different Depth Estimation Models.**\n\n    Thank you for the insightful comment. We agree that the robustness of our method to different depth estimation backbones is important. To address this concern, we conducted an ablation study using three commonly used monocular depth estimation models: **ZoeDepth \\[1]**, **Depth Pro \\[2]**, and **UniDepth \\[3]**. For all experiments, we fixed the tracking component to **DELTA \\[4]** and evaluated both the camera pose estimation accuracy and the depth accuracy of the dense 3D tracks on the **Sintel** dataset.\n\n    | Method           | ATE ↓     | RTE ↓     | RPE ↓     | Abs Rel ↓ | δ < 1.25 ↑ |\n    | ---------------- | --------- | --------- | --------- | --------- | ---------- |\n    | ZoeDepth         | —         | —         | —         | 0.814     | 46.1       |\n    | Depth Pro        | —         | —         | —         | 0.813     | 50.7       |\n    | UniDepth         | —         | —         | —         | 0.636     | 63.1       |\n    | Ours (ZoeDepth)  | 0.093     | 0.038     | 0.418     | 0.236     | 72.1       |\n    | Ours (Depth Pro) | 0.101     | 0.036     | 0.434     | 0.228     | 72.6       |\n    | Ours (UniDepth)  | **0.088** | **0.035** | **0.410** | **0.218** | **73.3**   |\n\n    As shown above, our method consistently improves over raw depth predictions across all depth models, especially in downstream tasks such as camera pose estimation. This demonstrates that our pipeline is robust to different depth estimation backbones. We will include this ablation table in the revised submission.\n\n---\n\n2. **Necessity of the 2D upsampler module and missing ablation.**\n\n    Thank you for the question. The 2D upsampler is crucial for achieving **efficient dense tracking**. Directly predicting dense 2D correspondences (e.g., using CoTrackerV3 \\[5]) is computationally expensive and memory-intensive, with no clear accuracy gain. To validate this, we compare CoTrackerV3 with and without our upsampler on the CVO-Clean dataset (7-frame sequences). All experiments are run on an A6000 GPU to avoid OOM issues.\n\n    | Method           | EPE ↓    | IoU ↑    | Avg. Time (min) ↓ |\n    | ---------------- | -------- | -------- | ----------------- |\n    | CoTrackerV3      | 1.45     | 76.8     | 3.00              |\n    | CoTrackerV3 + Up | **1.24** | **80.9** | **0.25**          |\n\n    As shown, the upsampler accuracy (↓EPE, ↑IoU) and drastically reduces runtime (\\~12× speed-up). This supports our design choice. Additional evidence can also be found in Table 2 and Table 3 of the main text of the DELTA paper.\n\n[1] Bhat, S. F., Birkl, R., Wofk, D., et al. *Zoedepth: Zero-shot Transfer by Combining Relative and Metric Depth*. arXiv 2023.\n\n[2] Bochkovskii, A., Delaunoy, A., Germain, H., et al. *Depth Pro: Sharp Monocular Metric Depth in Less Than a Second*. ICLR 2025.\n\n[3] Piccinelli, L., Yang, Y. H., Sakaridis, C., et al. *UniDepth: Universal Monocular Metric Depth Estimation*. CVPR 2024.\n\n[4] Ngo, T. D., Zhuang, P., Gan, C., et al. *DELTA: Dense Efficient Long-Range 3D Tracking for Any Video*. ICLR 2025.\n\n[5] Karaev, N., Makarov, I., Wang, J., et al. *CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos*. ICCV 2025."}, {"review_id": "mcETP1bmT4", "reviewer": "Reviewer_wHpN", "summary": "This paper proposes an optimization-based method to estimate camera poses and 3D dense tracking of dynamic objects in the world-coordinate system from monocular RGB inputs. This is achieved mainly by building on top of several vision foundation models such as UniDepth for depth prior, DELTA for 2D tracks, Uni4D for dynamic object masks. The sparse 2D tracks are lifted to dense 2D tracks using DELTA uplifting module. Then, tracking is performed on every frame and the overlapping tracks are removed. After the final optimization, the method achieves new state-of-the-art performance on several benchmarks, including camera pose estimation, depth accuracy, and 3D tracking performance.", "questions": "* It is mentioned that \"almost all\" means that the method filters some noisy and outlier tracks. However, I haven't found information about how this filtering is done. How much it affects accuracy? Is the threshold selection robust? \n* It is also mentioned (P3L120) that VGGT and other methods enable full 4D reconstruction. However, this is not true as VGGT works only for static scenes. Could you please elaborate?\n* L192 mentions that dynamic object masks are not accurate, and some dynamic objects can still exist in \"static\" regions. Can the error come also from dynamic object masks containing some actually static regions? Can this be fixed by the optimization?", "limitations": "Yes, the limitations are adequately discussed in the supplementary materials.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* This is the first method to estimate camera poses and 3D dynamic point tracking in world-coordinate system from monocular RGB streams. This is an important task, and the proposed solution will benefit many other downstream tasks.\n* A wide variety of tasks are also solved by the proposed method (camera pose, depth, 3D tracking), and new state-of-the-art results are achieved on several datasets.\n* The method is implemented in such a way that it runs in some reasonable time, by means such as camera pose estimation at a clip level.\n* The paper is well-written and seems to be straightforward to reproduce.\n\n\nWeaknesses:\n* St4RTrack is mentioned and it seems to be addressing a similar setup. Since their code is not released yet, would it be possible to follow some of their evaluation protocols to see how the proposed method compares to St4RTrack?\n* Dynamic masks are essentially also optimized by allowing other parts of the scene become dynamic. Why is the accuracy of those refined masks not evaluated?\n* It is mentioned that the monocular priors are not required to be accurate. However, this is not so well evaluated. What happens when there are inaccuracies in the depth predictions? What if other depth predictions are used? L139-147 are confusing because they mention that CoTracker or DELTA can be used (also, GroundingSAM or SegmentAnyMotion can be used, etc). However, this is not ablated. If I understood it correctly, only one type of each monocular prior was tried. It is important to ablate this as well.\n* As mentioned on L284, the performance boost is mainly visible for dynamic camera scenes. When the camera is static, the performance improvement is quite small.", "was_revised": true, "final_justification": "The detailed rebuttal addressed most of my concerns are addressed, but I still do not agree that VGGT can be used for 4D reconstruction since it even outputs a static point cloud. It's also very important to add those additional results in the camera ready. I keep my positive rating.", "rebuttal": "1. **Follow St4RTrack evaluation protocols.**\n\n    Thank you for pointing this out. We agree that comparing agains St4RTrack under similar evaluation protocols is valuable. While the official code and data split for St4RTrack are not publicly available, we made our best effort to follow their described protocol for World Coordinate 3D Point Tracking.\n\n    St4RTrack states:\n\n    > \"Our benchmark comprises four datasets, each containing **50 sequences** of **64 frames**. We then compute the prediction error and measure the percentage of points whose error falls below a threshold $\\delta_{\\mathrm{3D}}$ ( with **$\\delta_{\\mathrm{3D}} \\in \\{0.1\\text{m}, 0.3\\text{m}, 0.5\\text{m}, 1.0\\text{m}\\}$**) over **the first 64 frames**. From these, we **randomly sample 50 sequences** per dataset for evaluation.\"\n\n    Since the official code and exact data split of St4RTrack are not publicly available, we adopted the **TAPVid3D mini-split** (50 videos per dataset officially provided by TAPVid3D) and followed their described protocol by evaluating on the **first 64 frames** of each sequence.\n\n    While we cannot guarantee an identical evaluation setting, and therefore cannot make a strictly fair comparison with St4RTrack, we aimed to provide as meaningful a reference as possible. To this end, we re-implemented the key baselines reported in St4RTrack—**SpaTracker+MonST3R**—and evaluated them using the same **$\\mathrm{APD_{3D}}$** metric, after **global median alignment**, as specified in their paper.\n\n    | Method             | ADT ↑     | PStudio ↑ |\n    | ------------------ | --------- | --------- |\n    | SpaTracker+MonST3R | 54.15     | 63.10\\*   |\n    | **Ours (DELTA)**   | **75.18** | **65.23** |\n    \\* *Note: PStudio does not include camera motion; therefore, we only applied SpaTracker for this evaluation.*\n    \n    As shown, our method outperforms the key reported baseline. We believe this result provides strong evidence for the effectiveness of our approach under comparable conditions, despite the absence of results from St4RTrack.\n\n\n---\n\n2. **The accuracy of dynamic masks.**\n\n    Thank you for the insightful comment. Following your suggestion, we evaluated our method on the **DAVIS2016-Moving** benchmark using the evaluation protocol introduced in **Segment Any Motion** \\[1], which reports region similarity **J** and contour accuracy **F**.\n\n    | Method                      | J ↑      | F ↑      |\n    | --------------------------- | -------- | -------- |\n    | Baseline (VLM + GroundingSAM)   | 87.6     | 88.0     |\n    | Baseline + Ours (Refined Masks) | **91.8** | **91.2** |\n\n    Our refinement strategy improves both J and F scores, demonstrating that the optimized dynamic masks can yield more accurate segmentations.\n\n    \\[1] Huang, Nan, et al. *Segment Any Motion in Videos*. CVPR 2025.\n\n\n---\n\n3. **Ablation on Each Component.**\n\n    Thank you very much for your suggestions. Since the ablations for different trackers (e.g., **CoTrackerV3** and **DELTA**) have already been provided in Tables 1–4 in the main text, we focus here on the following two components:\n\n    1. Different **monocular depth estimation models**\n    2. Different **dynamic mask segmentation methods**\n\n    For all experiments, we fixed the tracking component to **DELTA** and evaluated both the **camera pose estimation accuracy** and the **depth accuracy** of the dense 3D tracks on the **Sintel** dataset.\n\n    ---\n\n    **Ablation on Monocular Depth Estimation Models**\n\n    | Method           | ATE ↓     | RTE ↓     | RPE ↓     | Abs Rel ↓ | δ < 1.25 ↑ |\n    | ---------------- | --------- | --------- | --------- | --------- | ---------- |\n    | ZoeDepth         | —         | —         | —         | 0.814     | 46.1       |\n    | Depth Pro        | —         | —         | —         | 0.813     | 50.7       |\n    | UniDepth         | —         | —         | —         | 0.636     | 63.1       |\n    | Ours (ZoeDepth)  | 0.093     | 0.038     | 0.418     | 0.236     | 72.1       |\n    | Ours (Depth Pro) | 0.101     | 0.036     | 0.434     | 0.228     | 72.6       |\n    | Ours (UniDepth)  | **0.088** | **0.035** | **0.410** | **0.218** | **73.3**   |\n\n    As shown above, our method consistently improves upon the raw depth predictions across all monocular depth estimation models，indicating that our pipeline is robust to the choice of depth backbone—even when the original predictions are less accurate.\n\n    ---\n\n    **Ablation on Dynamic Mask Segmentators**\n\n    | Method                    | ATE ↓     | RTE ↓     | RPE ↓     | Abs Rel ↓ | δ < 1.25 ↑ |\n    | ------------------------- | --------- | --------- | --------- | --------- | ---------- |\n    | Ours + VLM + GroundingSAM | **0.088** | **0.035** | 0.410 | **0.218** | **73.3**   |\n    | Ours + Segment Any Motion | 0.093     | 0.041     | **0.379**     | 0.224     | **73.3**    |\n\n    We also evaluate different sources of dynamic mask segmentations and observe comparable performance, further demonstrating the robustness of our pipeline.\n\n---\n\n4. **When the camera is static, the performance improvement is quite small.**\n\n    Our method is specifically optimized to leverage geometric consistency from stereo cues, which are more effective under dynamic camera motions. In static camera scenes, the benefits are naturally less pronounced.\n---\n5. **Filtering some noisy and outlier tracks.**\n\n\n    The filtering process is performed during the “Tracking every frame” stage (Lines 162–167). Specifically, as stated in the paper:\n\n    > *“To avoid wasting computation on these redundant 2D tracks in the subsequent computation, if a pixel resides near the tracking trajectory of arbitrary visible previous 2D tracks, then we discard the pixel.”* (Lines 165–167)\n\n    To implement this, we first compute the **complement set** by discarding pixels that lie in any previously visible 2D track trajectory. However, this operation may introduce **isolated pixels**, which are typically not meaningful for downstream scene understanding. To mitigate this, we construct **connected components** from the newly selected 2D tracking points and apply a size threshold $\\tau$ to remove small components. Only connected regions with more than $\\tau$ pixels are retained. This ensures that the preserved 2D tracks are **geometrically meaningful** and more likely to correspond to **coherent object parts**.\n\n    We found that this filtering procedure consistently improves accuracy, as most filtered points are either outliers or redundantly close to already tracked regions. Moreover, the threshold $\\tau$ is robust across different scenes: we use a fixed value of $\\tau = 40$ in all experiments without additional tuning.\n\n    **Ablation on $\\tau$ on Sintel and Bonn datasets:**\n    | Setting      | **Sintel** |           |           | **Bonn**  |           |           |\n    | ------------ | ---------- | --------- | --------- | --------- | --------- | --------- |\n    |              | ATE ↓      | RTE ↓     | RPE ↓     | ATE ↓     | RTE ↓     | RPE ↓     |\n    | w/o $\\tau$ | 0.105      | 0.038     | 0.442     | 0.018     | 0.007     | 0.601     |\n    | w/ $\\tau$  | **0.088**  | **0.035** | **0.410** | **0.016** | **0.005** | **0.564** |\n\n    The results show that the filtering mechanism leads to consistent improvements across different datasets.\n\n---\n6. **Clarification on VGGT’s Capability for 4D Reconstruction in Dynamic Scenes.**\n\n    Thank you for pointing this out. While VGGT was originally designed for static scenes, we found that it was trained on a variety of datasets, including those with dynamic content such as PointOdyssey and Virtual KITTI, as mentioned in their paper. Furthermore, recent works like Monst3r and Align3r have demonstrated that the 3R framework can be fine-tuned directly on dynamic datasets to achieve competitive dynamic reconstruction performance. In our own experiments, we also observed that VGGT is able to reconstruct dynamic scenes reasonably well in some cases. Based on this evidence, we therefore used the term \"full 4D reconstruction\" in a broad sense to indicate the potential capability of such methods to handle dynamic content. That said, we agree that VGGT is not explicitly designed for dynamic scenes, and its performance on such data may vary depending on scene complexity and motion types.\n---\n7. **Dynamic object masks containing some actually static regions.**\n\n    Dynamic object masks may indeed contain some regions that are actually static. However, this has minimal impact on the accuracy of the camera pose estimation because our method primarily relies on the consistency of truly static regions to solve for precise camera poses. Therefore, including a few static areas within the dynamic masks does not degrade the camera pose results.\n\n    Furthermore, for 3D tracking, we perform an additional optimization step focused on the dynamic regions. This optimization  can leverage the consistency constraints of 2D tracks to effectively reconstruct misclassified static regions, ensuring that the final 3D tracking accuracy is preserved."}, {"review_id": "ncmBzGDKOA", "reviewer": "Reviewer_iMow", "summary": "This paper address the problem of extracting dense 3D point tracking from monocular videos in a world-centric 3D coordinate.  It use pretrained visual foundation models to extract sparse (later upsampled to dense) 2D pixel tracks( CoTracker or DELTA for), monocular depth(Unidepth) and dynamics masks (VLM followed by Grounding-Sam). Then it performs stages of optimization to minimize 2D projection loss and some regularization loss as ARAP to get the camera pose, and dense 3D tracks. The author report evaluation metrics like depth accuracy,3D point tracking accuracy  and optical flow accuracy, showing state-of-the-art results compared with both feedforward methods and optimization-based methods.", "questions": "Which part is more fragile. This paper proposed an multi-stage optimization that chains the predictions of multiple foundation model together. It would be great to analyze common failure cases, e.g. which part of the stage is less reliable under some cases. I could imagein the part of dynamic mask, since it's generated with VLM and grounded SAM, it might miss some objects with semantic feature of static objects but are moving in the video. \n\n\nNovel view visualization will be better.  In the supp, I found videos of lifted 3D dynamic pointclouds at a static viewpoint. It would be very beneficial to include pointclouds visualizations of novel views, since novel views give better visualization of the 3D geometry.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThis paper tackles a very important problem, 3D dynamic tracking in a world-coordinates, and deliveres good results, especially quantitative results.  The writting is also in general clear. \n\n\n\nWeakness:\n\n1. I think the inherent weakness is about the complexity of the pipeline, making it hard to understand the current bottleneck and failure mode. I encourage the author providing more insight to it (mentioned in the question section as well)\n2. Another inherent weakness is that the proposed method is not a end-to-end feedforward model. It's an adhoc optimization methods, which makes it hard to scale (if we have more data, the performance of the optimization based methods might grow slower than end-to-end methods)", "was_revised": true, "final_justification": "I read the author's rebuttal, and I am satisfied with the author's analysis on the potential failure mode in his multi-stage algorithm and the promise to add novel view visualizations of the reconstruction. I raise my score to 4: weak accept.", "rebuttal": "1. **More insight to the method.**\n\n    Our proposed method follows a multi-stage pipeline consisting of four key components:\n\n    1. **All-frame Dense 2D Tracking**\n    2. **Initial Camera Pose Estimation** using monocular depth and static 2D tracks\n    3. **Joint Refinement** of camera poses and static 3D tracks\n    4. **Estimation of Dynamic 3D Tracks**\n\n    Throughout this process, we rely on off-the-shelf 2D tracking models to generate dense 2D tracks, which serve as the foundation of our pipeline. Monocular depth provides critical geometric priors that help prevent the optimization from getting trapped in poor local minima. We have validated the robustness of both components—monocular depth and 2D tracking—under commonly encountered conditions, as discussed in our response to Reviewer wHpN (Q3).\n\n    We also leverage dynamic masks to distinguish between static and dynamic regions. However, due to the imperfect nature of existing dynamic segmentation techniques, we design specific mechanisms to mitigate their limitations:\n\n    1. **Static masks may include dynamic objects.**\n    To address this, we introduce an *as-static-as-possible* constraint, which encourages only truly static regions to be used during pose optimization. This design is evaluated in our ablation study (see Figure 4 in the main text and our response to Reviewer wHpN, Q2).\n\n    2. **Dynamic masks may include static objects.**\n    As discussed in our response to Reviewer wHpN (Q7), this does not significantly impact the accuracy of camera pose estimation. Our method primarily relies on the consistency of truly static regions to recover camera poses, so a small number of misclassified static areas in dynamic masks has minimal effect. Furthermore, during the dynamic 3D tracking stage, we apply an additional optimization step on the dynamic regions. This step enforces 2D track consistency, which can correct some of the earlier misclassifications and ensures accurate 3D tracking.\n\n    In summary, our pipeline demonstrates good robustness to variations in monocular depth, 2D tracks, and dynamic masks. However, it may fail under the following two challenging scenarios:\n    1. **When dynamic objects dominate the camera’s field of view**, camera pose estimation becomes unreliable. If the entire scene is dynamic, even a human observer would struggle to distinguish motion, making reliable pose recovery infeasible.\n\n    2. **When the scene lacks texture**, the quality of 2D tracks becomes poor. Current state-of-the-art 2D tracking models still struggle to generalize to textureless environments, which remains a critical direction for future research.\n\n\n---\n2. **Justification of Optimization-based Framework.**\n\n    We appreciate the reviewer’s comment and agree that end2end methods are more attractive and promising due to their effiency. However, we believe such methods offer distinct advantages, particularly for the task of *world-centric dense 3D tracking*, where obtaining large-scale annotated data in real-world scenarios is extremely difficult.\n\n    Unlike end-to-end approaches that often require substantial supervision—such as accurate 3D annotations and ground-truth camera poses—our method decomposes the problem into several more tractable and annotation-efficient sub-tasks, including monocular depth estimation, dynamic region segmentation, and 2D tracking. These individual tasks have demonstrated strong robustness with recent advancements. Our proposed optimization framework then integrates them in a coherent and geometrically consistent manner, enabling accurate and scalable world-centric dense 3D tracking without relying on fully annotated datasets.\n\n    Furthermore, we emphasize that end-to-end and optimization-based methods are not mutually exclusive but rather complementary. For instance, St4RTrack leverages large-scale synthetic datasets for training, while still incorporating optimization steps to improve generalization to real-world data. Similarly, VGGT benefits from applying bundle adjustment, resulting in improved reconstruction accuracy (see Table 1 in VGGT). These examples highlight the practical utility of optimization-based techniques—especially when combined with end-to-end models—to enhance performance in complex and unconstrained real-world settings.\n---\n3. **Novel view visualization.**\n\n    Thank you for your suggestions. We will update the videos in the revised version to include visualizations from more diverse viewpoints and also provide interactive point cloud visualizations for clearer result interpretation. However, due to this year’s policy, we are unable to include these updated results during the rebuttal phase. We apologize for the inconvenience."}, {"review_id": "W9CuVPJcz3", "reviewer": "Reviewer_yGCi", "summary": "This paper proposes TrackingWorld, a novel pipeline for dense world-coordinate 3D tracking that estimates world-centric trajectories of almost all pixels from monocular video. The method introduces two key contributions:\n* All-frame Dense 2D Tracking: The approach leverages sparse 2D tracking [1] and the upsampling layer from DELTA [2] to enable tracking of all pixels across all frames, including newly emerging objects, while eliminating redundant tracks.\n\n* Optimization-based Camera Pose Estimation and World-centric 3D Tracking: A three-phase optimization strategy is proposed:\n  * (1) Initial camera pose estimation using monocular depths and static 2D tracking.\n  * (2) Joint refinement of camera poses and static 3D tracks.\n  * (3) Estimation of dynamic 3D tracks.\n\nThe proposed method is evaluated on several benchmarks, including Sintel, Bonn, TUM-D, ADT, and PStudio, and demonstrates improvements over prior methods in pose estimation accuracy, 3D tracking quality, and depth consistency.", "questions": "**Questions**:\n* L188–190: How are the camera poses from individual clips combined into a unified, globally consistent pose sequence for the entire video?\n* The current approach uses dense 2D tracks and monocular depth for the optimization step. Could the method alternatively leverage dense 3D tracks (e.g., from DELTA) to directly estimate camera poses and refine world-centric 3D trajectories?\n\n**Conclusion**:\n\nAlthough the paper addresses an interesting and valuable task, with a well-motivated pipeline and strong performance across several benchmarks, I find the technical novelty to be somewhat limited. The use of decomposed static and dynamic components for camera pose estimation, dynamic refinement, tracking-based correspondence for pose optimization, and the associated loss functions are largely based on existing methods. However, I appreciate the thoughtful integration of these components into a unified pipeline that jointly estimates camera poses and world-centric 3D trajectories, which adds practical value to the field. However, there are several issues in the evaluation protocol (see weaknesses above) that need to be addressed for the results to be fully convincing. I would be happy to increase my score if the authors can address these concerns during the rebuttal.\n\n[1] Karaev, Nikita, et al. \"CoTracker3: Simpler and better point tracking by pseudo-labelling real videos.\" arXiv preprint arXiv:2410.11831 (2024).\n\n[2] Ngo, Tuan Duc, et al. \"DELTA: Dense Efficient Long-range 3D Tracking for any video.\" arXiv preprint arXiv:2410.24211 (2024).\n\n[3] Li, Zhengqi, et al. \"MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[4] Yao, David Yifan, Albert J. Zhai, and Shenlong Wang. \"Uni4D: Unifying Visual Foundation Models for 4D Modeling from a Single Video.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n* The paper is clearly written and easy to follow, with rigorous and well-presented mathematical formulations.\n* Each design choice is well-motivated and thoroughly justified, supported by comprehensive ablation studies that validate their contributions.\n* The proposed method outperforms prior approaches in dynamic pose estimation, demonstrating its effectiveness in challenging tracking scenarios.\n\n**Weaknesses**:\n* Monocular Depth Inconsistency: The method relies on monocular depth estimators, which are known to produce temporally inconsistent depth, even on static objects. Although the proposed residual term for static 3D tracks and the dynamic 3D track optimization can partially address this issue, I am concerned that depth inconsistency may still significantly affect the final results. Prior works like Mega-SAM and Uni-4D explicitly refine video depth for temporal consistency. It would strengthen this paper if the authors also refined monocular depth across time and potentially provided consistent video depth as an additional output.\n* Questionable Depth Evaluation Setup: The evaluation protocol for depth accuracy of dense 3D tracks (Sec. 4.2.2) is unclear. The reported depth accuracy depends on both: (1) The quality of the 2D tracking to correctly localize the corresponding pixel at the target frame; (2) the accuracy of the 3D tracking depth itself. Without guaranteeing perfect 2D correspondence, the comparison between the ground-truth depth at the target pixel and the estimated depth can be misleading. The current evaluation does not disentangle these two sources of error, which undermines the validity of this metric.\n* Sparse 3D Tracking Evaluation Details: In Sec. 4.2.3, the authors mention randomly selecting video subsets for evaluation but do not explain the selection criteria or whether this ensures a fair comparison to prior work. A more standardized setup, such as using the TAPVid3D mini-split (50 videos per dataset), would strengthen the evaluation. Additionally, it is unclear whether the sparse 3D tracking is evaluated in world or camera coordinates—this should be specified.\n* Lack of Runtime Baseline: The reported runtime of 10 minutes per 30-frame video is considerably slower than methods like Mega-SAM or Uni-4D, though those methods do not output world-centric trajectories. A simple baseline would be to combine camera poses and consistent depths from Mega-SAM [3] or Uni-4D [4] with DELTA’s camera-centric 3D tracking (or dense 2d tracking from Cotracker3 + DELTA’s upsampler) and project them into the world frame. It would be helpful to see how this simpler and potentially faster alternative compares in both accuracy and runtime.", "was_revised": true, "final_justification": "Given the strong performance and the importance of the task, I find this work impactful despite the pipeline being built on relatively known components. The integration of these components into a pipeline that achieves impressive results is noteworthy. The authors' rebuttal addressed most of my concerns. I therefore raise my score to borderline accept.", "rebuttal": "1.  **1) Monocular Depth Inconsistency.**\n\n    Our pipeline is robust to inconsistencies in monocular depth. We tested our pipeline using different monocular depth estimation models and observed consistently depth improvement. This is because our framework jointly optimizes camera poses and enforces consistency across depth predictions from different frames through 2D tracking associations. For detailed experimental results, please refer to our response to Reviewer tFqV, Q1.\n\n    **2) Can the Method Produce Consistent Video Depth?**\n    \n    Yes. Since our method aims to achieve dense 3D tracking for nearly all pixels, it can naturally generate consistent video depth outputs. This can be achieved through a straightforward post-processing step (interpolation-based propagation).\n\n    To address your concern, we now describe how we can derive temporally consistent **video depth** from our dense 3D tracking results.\n\n    Assume we have obtained $M$ dense 3D tracking points across the video. At each frame $t \\in \\{1, 2, \\dots, T\\}$, we denote the set of 3D tracking points that are visible as $\\mathcal{M}_t $, and their camera-centric depths as $D_t$. Let $D_t^{\\text{raw}}$ be the raw monocular depth prediction for frame $t$.\n\n    We first align the raw depth with the optimized 3D tracking depth using a **scale ratio**. Unlike some global alignment strategies that compute one scale per frame, we compute **a per-point local scale** between the tracking depths and the raw depth.\n\n    For each tracking point $\\mathbf{M}_t^i \\in \\mathcal{M}_t$, its corresponding raw depth value is $D_t^{\\text{raw}}(\\mathbf{p}_t^i)$, where $\\mathbf{p}_t^i$ is the projected 2D location of $\\mathbf{M}_t^i$ on the image plane. The **scale ratio** at that point is defined as:\n    $$\n    r_t^i = \\frac{D_t(\\mathbf{p}_t^i)}{D_t^{\\text{raw}}(\\mathbf{p}_t^i)}\n    $$\n    Then, for any pixel $\\mathbf{p}_t$ in frame $t$, we estimate its final depth $\\hat{D}_t(\\mathbf{p}_t)$ via **weighted scale propagation** from nearby 3D tracking points.\n\n    Let $\\mathcal{N}_t(\\mathbf{p}_t) \\subset \\mathcal{M}_t$ denote the set of its $k$ nearest neighbors in the image plane. For each neighbor $\\mathbf{M}_t^j \\in \\mathcal{N}_t(\\mathbf{p}_t)$, we compute a 3D Euclidean distance between $\\mathbf{p}_t$ and $\\mathbf{M}_t^j$, using their raw-depth-lifted 3D coordinates. That is:\n\n    $$\n    \\mathbf{P}_t = D_t^{\\text{raw}}(\\mathbf{p}_t) \\cdot K^{-1} \\tilde{\\mathbf{p}}_t, \\quad\n    \\mathbf{Q}_t^j = D_t^{\\text{raw}}(\\mathbf{p}_t^j) \\cdot K^{-1} \\tilde{\\mathbf{p}}_t^j\n    $$\n\n    $$\n    d_t^j = \\| \\mathbf{P}_t - \\mathbf{Q}_t^j \\|_2\n    $$\n\n    where $K$ is the camera intrinsic matrix and $\\tilde{\\mathbf{p}}$ denotes the homogeneous coordinate of pixel $\\mathbf{p}$.\n\n    We define a weight $w_t^j$ for each neighbor as the **inverse distance weight**:\n\n    $$\n    w_t^j = \\frac{1}{d_t^j + \\epsilon}\n    $$\n\n    where $\\epsilon$ is a small constant to avoid division by zero. The weights are then normalized:\n\n    $$\n    \\tilde{w}_ t^j = \\frac{w_t^j}{\\sum_{j=1}^{k} w_t^j}\n    $$\n\n    Using these weights, we compute the interpolated scale ratio $r_{\\mathbf{p}_t}$ at pixel $\\mathbf{p}_t$:\n\n    $$\n    r_ {\\mathbf{p}_ t} = \\sum_ {j=1}^{k} w_ t^j \\cdot r_t^j\n    $$\n\n    Finally, we compute the aligned depth at pixel \\$\\mathbf{p}\\$ as:\n\n    $$\n    \\hat{D}_ t(\\mathbf{p}_ t) = r_ {\\mathbf{p}_ t} \\cdot D_ t^{\\text{raw}}(\\mathbf{p}_ t)\n    $$\n\n    Through this approach, we effectively propagate the accurate scale information from our depth-consistent 3D tracks to the entire image, yielding a temporally consistent and spatially coherent video depth. Quantitative results for our video depth can be found in the following table.\n\n    | Category|Method|Sintel|Bonn|TUM D|\n    |-|-|-|-|-|\n    |||Abs Rel ↓ / δ<1.25 ↑|Abs Rel ↓ / δ<1.25 ↑|Abs Rel ↓ / δ<1.25 ↑|\n    ||Depth Anything V2| 0.348 / 59.2 | 0.106 / 92.1 | 0.211 / 78.0 |\n    | Single-frame | Depth Pro| 0.418 / 55.9 | 0.068 / **97.4** | 0.126 / 89.3 |\n    || ZoeDepth| 0.467 / 47.3 | 0.087 / 94.8 | 0.176 / 74.5 |\n    | Video depth| ChronoDepth| 0.687 / 48.6 | 0.100 / 91.1 | - / - |\n    || DepthCrafter| 0.292 / 69.7 | 0.075 / 97.1 | - / -|\n    || Align3R (Depth Pro)| 0.263 / 64.1 | 0.058 / 97.1 | 0.111 / 88.9 |\n    | Joint video depth & pose| Uni4D| 0.236 / 70.6 | **0.057** / 97.2 | 0.091 / 91.5 |\n    || **Ours (DELTA)**| **0.222 / 72.6** | 0.058 / 97.3 | **0.086 / 92.3** |\n\n---\n2. **Questionable Depth Evaluation Setup.**\n\n    We agree that our evaluation metric inherently couples depth and correspondence. The reason is that since our goal is to simultaneously reconstruct in 3D and track pixels in the reconstructed 3D space, such a metric evaluates both aspects about 3D reconstructed points and the correspondences across frames.\n\n    To further show the evaluation of the video depth only, we have provided a quantitative comparison of video depth outputs in our response to Q1.\n\n    This evaluation metric only shows the correspondences across frames and depth map. To demonstrate the quality of both camera-centric and world-centric 3D tracking, we also report the sparse camera-centric 3D tracking results in Table 3 of the main text, and the sparse world-centric 3D tracking results in our response to Reviewer wHpN’s first question. Unfortunately, due to the lack of publicly available real-world datasets with ground-truth dense world-centric 3D tracks, we are currently unable to report dense-level evaluation in this setting but only provide the results on world-centric sparse 3D tracks.\n        \n\n---\n3. **Sparse 3D Tracking Evaluation Details.**\n\n    Thank you for your suggestions. The video subsets for evaluation are selected at fixed intervals: for ADT, we sample one video every 100 videos, and for PStudio, one video every 20 videos. All baseline results reported in the table were reproduced by us using the official repositories under consistent settings. To ensure a more standardized evaluation, we now conduct experiments on the TAPVid3D mini-split.\n\n    | Method| AJ ↑| $\\mathrm{APD_{3D}}$ ↑ | OA ↑| AJ ↑| $\\mathrm{APD_{3D}}$ ↑ | OA ↑    |\n    |-|-|-|-|-|-|-|\n    | CoTrackerV3 + Uni| 13.9 | 21.0| 88.7    | 14.3    | 23.1| **87.5**|\n    | DELTA| 15.1| 23.2| 89.8    | 14.9    | 24.8| 75.4 |\n    | Ours (CoTrackerV3)   | 22.7    | 31.2| 88.8    | 14.0   | 24.2| **87.5**|\n    | Ours (DELTA) | **23.1**| **32.5**| 90.3    | **15.3**| **25.3**| 75.9    |\n\n    Additionally, the sparse 3D tracking results shown in the above table and in Table 3 of the main text are evaluated in camera coordinates. For world coordinates, we have also provided supplementary results in our response to Reviewer wHpN’s first question.\n\n\n\n---\n4. **Lack of Runtime Baseline.**\n\n    Thank you for your suggestions. Due to time constraints, we have currently evaluated one of the baselines you recommended: using camera poses and consistent depths from Uni-4D combined with dense camera-centric 3D tracking from DELTA.\n\n    Since our method and the baseline share the same dense camera-centric 3D tracking pipeline from DELTA, we focus the comparison on the optimization stage and the associated runtime.\n\n    Uni-4D estimates poses in a streaming fashion, i.e., it estimates the pose between frame 1 and 2, then frame 2 and 3, and so forth. Consequently, the total pose estimation time grows linearly with the number of video frames. In contrast, our method estimates poses in a clip-to-global parallel manner, significantly reducing computation time. \n    \n    Regarding 3D tracking accuracy, our method achieves better performance by more effectively distinguishing between static and dynamic regions using dynamic masks, and by jointly optimizing both pose and point trajectories. This leads to higher-quality reconstructions compared to the baseline.\n\n    Next, we report detailed results on camera pose estimation (Sintel: frames 30–50) and world-coordinate 3D point tracking (ADT: first 64 frames). The ADT evaluation follows the same experimental setup as described in our response to Reviewer wHpN’s first question. All experiments are conducted on the same CPU and GPU device to ensure fair comparison.\n\n    | Setting | Sintel ATE ↓ | Sintel RTE ↓ | Sintel RPE ↓ | Sintel Avg. Time (min) ↓ | ADT $\\mathrm{APD_{3D}}$ ↑ | ADT Avg. Time (min) ↓ |\n    |-|-|-|-|-|-|-|\n    | Baseline (Uni-4D + DELTA) | 0.118| 0.048| 0.610| 19| 68.95|28 | \n    | Ours (DELTA)| **0.087** | **0.036**  | **0.406**| **15**| **75.18** | **20** |\n---\n5. **Global pose from clips.**\n\n    We estimate the camera poses for the entire video sequence in a globally consistent manner using the formulation in Equation (2) of the main text. Specifically, we compute the relative poses between adjacent frames within each clip in parallel. Additionally, we estimate the relative pose between the last frame of the i-th clip and the first frame of the (i+1)-th clip. These inter-clip constraints allow us to link all frames together, forming a unified and globally consistent pose trajectory across the entire video.\n---\n6. **Alternatively leverage dense 3D tracks.**\n\n    Yes. And we have explored this alternative approach. Specifically, we replaced the dense 2D tracks and monocular depth with dense 3D tracks (obtained from DELTA) for direct camera pose estimation. However, we observed no significant improvement in accuracy, and thus we did not include the results in the main text. For completeness, we report the comparison below:\n\n    | Setting|Sintel ATE ↓| Sintel RTE ↓| Sintel RPE ↓|\n    |-|-|-|-|\n    | Dense 2D tracks + monocular depth | **0.087** | 0.036 | 0.406 |\n    | Dense 3D tracks| 0.092     | **0.035**     | **0.403**     |"}]

Author rebuttal:

Meta-review: The paper presents TrackingWorld, a optimization‑based pipeline that jointly estimates camera poses and dense world‑centric 3D trajectories from a monocular video. The authors convincingly integrate several high‑performing vision foundation models (depth, 2D tracks, dynamic masks) and demonstrate consistent, state‑of‑the‑art results on multiple benchmarks. 
The authors have successfully addressed most of the reviewers' concerns (depth‑model robustness, necessity of the upsampler, filtering of noisy tracks, etc.), resulting in two "accept" and two "borderline accept" ratings for the paper. Given the solid technical contribution, comprehensive evaluation, and satisfactory responses to all major reviewer questions, we recommend accept the paper.
Decision: Accept (poster)