TAPIP3D: Tracking Any Point in Persistent 3D Geometry
Authors: Bowei Zhang
Lei Ke
Adam W Harley
Katerina Fragkiadaki
Venue: NeurIPS 2025 poster ()
Keywords: 3d point tracking
Primary area: applications
TL;DR: Long-term feed-forward 3D point tracking in persistent 3D point maps.

Abstract: We introduce TAPIP3D,  a novel approach for long-term 3D point tracking in monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized spatio-temporal feature clouds, leveraging depth and camera motion information to lift 2D video features into a 3D world space where camera movement is effectively canceled out. Within this stabilized 3D representation, TAPIP3D iteratively refines multi-frame motion estimates, enabling robust point tracking over long time horizons. To handle the irregular structure of 3D point distributions, we propose a 3D Neighborhood-to-Neighborhood (N2N) attention mechanism—a 3D-aware contextualization strategy that builds informative, spatially coherent feature neighborhoods to support precise trajectory estimation. Our 3D-centric formulation significantly improves performance over existing 3D point tracking methods and even surpasses state-of-the-art 2D pixel trackers in accuracy when reliable depth is available. The model supports inference in both camera-centric (unstabilized) and world-centric (stabilized) coordinates, with experiments showing that compensating for camera motion leads to substantial gains in tracking robustness. By replacing the conventional 2D square correlation windows used in prior 2D and 3D trackers with a spatially grounded 3D attention mechanism, TAPIP3D achieves strong and consistent results across multiple 3D point tracking benchmarks. Our code and trained checkpoints will be public.

TАPIP3D: Tracking Any Point in Persistent 3D Geometry
Bowei Zhang 12-
Lei Ke
Adam W. Harley Peking University
Katerina Fragkiadaki Stanforr University
Carneeiee Mellon University
Abstract
We introduce TAPIP3D, a novel approach for long-term 3D point tracking in monocular RGB and RGB-D videos. TAPIP3D represents videos camera- stabilized spatio-temporal feature clouds, leveraging depth and camera motion information to lift 2D video features into a 3D world space where camera movement is effectively canceled out. Within this stabilized 3D representation, TAPIP3D iteratively refines multi-frame motion estimates, enabling robust point tracking over long time horizons. To handle the irregular structure of 3D point distributions, we propose 3D Neighborhood-to-Netee to-Neighborhood (N2N) attention mechanism 3D- aware contextuaiization strategy that builds informative, spatially coherent feature neighborhoods to support precise trajectory estimation. Our 3D-centric formulation significantly improves performance over existing 3D point tracking methods and even surpasses state-of-the-ad 2D pixel trackers in accuracy when reliable depth is available. The model supports inference in both camera-centric (unstabilized) and world-centric (stabilized) coordinates, with experiments showing that compensating for camera motion leads substantial gains in tracking robustness. By replacing the conventional 2D square correlation windows used in prior 2D and 3D trackers with spatially grounded 3D attention mechanism, TAPIP3D achieves strong and consistent results across multiple 3D point tracking benchmarks. Project Page: tapip3d.githuub.i
1 Introduction
Tracking points over time in video especially through occlusions has become a valuable tool in robotics and action recognition [20 [2]. Fine-grained motion estimation at the particle level offers a unified framework for capturing temporal changes object pose, part articulation, deformable structures, and even granular materials. However, most existing point trackers operate either directly in pixel space 20 12 or in pixel space augmented with depth information [48] 33). Yet, the predominant source of apparent motion in video often camera movement, not object motion. Since real-world dynamics unfold in 3D, tracking points directly 3D spacc-paatee rather than in the image plane-man be more natural and effective.
In this paper, we ask: Can recent advances in camera pose and depth estimation enable more effective 3D point tracking by explicitly representing and tracking points in 3D, while compensating for camera motion?
To this end, we propose TAPIP3D for Tracking Any Point in Persistent 3D Geometry, a 3D point tracking method that represents and iteratively updates multi-frame 3D point trajectories through spatio-temporal attentions in an RGB-D video. Our method represents a video as spatio-temporal 3D feature cloud, Each point in the cloud represents 2D feature vector lifted a corresponding 3D coordinate (X, Y, Z) using sensed or estimated depth. Under known camera motion, we construct
"Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Point Tracking 3D Persistent World Space
UVD
spoce:
XYZ
space:
Camera pose sequence sampled time steps.
Reconstructed 3D point clouds from monocular video.
Trajectory Visualizution
Figure 1: TAPIP3D performs long-term 3D point tracking in a persistent, camera-stabilizee world space of 3D feature clouds, surpassing prior methods [48] 30 that operate in camera-dependent UVD (UV pixels + Depth) spaces. Using depth maps and camera poses provided or estimated via MegaSaM [25], TAPIP3D constructs 3D world space in which camera motion is effectively canceled out. As a result, the 3D trajectories of sampled dynamic points represerereeeetted XYZ world coordinates significantly smoother and more linear than those produced in UVD space.
camera-stabilized spatio-temporal feature clouds by applying extrinsics, as shown in Figure I Our method performs tracking in either the camera world 3D feature space by directly featurizing scene coordinates using 3D attentions. Specifically, it identifies the local 3D neighborhood for each query trajectory coordinate as a query group and replaces traditional 2D CNNs with novel wtwprrrooooott-iigooooooboe
cross-attention design. This mechanism enables the deep feature representation of query point group at given timestep attend to the features of key point group, i.e., its neighboring 3D points at future timesteps. Moreover, within these cross attentions, we incorporate the 3D relative offset between the query trajectory point and the neighboring target point into the attention values, alongside appearance features, thereby enhancing spatial context awareness.
We test TAPIP3D in established 3D point tracking benchmarks of TAPVid3D 22. LSFOdyssey (40), Dynamic Replica [18] and Dex YCB [ for tracking points in 2D and 3D, using depth from ground- truth, sensors. or estimators, in camera and world 3D coordinate frames. We show TAPIP3D outperforms all previous methods in 3D point tracking metries, with an especially large margin when accurate depth is available. While existing methods pursue 3D tracking in camera coordinates (30 48 our experiments show that it better track in a "world" coordinate frame, which can be estimated by recent camera and depth estimation methods such as MegaSaM 25 In addition to showing the impact of coordinate system choices, we ablate the contributions to performance from 3D-centric featurization and quality of depth estimation.
In summary, we present TAPIP3D for 3D point tracking from RGB-D and RGB videos that uses 3D feature clouds for video to estimate 3D point tracks, which demonstrates state-of- of-the-art performance when ground-truthe truth / sensor depth is available and competitive performance with estimated depth [25]. It can exploit recent advances in depth and camera pose estimation [25] 47 and deliver both camera and world-centric 3D tracks. To our knowledge, TAPIP3D is a pioneering 3D tracking method capable of tracking in 3D world-centric space, with camera motion factored out. We will release our code upon acceptance.
2 Related Work
2D and 3D Point Tracking Recent advances in point tracking formulate task as estimation of multi-frame point trajectories, moving beyond traditional pairwise optical flow estimation. Inspired by earlier work on particle video (35), PIPs (12) introduces an iterative refinement approach with dense cost maps. This work along with the Tracking Any Point (TAP) benchmark 7 have catalyzed series

---

of advancements 223 24 including the use of multi-poin context [20] 19, enhanced initialization schemes [ broader correlation contexts 52 and post-hoc densification techniques [22].
Beyond 2D, recent works such as SpatialTracker [48] have extended point tracking to 3D by in-
corporating depth information. SpatialTracker employs Triplane representation [31 where 3DD featured point clouds per frame from pixel (UV) and depth (D) coordinates are projected onto onto three orthogonal planes. Triplane featurization is faster than our N--N attention, but this speedup comes at the cost of decreased performance. The works of DELTA 33 and SceneTracke 40) also operate on a UVD coordinate system to separately compute appearance and depth correlation features, directly extending the 2D method of CoTracker [20]. DELTA [30] propases a coarse- to fine trajectory estimation that manages to estimate dense point tracks across the whole image plane, instead of on a set of sparse locations. In contrast to previous 3D point trackers, TAPIP3D leverages an explicit spatio-temporal 3D feature cloud video representation, instead of a 2.5D one of the image-space and depth, for feature extraction and tracking. By fully utilizing the underlying geometric structure of the scene, our method achieves better point tracking accuracy and consistency. Moreover, our model can estimate trajectories in 3D world space, by canceling inferred camera motion from the scene image pixels during lifting. which helps performance further.
Point Tracking via Reconstruction 3D point tracks can also be extracted from monocular videos through test-time optimization by fitting dynamic radiance fields, such as dynamic NeRFs 33 41
or Gaussians 29 42 36 6 to the observed video frames supervised from reprojection RGB, depth and motion error. These approaches require per-video optimization and are thus computationally expensive. In contrast, TAPIP3D is a learning-based feed-forward approach for 3D point tracking.
Scene Flow Estimation Scene flow estimation extends optical flow into three dimensions by considering pairs of point clouds and estimating the 3D motion between them ||
27 46
31. Recent works incorporate rigid motion priors, explicitly [39] or implicitly 49 or leveraging diffusion models 26). Linking 3D scene flow estimates terminates at pixel occlusions by design, similarly to linking 2D flow estimates. Instead our work focuses on inferring multiframe point trajectories through occlusions.
Bearning-Basee 3D Foundation Models Recent methods in learning based camera motion and depth estimation, such as MoGe 44. DUS(3R 45). MonST3R [50), and MegaSaM [25] focus on
predicting dense 3D reconstructions from single images, image pairs, or videos, bringing the vision of video to 4D translation closer. MegaSaM [25] delivers highly accurate camera motion and depth by combining learning based initialization and updates with second order optimization, extending earlier work of DROID-SLAM [8). None of the methods above addresses the problem of estimating the 3D point motion, which is the focus of our work. TAPIP3D builds upon the progress of recent learning-basee 3D camera motion and depth estimation methods to explore their use in 3D point tracking, by lifting videos to a 3D world space which makes the scene points easier to track than in the original 2D image plane.
3 Method
Overview. TAPIP3D takes as input an RGB-D video, and outputs 3D trajectories for query points, specified by XY coordinates. If given camera poses, TAPIP3D outputs trajectories in the coordinate system of the first camera (i.e., "world" coordinates). TAPIP3D begins by featurizing the RGB-D data into per-timestep feature maps, and initializing 3D tracks for the query points, and then iteratively refining the tracks with reference to 3D-based lookups in the features.
Concretely, we take as input an RGB video of S frames V = {T' E R J=-1+ a sequence of depth maps D (D' E H }o= camera intrinsics (constant over time), and optionally, sequence of camera poses. The input depth maps can originate from an off-the-shelf monocular depth
estimator 44 depth sensors, GT depth provided by simulation environments. Camera intrinsics and extrinsics can either be estimated algorithmically by video 3D reconstruction models (55 or be provided by the dataset. We are given a set of Q different query points Q = {X y Zzd).
q = 1... Q). where I, represents the timestep at which a query is provided. Queries may also be specified in 2D, in which case we lift them to 3D using depth and camera intrinsics before sending them to the model. For each query, our model produces 3D point trajectory Ty = (X;, V 2;), 8
1... S, and visibility trajectory 0g- The architecture of TAPIP3D is illustrated in Figure 2
3

---

Timestep O
Query Noighbors
Timesop
30 KNN neighbors (Ours
20 Fixed Neighbors mothods)
Input Video
Depth Cam,
Re-somple
3D Neighborhood
Timestep
Updating Transformer
Flow
3D Position xf
Visibiliity of
Extracted 203 Features
Dynamic 3D Fasture Clouds
Art,
Architecture of TAPIP3D. The model takes RGB frames and corresponding 3D point maps as imput, computes features from the RGB frames, and transfers them the 3D points, forming feature cloud for each timestep. Using camera poses, these feature clouds can be arranged in either world space or camera space. We then apply our 3D neighborhood module (Figure 3 to extract features local to the estimated tracks, followed by a transformer, which iteratively updates the estimated trajectories. Top right: Illustration of the difference between 3D k-NN (used in our approach) and fixed 2D neighborhoods (used in prior works [19 5.
Figure 2:
Tracking in world space camera space. Previous 3D point trackers 30 3D point trajectories by concatenating depth values to pixel coordinates, which we refer to as UVD space. In contrast, compensate for camera motion by transforming all frames' points into the first camera's 's coordinate system (if poses available). Disentangling camera motion from scene motion is helpful because makes the 3D tracks more predictable.
40
48
represent
XYZ Z coordinates, which allows to better leverage 3D priors, and also
3.1 Preliminaries: Point tracking as iterative trajectory refinement
TAPIP3D builds the trend of iterative architectures for point tracking 12 from the most recent such tracker, CoTracker3
S
20
19
We build
CoTracker3 overview. CoTracker3 tracks 2D points across time by iteratively refining their coordinates and visibilities through transformer. The transformer model is designed operate on fixed-length sequences of T frames only, and longer videos are processed a bodding-windood manner. Given such sequence and set of query points, for each query it maintains and updates per-frame (2D) locations T (uy vf), and confidences cy and visibilities of The trajectory Ta initialized by copying the query coordinate all frames (zero motion assumption); Gg and Og are initialized with zeros. The model then performs M iterations refinement, producing additive updates the coordinates and visibilities and confidences.
At each iteration m, CoTracker3 samples a patch of 2D features Ft E RC around the current location TH and samples a similar 2D patch around the query's 's reference location, and computes a 4D correlation Corr' between these two feature patches. This correlation provides signal for how well the image content matches the query, at the current estimated location. This correlation information, along with confidence and visibility estimates from the previous step, and position embeddings for motion and time, form a token for each timestep of each track:
(Corr' cg 0% 164 -
( (41 -0) 7()),
where is a Fourier-based position embedding function. The trajectory tokens are then updated through spatio-temporal attentions, which predict updated point positions, confidences and visibilities. At test time, to track points in a video longer than T frames, CoTracker3 advances the inference window by frames, and initializes the first half of the trajectories with the estimates of the previous window, and then repeats inference.

---

3.2 Tracking in a Persistent 3D Geometric Space
Video representation 3D-enriched 2D maps. Our key innovation is represent the input video as a sequence of multi-scale 2D feature maps enriched with 3D coordinates [14] 15] each 2D cell holds both a C-dimensional feature vector and 3D (X, Y, Z) position, where the 3D positions come from unprojected depths. This representation allows us to treat the data as a sequence of featurized point clouds (or "feature clouds"), despite their 2D grid arrangement in memory. Compared to other 3D scene representations such as tri-plane encoders 48]] and UVD coordinates (30), our representation encodes 3D geometry more explicitly and without distortions or approximations. We first encode each video frame with a 2D image encoder with downsampling rate L, producing sequence of feature maps at reduced resolution H W. where W and H are the original width and height of the image. We convert these into L levels of multi-scale features via 2D downsampling: Fi = (F' R R क x (3+ It=1: for ....... L. Our downsampling uses nearest- neighhor interpolation for coordinates and average pooling for features.
3D biighborhood-ooNeee Noeei-hboooh Attention. Our next step is to con- textualize each timestep of each trajec- tory (Tg) with information local to in the input data. Previous methods com- pute correlation features for each tra- jectory point using 2D neighborhoods,
Query Frame
Support Tokens
Poine ML
Pool
Croes
Attention
N
Current Frame
either on the image plane [20] [1 or
on triplanes [48). Our representation
Liniy
30
Point MLP
Az.
enables us to use 3D neighborhoods this step, which lets the model take advantage of the fact that the targets are a 3D world and are often disam- biguated by 3D distance. Our overall strategy illustrated in Figure
Pool
Context Tokens
(N2N) at- tention. Given a 3D query point specific timestep, we first identify its local 3D neighbors using K-NN to form a sup- port group. Then, within the point cloud at another timestep, we find another K nearest 3D neighbors to construct context tokens. We apply bi-directional cross-attention between the support tokens and context tokens capture apatio-temporiy correspondences.
Figure 3: 3D Segghboroood-o--Nighbbboooo
We remind the reader that there are Q queries, and the model operates window of T timesteps at once.
We begin by supplementing each query point with features from lo- cal neighborhood of points. For trajectory Ty whose query originates in frame tg. we retrieve K nearest neighbors from the feature cloud of that timestep (c.g., K = 32), using 3D distances to find neighbors. For each neighbor, we also store its relative offset from the query, and compute positional embedding from this offset. These features, with their position embeddings, serve as "support" tokens for the query, capturing shape information about the query. In total, we have Q K support tokens.
Next, for each trajectory, at each timestep, we use the current estimated coordinate (T₂)) to obtain K nearest neighbors from that timestep's 's feature cloud. For each neighbor, we also store its relative offset from the estimated coordinate (r²), and compute a positional embedding from this offset. These K features, with their position embeddings, serve as "context" tokens, capturing 3D-localized information about the scene at that timestep. In total, we have Q T K context tokens.
We now have a bag of "support" tokens available for each track, and a bag of "context" tokens available for each track at each timestep, and we would like to compute information about how well each track is corresponding at every timestep. For each query, we copy its "support" bag to all timesteps (making a total of Q Q K support tokens), and then perform bi bi-directional cross-attention between each "support" bag and its corresponding "context" bag, followed by a per-token MLP. We then compress each bag into single token, via attention pooling: free-variable tokens (one per bag type) cross-attend to the bags, retrieving their contents, yielding a total of Q T 2 tokens (two summaries per query per timestep). We reduce this to Q T summaries with sum.
5

---

We perform this 3D neighborhood-to htwpbbbbroooo---neiiiiiie fo-neighborhood contextualization for all scale levels in parallel, yielding a total Q- T L tokens. We concatenate across scales, produce Q x ' vectors capturing the multi-scale neighhorhood information, which we denote as AV
3D trajectory updating transformer To complement the multi-scale neighborhood features with information about the trajectory itself. we concatenate sinusoidal embeddings of motion vectors implied by the trajectory, sinusoidal embeddings of 2D coordinates implied by the trajectory, occlusion estimates, and a sinusoidal embedding of the timestep, creating general-purpose purpose trajectory tokens GA
G [N 7(-/ - -(7 Tg), ((=((())))) og (()],
where y denotes the Fourier encoding function and n, R - IR? denotes camera projection. Pixel coordinates explicitly included to assist the model in identifying points located outside the image boundaries.
These trajectory tokens are processed by transformer utilizing proxy tokens, following Co- Tracker3 [19. That is, we use virtual trajectories (that do no correspond any query points) to help approximate spatial and temporal attention across our main Q 'T' tokens. The transformer outputs additive updates for positions and visibility scores:
Tg ATg
og = og Ao
which serve as inputs for subsequent iterations.
Training We train with four iterative inference steps, and supervise each iteration's output. Let (To) denote the predicted trajectories, and let { denote the ground truth trajectories, and similarly define {og) { and {ō for visibility. We define the loss for one iteration as:
L Q T
- + CB(O, oy)
(4)
d
where d denotes the depth of Tig and 25 scales down the loss of far-away points. CE denotes the binary cross entropy loss and Ovis is weighting factor balancing the visibility loss with the tracking loss. During training, we augment the training data by applying random rigid transforms to each frame's point cloud.
Implementation Details Our model trained on the Kubric MOVi-F dataset 9.. We initialize the image encoder with CoTracker3's 's pre-trained weights [19]. We train on 8 L40S GPUs with a batch size of for 200K iterations, which takes roughly 4.2 days and consumes only 20GB VRAM with BF16 mixed precision. We train and test with a window length of 7 = 16. We train with videos of length 24 (making two windows per batch), and 384 trajectories per sample. We scale the coordinates in the point maps to unit variance before processing each window to simplify learning. We optimize using AdamW (28) with the learning rate and weight decay both set to 5e-4. We use a learning rate schedule with 10k warmup steps followed by cosine annealing. During inference, under BF16 mixed precision, our model achieves a speed of 10 FPS and consumes around 2.6GB of VRAM when tracking 1024 query points across 32 frames an L40S GPU.
4 Experiments
We evaluate TAPIP3D on both 3D and 2D point tracking benchmarks, on videos from simulated and real-world scenes with various depth modalities, including sensor depth, estimated depth, and simulator depth. Our 2D estimates are obtained by projecting the inferred 3D point tracks onto the image plane. Our experiments aim answer the following questions: (1) How does TAPIP3D compare with the state of the art in 3D and 2D tracking? (2) How does depth quality (estimated versus GT) affect performance of TAPIP3D? (3) How does camera vs. world space affect performance? (4) How does 3D neighborhood wwwoooodttt-nnhhhooe to-neigbborhood attention compare against other contextualization strategies?
Evaluation metrics We adopt the 3D point tracking metrics from TAPVid-3D 21: : APD3p
(< dmgg) quantifies the average percentage of visible points whose 3D positional errors fall within

---

hapth-adaptive thresholds that scale with the ground-trui truth depth. Occlusion Accuracy (OA) measures the correctness of visibility predictions, while Average Jaccard (AJ) jointly evaluates position and visibility accuracy through the intersection-over-unio between correctly predicted visible points and ground truth.
Table I: Comparison of long-term 3D point tracking methods on the large-scale real-world TAPVid- 3D [2 benchmark. We take the estimated depth from MegaS AM [25]. For TA TAPIP3D-word we also leverage the camera pose estimation from MegaSAM (25). Note that ADT 32 has significant
camera motions while PStudio [17 has a static camera. M- H-SaM: depth and camera poses estimated
by MegaSaM [25].
ADT APD
DriveTrack APD30 1 0A1
PStudio
Average
Methods
AJao
OAt
AJao
AJaD f APDzp 1 OAt
Al
APD 1 OAt
3 19
M-SaM
20.4
30.1
89.8
14,1
20.3
88.5
17.4
27.2
85.0
17.3
25.9
87.8
SpatiaTrracke 48 M-SaM
15.9
23.8
90.1
7.7
13.5
85.2
15.3
25.2
78.1
13.0
20.8
84.5
DELTA
30 M-SaM
21.0
29.3
89.7
14.6
22.2
88.1
17.7
27.3
81.4
17.8
26.3
86.4
TAPIPAD-camert M-SaM
21.6
31.0
90.4
14.6
21.3
82.2
18.1
27.7
85.5
18.1
26.7
86.0
TAPIP3D-world M-SaM
23.5
32.8
91.2
14,9
21.8
82.6
18.1
27.7
85.5
18.8
27.4
86.4
Datasets We evaluate the performance of our model and baselines in the following synthetic and real world datasets: 1) TAPVid3D (2T) integrates videos from three distinct real-world datasets covering diverse scenarios: Drive'Track []] Panoptic Studio [17], and Aria Digital Twin [32] Together, these datasets provide a total 4,569 evaluation videos, with video lengths ranging from 25 to 300 frames. 2) DexYCR-Pt: dataset we introduce by leveraging ground-truth object and hand poses from manipulation scenes in Dex YCB to generate accurate 3D point tracks on 8,000 real-world RGB-D videos. 3) LSFOdyssey [40]: a synthetic dataset that contains 90 videos with complex but realistic long-range motion using humanoids, robots, and animals. Each video is 40 frames long and has 3D point trajectories and occlusion labels, sourced from PointOdyssey [51] 4) DynamicReplica [8] a synthetic 3D dataset that contains 500 videos of articulated human and animal models. Each video is 300 frames long.
Baselines We compare TAPIP3D against the following 2D and 3D point trackers: 1) DELTA 30 a very recent state-of-the-art 3D point tracker that uses depth as an additional input for computing cross-correlations, essentially extending CoTracker [20] to UVD space. SpatialTracker [48 a 3D point tracker that builds upon CoTrrackr's ' architecture and uses a tri-plane representation to featurize (u, v, d) coordinates.
3) CoTracker3 [19], which operates in 2D, but we lift its trajectories to 3D using the depth at the trajectory coordinates.
We consider the following versions of our model:
1) TAPIP3D-world: This is TAPIP3D tracking in world coordinates, by using the camera pose trajectory to cancel camera motion in the 3D coordinates.
2) TAPIP3D-camera. This is TAPIP3D tracking in camera coordinates, where information of camera pose is not used, and thus the observed 3D motion reflects the combination of camera motion and scene motion.
In all the evaluations, our model is trained on the Kubric MOVi-F dataset 99] TAPIP3D-wond and TAPIP3D-camer adopt the same trained checkpoint but conduct inference in the two different 3D coordinate systems.
4.1 Point Tracking Evaluation
We first evaluate TAPIP3D against strong baseline methods [19] 48 30 on the challenging real-world TAPVid-3D 21 benchmark. Table Sooberrium the comparative 3D tracking performance, utilizing
depth and camera poses obtained from state-of-thee -he-hhh-rrttr art 3D reconstruction models (25) for each frame. TAPIP3D consistently outperforms recent state-of-the-are methods, including SpatialTracker [30] and DELTA 48) across all three subsets in terms of AJ We also visualize the 3D point tracking results Figure 4 3D point tracks from TAPIP3D are noticeably more accurate than those of the baselines. Notably, on the ADT dataset (32) characterized by particularly challenging camera motions, TAPIP3D-world significantly surpasses DELTA 30, achieving improvements of 2.5 points in AJap and 3.5 points in APD3D. Furthermore, TAPIP3D-world demonstrates clear advantage
7

---

TO
T2
Ours
Spa macker
3D Point Cloud of Query Video Frame
DELTA
Figure 4: Visual results comparison. We qualitatively compare the 3D point tracking results between TAPIP3D with state-of-the-ar approaches SpaTracker [48] and DELTA 30 under the same 3D point cloud produced by MegaSaM (25), where we observe serious point drifting issues of SpaTracker and DELTA under large 3D object motion. Zoom for better view.
over TAPIP3D-camera. highlighting the benefits of conducting tracking in a persistent 3D world space rather than in camera-centric coordinates.
Next, we present results for 3D point tracking on the real-world manipulation dataset DexYCE [4]. shown in Table 3 TAPIP3D notably boosts performance over the previous best method [30], improving AJaD from 26.4 to 30.3. Additionally, we observe that SpaTracker [48] achieves limited
performance, even falling below the depth-lifted CoTracker3 [19]. This suggests that Spa Spackr''sr's tri-
plane representation struggles to leverage the accurate sensor depth provided by Dex YCB effectively.
Table 2: Comparison of long-term 3D and 2D point tracking results LSFOdyssey 40 and Dynamic
Replica [18) benchmarks with improved input depth quality (MegaSAM (25
GT Depth). The
best results per column are highlighted in bold. GT denotes using GT depth and camera poses
provided by the dataset. M-SaM: depth and camera poses estimated by MegaSaM (25).
LSFOdyssey
Dynamic Replica
Methods
AJso
APDap
AJı
APD
OAf
AJ
APDso
AJa
APD
OAt
CoTrucker3 19
M-SuM
19.0
28.1
75.0
88.0
88,5
20.0
30.2
61.4
80.0
86.2
SpatialTracker
48
M--aM
14.8
22.8
73.8
88.5
85.7
6.6
12.0
54.1
72.8
81.7
DELTA
30
M-SaM
18.9
28.0
75,2
90.2
83.7
18.0
27.2
61.3
77.3
81.0
TAPIP3D-camera M-SaM
20.2
28.6
70.4
86.2
83.8
20.6
30.3
55.3
77.1
80.2"
TAPIP3D-world M-SaM
20.5
29.2
72.3
87.5
83.6
20.6
30.2
56.1
77.7
78.6
CoTracker3 +
28.4
35.0
75.0
88.0
88.5
27.4
38.0
61.4
80.0
SpatialTracker 48 GT
8.2
13.3
70.6
85.9
84.9
7.6
13.6
56.4
74.7
83.8
DELTA 30
37.7
50.1
72.4
88.4
82.3
27.4
37.7
65.6
80.5
83.8
TAPiPP3Dccmmer: + GT
68.3
83.2
76.0
91.2
86.2
53.7
70.8
64.6
84.7
84.77
TAPIP3D-world GT
72.2
85.8
78.5
92.8
86.9
55.5
72.8
66.2
85.7
85.3
Lastly, we evaluate our model's s 3D point tracking performance on the synthetic LSFOdyssey [40] and DynamicReplica (18) benchmarks, as shown in Table Comparisons are conducted between our method and baselines under different depth sources: ground-truth depth from simulation, and esti- mated depth from MegaSaM (25). We report two notable findings: 1) TAPIP3D-world outperforms all competing 3D point trackers in AJap when provided with depth estimation from either MegaSaM [25) or ground-truth depth. Moreover, utilizing ground-truth depth, our approach even outperforms the tracking accuracy of CoTracker3 [19]. 2) As depth quality improves (MegaSaM 22 ->
GT Depth). the performance enhancement of our method in APDaD is substantially greater com- pared to DELTA (30) and SpaTracker
48 underscoring the effectiveness our 3D point feature representation in world coordinates.

---

Table 3: 3D point tracking comparison on
Table 4: Ablation on the searching neighbors meth-
DexYCB-Pt using Sensor Depth (SD).
ods when performing correlation. We compare 3D k-NN vs. Fixed 2D Neighbors in DexYCB-Pi bench- mark using Sensor Depth (SD).
DexYCB-Pt
Methods
AJap
APD 1 OAt
CoTrucker3 + SD 19
14.9
26.1
70.9
Methods
AJ t APD t OAt
SpatialTracker
48
5.5
11.4
66.8
DELTA 30
26.4
43.3
72.8
Fixed Neighbors in 2D
27.7
50.0
67.7
k-NN in 3D
29.8
51.6
70.9
TAPIP3D (Ours)
30.3
52,4
71.3
Table 5: Ablation experiments of TAPIP3D. To isolate the influence of depth and camera pose, the annotations provided by the LSFOdyssey dataset [40]. N2N Att.: Neighborhood Sertorhhoo-tt-Neeighhee to-Neighborhood Attention. Cam: 3D camera space. World: 3D world space. Note that the Dex YCB-Pt video dataset is with static camera such that the result number for Cam and World is the same.
Coord Systems
LSFOdyssey
LSFOdyssey
DexYCB-Pt
Methods
aJ t APDap
OA
AJ1D f
APD 1 OA AJaD 1 APD
OA1
UV +D
63.4
77.0
87.0
Cam w/o N2N
59.4
72.7
88.0
26.3
46.6
68.8
+ l))(D)
62.9
77.9
84.1
Cam w N2N Att.
67.1
81.6
85.8
29.8
51.6
70,9
XYZ (Cam)
67.1
81.6
85.8
World wfo N2N At..
62.1
75.1
88.2
26.3
46.6
68.8
XYZ (World)
70.7
84.1
86.6
World w N2N At.
70.7
84.1
86.6
29.8
51.6
70.9
(a) 3D Coordinates Systems
(b) Ablation on N2N Attention
4.2
Ablations
In Tables aand we ablate our model's design choices regarding the selection of nearest neighbors in 3D vs. 2D space, UVD vs. XYZ space for point tracking and neighborhood sestthooo-----tttteete to-neighborhhod attention vs. conventional point-to-neighhorhood cross attention for feature extraction. We draw following conclusions:
M Computing nearest neighbors in 3D space is better than fixed 2D. In Table we see that our 3D k-NN design improves the AJıD metric from 27.7 29.8. This indicates that 3D k-NN mechanism leverages 3D geometric information to effectively filter out irrelevant 2D neighbors during correlation, illustrated in Figure
TAPIP3D that tracks in XYZ (world) and XYZ (Cam) space outperform versions that track in VD and UV+log(D) space, the latter two commonly adopted in existing offte-of-the-ar methods 20 30 Specifically, tracking world XYZ coordinates performs best (AJan metric). Tracking in world coordinates generalizes effectively to scenarios with significant camera motion, such as those LSFOdyssey [40]
Tracking in world space helps. In Table 5a
Considering support points during attentions helps. In Table 5b we compare the proposed
Attention mechanism against conventional point-to-neighborhood cross attention. Specifically. we replace the Neighborhood-to-NNer to-Neighborhood Attention, which groups each query point with its neighboring context points in both spatial dimensions, with standard 2D attention that considers only the individual query points. Our experiments show that Neighborhood- to- Noighhoboodood Attention significantly improves the APD3D metric from 75.1 to 84.1. This region- to-region cross-attention approach notably mitigates matching ambiguities by incorporating richer context compared to the simpler wiin--o--neigboorhood matching baseline. Similar conclusion was reached in LocoTrack [5] but for 2D image plane using CNNs, which we extend to 3D feature clouds using cross-attention.
Limitations
Despite achieving ofte-of-the-ar results, TAPIP3D's performance can be affected by the fidelity of the input depth maps. As a model that works in XYZ space, it expects the input geometry to remain stable and consistent across frames to ensure well-defined 3D trajectories. While MegaSaM generally provides reliable geometry, failures may occur in scenes with extreme depth variations or small, distant elements that appear blurred. In such cases, depth flickering or incorrect surface connections may arise, Icading to degraded tracking quality. Furthermore, when high-qualit depth

---

(e.g., from sensors) is not available, TAPIP3D may perform worse on 2D metrics than UVD-space or purely 2D tracking methods, since its 2D trajectories are projected from 3D estimates and rely on geometrically consistent data. This limitation can be mitigated via depth map pre-processing methods such as completion and noise filtering. With ongoing advancements in 3D vision reconstruction models offering increasingly accurate priors, we anticipate continued improvements in the robustness of our method.
Conclusion
We introduced TAPIP3D, novel approach for multi-frame 3D point tracking that represents videos as spatio-temporal 3D feature clouds in either camera-centric or world-centric coordinate frame, and employs feature clouds, and iteratively updates the trajectories. By lifting 2D video features into a structured 3D space using depth and camera motion information, TAPIP3D addresses fundamental limitations of previous 2D and 3D tracking methods, particularly in handling large camera motion. We validated TAPIP3D on established synthetic and real world tracking benchmarks demonstrating state-of-the- performance in 3D point tracking when ground-truth depth is available and competitive performance with estimated depth. By leveraging advances in depth estimation and camera pose prediction, TAPIP3D paves the way for more robust and accurate multi-frame tracking in both RGB and RGB-D videos.
attention to contextualize the estimated tracks in the
Acknowledgements
This material based on work supported by an NSF Career award, an Amazon faculty award, and AFOSR Grant FA9550-23-1-0257.
References
Arjun Balasingam, Joseph Chandler, Chenning Li, Zhoutong Zhang, and Hari Balakrishnan. Drivetrack: benchmark for long-range point tracking in real-world videos. In CVPR, 2024.
[2] Weikang Bian, Zhaoyang Huang, Xiaoyu Shi, Yitong Dong, Yijin Li, and Hongsheng Li. Context-pips persistent independent particles demands spatial context features. NeurIPS, 36, 2023.
[3] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J Guibas, Jonathan Tremblay, Sameh Khamis, al. Efficient geometry- 3d generative adversarial networks. In CVPR, 2022.
[4] Yu- Wei Chao, Wei Yang, Yu Xiang, Pavlo Molchanov, Ankur Handa, Jonathan Tremblay, Yashraj S Narang, Karl Van Wyk, Umar lqbal, Stan Birchfield, et al. Dexycb: A benchmark for capturing hand grasping of objects. In CVPR, pages 9044-9053, 2021.
[5] Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon- Youn- Lee. Local all pair correspondence point tracking. In ECCV, 2024.
[6] Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. In NewrlPS, 2024.
[7] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. In NeurIPS, 2022.
[8] Carl Doersch, Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta. Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In ICCV, 2023.
[9] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herraann Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji. Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour. Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu. Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A scalable dataset generator. In CVPR, 2022.
10

---

[10] Xiuye Gu, Yijie Wang. Chongruo Wu, Yong Jae Lee. and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation large-scale point clouds. In CVPR, pages 3254-3263, 2019.
[11] Simon Hadfield and Richard Bowden. Kinecting the dots: Particle based scene flow from depth sensors. In ICCV, 2011.
[12] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle vidco revisited: Tracking through occlusions using point trajectories. In ECCV, 2022.
[13] Michael Hornacek, Andrew Fitzgibhon, and Carsten Rother. Sphereflow: 6 dof scene flow from rgh-d pairs. In CVPR, 2014.
[14] Ayush Jain, Pushkal Katara, Nikolaos Gkanatsios, Adam W Harley, Gabriel Sarch, Kriti Aggarwal, Vishrav Chaudhary, and Katerina Fragkiadaki. Odin: single model for 2d and 3d segmentation. In CVPR, 2024.
[15] Ayush Jain, Alexander Swerdlow. Yuzhou Wang. Sergio Arnad, Ada Martin, Alexander Sax. Franziska Meier, and Katerina Fragkiadaki. Unifying 2d and 3d vision-language understanding. arXiv preprint arXiv:2503. 10745, 2025.
[16] Linyi Jin, Richard Tucker, Zhengqi Li, David Fouhey, Noah Snavely, and Aleksander Holynski. Stereo4d: Learning how things move in 3d from internet stereo videos. arXiv preprint, 2024.
[17] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, lain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. Panoptic studio: A massively multiview system for social motion capture. In ICCV, 2015.
[18] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dynamiesterco: Consistent dynamic depth from stereo videos. In CVPR, 2023.
[19] Nikita Karaev, Turii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.1181/ 2024.
[20] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. In ECCV, 2024.
[21] Skanda Koppula, Ignacio Rocco, Yi Yang, joseph heyward, Joao Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. Tapvid-3d: A benchmark for tracking any point in 3d. In NeurlPS, 2024.
[22] Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. In CVPR, 2024.
[23] Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers detection. In ECCV, 2024.
[24] Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Bohan Li, Tianhe Ren, and Lei Zhang. Taptrv2: attention-based position update improves tracking any point. Advances in Neural Information Processing Systems, 2025.
[25] Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang. Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, and Noah Snavely. MegaSaM: Accurate, fast and robust structure and motion from casual dynamic videos. arXiv preprint, 2024.
[26] Jiuming Liu, Guangming Wang. Weicai Ye, Chaokang Jiang, Jinru Han, Zhe Liu, Guofeng Zhang, Dalong Du, and Hesheng Wang. Ditflow3d: Toward robust uncertainty-avy scene flow estimation with iterative diffusion- based refinement. In CVPR, 2024.
[27] Xingyu Liu, Charles R Qi, and Lconidas Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019.
[28] Tlya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
[29] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
[30] Tuan Duc Ngo, Peiye Zhuang, Chuang Gan, Evangelos Kalogerakis, Sergey Tulyakov, Hsin- Ying Lee, and Chaoyang Wang. Delta: Dense efficient long range 3d tracking for any video. In ICLR, 2025.
11

---

[31] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d recon- struction by learning particle dynamics. In ICCV, 2019.
[32] Xiaging Pan, Nicholas Charron, Yongqian Yang. Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. ICCV, 2023.
[33] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Bruall.. Brualla. Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[34] Julian Quiroga, Thomas Brox, Frédéric Devernay, and James Crowley. Dense semi-rigid scene flow estimation from rgbd images. In ECCV. 2014.
[35] Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. IJCV, 80:72-90 91, 2008.
[36] Jenny Seidenschwarz, Qunjie Zhou, Bardienus Duisterhof, Deva Ramanan, and Laura Leal-Taixé. Dynomo: Online point tracking by dynamic online monocular gaussian reconstruction. arXiv preprint arXiv:2409.02100 2024.
[37] Pei Sun. Henrik Kretzschmar, Xerxes Dotiwalla. Aurélien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020.
[38] Zachary Teed and Jia Deng. DROID-SLANE Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras. In NeurIPS, 2021.
[39] Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid- motion embeddings. In CVPR, 2021.
[40] Bo Wang, Jian Li. Yang Yu, Li Liu, Zhenping Sun, and Dewen Hu. Scenetracker: Long-term scene flow estimation network. arXiv preprint 199944 2024.
[41] Qianqian Wang, Yen-Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. In ICCV, 2023.
[42] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhenggi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint 24xx:2407.13764 2024.
[43] Qianqian Wang. Yifei Zhang, Aleksander Holynski, Alexei A. Efros, and Angjoo Kanazawa. Continuous 3d perception model with persistent state. In CVPR, 2025.
[44] Ruicheng Wang, Sicheng Xu, Cassie Dai, Jianfeng Xiang, Yu Deng, Xin Tong, and Jiaolong Yang. Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision. arXiv preprint arXiv:2410. 191115, 2024.
[45] Shuzhe Wang, Vincent Leroy. Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Geometric 3d vision made easy. In CVPR, 2024.
[46] Zirui Wang, Shuda Li, Henry Howard-Jenkins, Victor Prisacariu, and Min Chen. Flownet3d+4: Geometric losses for deep scene flow estimation. In WACV, 2020.
[47] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundation- stereo: Zero-shot stereo matching, 2025.
[48] Yuxi Xiao, Qianqian Wang. Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In CVPR, 2024.
[49] Gengshan Yang and Deva Ramanan. Leaminng to segment rigid motions from frames. In CVPR, 2021.
[50] Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, Deqing Sun, and Ming-Hsuan Yang. Monst3r. A simple approach for estimating geometry the presence of motion. In ICLR, 2025.
[51] Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas J. Guibas. Pointodyssey: large-scale synthetic dataset for long-term point tracking. /CCV, 2023.
12

---

Supplementary Material
In this supplementary material, Section A further training details of our model, including a memory design that significantly helps reduce VRAM usage. Section explains the details of our evaluation setup. In Section[C we provide details regarding the inference speed of our model. In Section D we present additional visualization FoundationSterro [47]. powerful stereo matching method. Section provides further ablation results to offer insights into model behavior. Additionally, we report the evaluation results of our model HAPVid-DAVIS 7 in Section G For extensive visual comparisons of our results in please refer to our project page.
our method using stereo depth, enabled by
Section H
A
Additional Training Details
Table 6 lists the hyperparameters used during training. We adopt all the data augmentations from PIPs [12 (also used CoTracker3 [9]). including random Gaussian blur, random occlusion, random cropping, and color jittering, along with smoothly varying sequence of random rigid transformations applied on the input point clouds to improve camera-centric performance. During training, we perform Mrain 4 iterative refinements per window. The loss at the m-th iteration is weighted by 7 M where is the discount factor. The total loss for each sample is computed as the sum of these discounted losses across all iterations and windows. To mitigate unstable gradients in the early training stage, we additionally scale the total loss by 0.005.
Table 6: Training hyperparameters
Hyperparameter
Value
Learning rate
0.0005
Weight decay
0.0005
Iteration refinements (Mmiin)
LR schedule
OneCycleI.R
Training steps
200,000
Batch size
Optimizer
Adam W
Max grad norm
10.0
Visibility loss weight (avis)
3.0
Loss discount factor (7)
0.8
Total loss multiplier
0.005
During training, we observed that the a large amount of GPU memory, as memory usage accumulates over the course of iterative predic- tions. To reduce this VRAM overhead, we introduced memory-saving strategy that significantly helps reduce VRAM usage. Specifically, visibilities after each iteration, allowing the loss for each window and iteration to be computed and backpropagated independently. We immediately backpropogate the loss of each window as soon is ready, and release the part of the computation graph that is no longer needed. The only component that needs to retain shared computation graph across iterations is the image encoder. To avoid repeated backpropagation through it, we truncate the gradient at the image features and accumulate gradients there. These accunulated gradients are then backpropagated once, after the final iteration of each batch.
attention module consumes
detach the gradients of the predicted coordinates and
13

---

This strategy ensures that VRAM usage remains constant regardless of the number of iterations and windows. Despite its simplicity, it has minimal impact on training speed and does not sacrifice model performance, while significantly reducing memory usage-from over 48GB (which previously caused GPU memory overflow) to approximately 20GB.
B Evaluation Details
For evaluation, we use the metrics defined in TAPVid 7 and TAPVid-3D [2] throughout the paper. All baselines, as well as our method, operate in window-by-winden manner, so they only predict trajectories after the query frame (i.e., in the forward direction). However, the TAPVid-3D metric also requires evaluating trajectories before the query frame (i.e., in the backward direction). Therefore, to obtain these backward trajectories, we reverse the video in time and run inference again.
Following previous 2D tracking methods [19 [ we augment the provided query points with additional support queries sampled from a 2D grid in the first frame. These points are lifted into 3D and inferred jointly with the original queries. The grid resolution is set to 16 x 16 for all models. For datasets with known camera intrinsics, we use the ground-truth intrinsics both to lift image features into 3D feature clouds and to convert the outputs of UVD-space models into XYZ coordinates. We do not apply the trick of treating low-confidence point as occluded, as done in CoTracker3 [[].
Our model additionally requires scale factor to normalize the coordinates when constructing spatio- temporal feature clouds. During training, we use the standard deviation of all 3D points in the feature clouds as the normalization factor. To reduce the influence of extreme values from depth estimates during evaluation, we assume rough upper bound for the depths of interest, which is computed as twice the maximum depth of ground truth tracks. We then compute the scale factor as the standard deviation of 3D points whose depths fall within the upper bound. We note that points lying out of the depth upper bound are not excluded from the feature clouds and can still be tracked by the model.
C Inference Speed
To assess the efficiency of our model, we measure its inference speed on a single L40S GPU using BF16 mixed precision. We evaluate the model on 32-frame video sequences, where 1,024 query points are provided in the first frame. Our model achieves an inference speed of 1 11.3 FPS. It is approximately 1.3x slower than SpaTracker [48] on the same videos. It is of course substantially faster than optimization-base methods such as Shape-of-Motion
42 which require several hours of
inference-time optimization for 300-frame sequence.
For monocular videos, we find that generating depth maps and estimating camera poses are the most time-consuming components in our pipeline. For instance, processing sequence from DexYCH takes 121.1s for geometry estimation with MegaSaM, yet our model only requires 6.4s for point tracking. Among this, feature extraction accounts for 0. 0.1 encoding trajectory tokens for 6.04s. and the updating transformer for 0.25s. We also observe that our model is more robust to degraded camera pose quality than to degraded depth estimation. Consequently, one potential way to accelerate the pipeline is to use more efficient depth estimator, such as CUT3R 43), especially in scenarios where sensor depth is available. We expect that future improvements depth estimation models will help reduce this bottleneck.
D Inference with Stereo Videos
While method requires reasonably accurate unprojected point clouds as input, obtaining high- quality depth maps remains challenging in many real-world scenarios. To mitigate the limitation, we demonstrate how our method can be combined with FoundationStereo [47]. recent advancement in stereo matching, to produce metric-scaled point tracks from in-the-wild stereo videos.
Specifically, we follow the pipeline proposed in Stereo4D [16] to download stereo videas in the VRI80 format from You'ube and rectify them. We then estimate disparity maps frame by frame using FoundationStereo [47, using them to unproject the image features into camera-centric spatio- temporal feature clouds. Figure 5 shows two examples produced by this process. Comparison videos are provided in the supplementary material SAAPIP3D-supp.mpp While the examples are
14

---

Queried Frame
TO
T2
Figure 5: Examples of running our model with depth maps estimated by FoundationStereo 47 Tracking is performed in the camera-centra frame, so the trajectories above include both camera and object motion. Refer our videos on the project page for better visual comparison.
camera-centric tracking results, one can also combine the estimated disparity with camera poses from methods such MegaSaM [25 to track in world-centric coordinates.
E
Additional Ablations and Analyses
E.1
Ablation on Neighborhood Size
We train our model with K= = 32, where K denotes the number of neighbors used in the N2N attention module, and evaluate its inference performance under different K values on both LSFOdyssey and Dynamic Replica. As shown in Table 7 the performance of TAPIP3D improves as K increases, though the gain saturates when increases from 32 to 48 on LSFOdyssey. The computational complexity of the N2N attention module, given by O(TNK) K + T2KD2), makes it a major bottleneck our model, where T is the number of frames, N the number of tracked points, and D is the feature dimension. We set K 32 as a trade-off between performance and efficiency.
Table Ablation on Neighborhood Size K
LSFOdyssey
Dynamic Replica
Neighborhood Size K
AJaD
APDaD f OAsp 1
AJap f
APD3p 1
OAap 1
K = 16
67.9
84.5
83.6
52.3
68.8
83.2
K 24
71.1
85.8
85.9
55.5
73.0
84.8
K 48
72.6
85.3
88.6
57.3
74.3
85.8
K 32
72.2
85.8
86.9
55.5
72.8
85.3
E.2 Robustness to Image Blur
Real-world images of dynamic objects often exhibit blurriness, especially under low resolution and fast motion. To assess model's robustness to such degradation, evaluate WoPll3D-world + MegaSaM on the LSFOdysscy dataset using synthetic Gaussian blur of varying strength. For each frame, the input image is convolved with Gaussian kernel of standard deviation E (0.0, 0.5, 1.0, 2.03. Since image blur affects the reliability of depth estimation, we re-estimate the depth maps of blurred images using MegaSaM.
As shown in Table 8, our model consistently achieves higher 3D accuracy than baseline DELTA across all blur levels. We observe that even under heavy blur, the predicted 3D points remain well aligned with scene geometry, although the depth estimated by MegaSaM becomes less consistent across frames.
15

---

Table Effect of Gaussian blur on LSFOdyssey. Blurred images are used for both TAPIP3D and DELTA, and corresponding depths are re-estimated using MegaSaM.
TAPIP3D (ours)
DELTA
B (px)
APD3D
APD20
APD3D
APD2D
0.0
29.2
87.5
28.0
90.2
0.5
21.2
87.4
20.3
90.3
1.0
20.6
86.6
19.8
90.3
86.0
17.5
90.2
E.3 Robustness to Camera Pose Errors
To evaluate our model's robustness against inaccurate camera poses, we synthetically perturb the ground-trute camera orientations. For each frame. we apply rotation around random axis, where the rotation angle is sampled from a zero-mean Gaussian distribution with a standard deviation of {0.5", 1.0% 3.
As reported in Table 9 our model, TAPIP3D-worId, demonstrates considerable resilience to these perturbations. The performance degrades gracefully the noise level increases, and our model consistently outperforms the DELTA baseline even under the perturbation of a 2.0%.
Table 9: Impact of camera pose perturbations. We add Gaussian angular noise with standard deviation to the ground truth camera orientations and report results on the LSFOdyssey dataset.
(deg)
Method
AJ
APD3p
OA
N/A
DELTA
37.1
50.1
82.3
N/A
TAPIP3D-camera
68.3
83.2
86.2
0.0
TAPIP3D-world
72.2
85.8
86.9
0.5
TAPIP3DD-world
70.4
84.6
86.8
1.0
TAPIP3D--orld 67.8
82.7
86.4
2.0
TAPIP3D-word
64.9
80.3
85.7
Evaluation with Sparse Depth
In some real-world scenarios, we only have access to sparse depth measurements (e.g., from LiDAR sensors) instead of dense depth maps. To assess the usefulness of our approach under such conditions, we construct a benchmark consisting of 50 sequences from the Waymo Open Dataset [37 (a subset of TAPVid-3D), paired with depth map annotations obtained by densifying sparse LiDAR measurements using a simple neirhor interpolation, following the approach in DriveTrack M
We report the performance of TAPIP3D-cament DELTA, and CoTracker3 on this benchmark in Table 10 TAPIP3D-cames achieves the best 3D accuracy among the compared methods. These results demonstrate that our method is highly usable and competitive when only sparse depths are available.
Table 10: Evaluation with sparse depth on a subset the Waymo Open Dataset [7]. Depth maps are obtained by densifying sparse LiDAR signals via nearest-neighbo interpolation.
Method
AJ
APDsp
AJıD
APD2D
CoTracker3
16.5
24.1
63.7
78.8
DELTA
21.0
30.8
71.4
85.0
TAPIP3D-camera
21.8
31.6
67.7
82.2
16

---

G TAP-Vid Evaluation
2D point tracking benchmark, TAPIP3D-wond achieves AJ of 58.9, an APD of 71.2, and an OA of 89.6, which are lower than those of recent state-of-the-art 2D trackers.
For reference, also evaluated TAPIP3D-world on TAP-Vid-DAVIS 77,.
We note that TAPIP3D is not designed to achieve state-of-the-art 2D tracking performance, and the 2D tracks obtained by projecting 3D trajectories 2D cannot reach sub-pixel accuracy in monocular settings. The performance further affected by MegaSaM's 's failures on some of sequences, which lead to severely noisy depth estimates and unstable point clouds.
H 3D Track Visualization
For extensive 3D visualizations of our results. please refer to the accompanying project page, where we compare our method with the baselines SpaTracker [88, CoTracker3 [19 and DELTA [30]
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We provided extensive experiments to support the paper's claims.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please check Sec 4.2 of the paper.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: No theoretical results, and the assumptions have been made explicit in the paper.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provided experiment descriptions in the main paper and appendix, and code is available online.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will make both code implementation and data public upon paper's acceptance.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided detailed experiment settings and details in the Sec 4 of the main paper and our appendix.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct experiments diverse datasets and follow the protocol used by previous works for fair comparisons.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
18

---

Justification:
This information is included in our implementation details at the end of Sec 3.
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our paper follows the NeurIPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We studied foundational 3D computer vision topic with directly asso- ciated negative social impact. We illustrate the importance of our studied problem in the introduction of the paper.
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our work does not pose such misuse risks.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly cited papers and resources used our experiment.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include code with readme and comments.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We don't have experiments involving crowdsourcing research with human subjects.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We do not involve crowdsourcing research with human subjects.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
19

---

Answer:
[NA]
Justification: We do not involve LLMs as any important, original, or non-standard compo- nents.
20


Reviews:
[{"review_id": "k8jP9QsWIc", "reviewer": "Reviewer_4cTm", "summary": "The paper transforms the 3D tracking problem from a frame-to-frame 2D problem into a frame-to-model 3D tracking approach. This method effectively decouples object motion and camera motion in the scene to simplify the problem. It employs a novel neighborhood attention mechanism instead of traditional matching methods to optimize the entire 3D-based tracking approach. Through extensive qualitative and quantitative analysis on datasets, each designed component significantly enhances the overall system while maintaining good real-time performance.", "questions": "Have you investigated the impact of varying degrees of input image blur on the system? For example, motion blur caused by camera or object movement, or severe blur due to defocus.\n\nMost of the qualitative results focus on objects with significant motion in the foreground. Have you explored whether robust 3D tracking can be achieved for objects at a far distance with significant blur, such as people in the audience seats in a football field?", "limitations": "No, I believe that, beyond the illumination flickering and certain depth map dependency discussed by the authors, this work also has the limitation of struggling to achieve accurate trackig under severe motion blur. Additionally, depth maps estimated from monocular inputs are affected by blur, leading to erroneous edge estimation.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Paper Strengths:\n\n1. The paper achieves more efficient optimization by decoupling the representation of tracking processing from camera motion, while simplifying the complexity of the target function in high-dimensional spaces.\n\n2. The paper enhances tracking consistency in 3D space through cross-attention in 3D neighborhoods.\n\n3. The proposed system exhibits certain robustness to occlusions and can benefit from the quality of depth information.\n\nMajor Weaknesses:\n\n1. The paper mentions that systems like NeRF incur significant computational overhead but does not provide a detailed comparison. To my knowledge, 4DGS [1] can achieve real-time performance at 82 FPS with good results.\n\n2. Regarding robustness, the paper does not explicitly investigate the impact of blurry input images on the overall system. It only explores the system’s usability under different depth conditions and low-light scenarios.\n\n[1] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4D Gaussian splatting for real-time dynamic scene rendering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024, pp. 20310–20320.", "was_revised": true, "final_justification": "Overall, this is a good paper with solid contributions and good experimental validation which merit publication. The authors addressed the reviewers concerns well in the rebuttal. Therefore, I would raise the score to 5.\nI trust the authors to incorporate as much reviewer feedback as possible into the final version.", "rebuttal": "### Response to Reviewer 4cTm\n\nWe thank the reviewer for the thoughtful and detailed feedback, and for highlighting the strengths of our decoupling strategy and 3D N2N attention mechanism.\n\n**1. Efficiency Comparison with Optimization-Based Methods**\n\nWe acknowledge the concern regarding efficiency comparisons with optimization-based scene representation methods such as 4DGS [1] and appreciate the opportunity to clarify. \n\nWhile 4DGS reports high rendering frame rates of 82 FPS, this only reflects only its rendering speed after training. The method requires time-consuming per-sequence optimization, with training times reaching up to 30 minutes per sequence on the HyperNeRF dataset, as reported in Table 2 of [1]. Similar limitations apply to other optimization-based methods such as NeRF. \n\nIn contrast, TAPIP-3D performs long-term point tracking in a fully feedforward manner, running at approximately 10 FPS without any per-sequence training (fitting). \n\nGiven the significant differences in computational cost and target applications, direct efficiency comparisons between these methods are not informative in practice. We will clarify this important distinction in the next version of the paper.\n\n\n**2. Robustness to Blur**\n\n\n**2.1 Impact of Blur at Varying Degrees**\n\nOur method augments the training data by introducing motion blur and randomly applying Gaussian blur to both images and depth maps to improve robustness against blurry inputs. We also adopt all the 2D data augmentations from CoTracker3 as mentioned in L10 of the Supp. file, which includes gaussian blur to images. We will detail it in the paper. \nAs a result, our model can tolerate some degree of blur. \nTo evaluate the model’s robustness more concretely, we present a table below showing quantitative results of TAPIP3D-world + M-SaM on the LSFOdyssey dataset with varying levels of gaussian blur applied:\n\n| σ (Blur Level) | Method   | APD-3D     | APD-2D     |\n|----------------|----------|------------|------------|\n| 0 (No Blur)    | TAPIP-3D (ours) | 29.2       | 87.5       |\n|                | DELTA    | 28.0       | 90.2       |\n| 0.5            | TAPIP-3D (ours) | 21.2       | 87.4       |\n|                | DELTA    | 20.3       | 90.3       |\n| 1.0            | TAPIP-3D (ours) | 20.6       | 86.6       |\n|                | DELTA    | 19.8       | 90.3       |\n| 2.0            | TAPIP-3D (ours) | 18.1       | 86.0       |\n|                | DELTA    | 17.5          | 90.2          |\n\nNote that in the table above, our model's 3D performance surpasses the baseline, DELTA, at every blur level, demonstrating competitive robustness. From the visualizations, we observe that with blurry inputs, the points align reasonably well with objets, while depth estimates of the same object become less consistent across frames, possibly due to blurry inputs being out-of-distribution for MegaSAM.\n\n**2.2 Tracking Blurry and Distant Objects**\n\nWe acknowledge that low-resolution and distant objects may lead to degraded geometry, particularly around edges. To mitigate artifacts from edge interpolation caused by depth estimation, we apply a simple edge-removal preprocessing step before feeding the depth maps into our pipeline. This approach is commonly adopted in 3D visualization for depth models.\n\nWith this preprocessing in place, we observe that on DriveTrack (a subset of TAPVid-3D), where vehicles often appear small, blurry, and distant, our model is often able to track points on these objects despite the sparse pixel density after unprojection. \nHowever, we acknowledge some failure cases, particularly when two distant vehicles occlude each other. In such scenarios, the estimated depth geometry may become ambiguous, with disconnected objects appearing fused, making it hard for our model to distinguish between them. We believe this issue is related to the low resolution of the depth model (MegaSAM operates at 192×256 and is upscaled prior to use in our model). We expect that future improvements in foundational geometry (depth and camera pose)  estimation models will help mitigate these challenges. We will include a discussion of these limitations in the final version of the paper.\n\n\n\n----------\n\n**References:**\n\n[1]  Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4D Gaussian splatting for real-time dynamic scene rendering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024, pp. 20310–20320."}, {"review_id": "w3c7pm3zSh", "reviewer": "Reviewer_Sbrq", "summary": "This paper addresses the problem of 3D point tracking. It begins by extracting depth and camera parameters from MegaSAM, with the flexibility to incorporate higher-quality depth data if available. Subsequently, it extracts hierarchical image features using CoTracker's feature backbone. These image features are then concatenated with point maps derived from MegaSAM. The core of the model involves a transformer-based mechanism that performs \"neighbors-to-neighbors\" attention. This approach can be conceptually understood as a 3D extension of the 4D local correlation found in LocoTrack, or, from another perspective, as a local-to-local point-cloud registration. A crucial distinction is that point cloud matching is performed in world space rather than camera space, a formulation that simplifies trajectory complexity, as illustrated in Figure 1.", "questions": "I have a question regarding the substantial performance gap between TAPIP-3D and its competitors in LSFOdyssey; further explanation would be beneficial. Additionally, it's somewhat perplexing to observe that SpatialTracker's performance in LSFOdyssey drops when using ground truth depth compared to MegaSAM predictions.", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "**Strength**\n- The paper is well-written, well-structured, and easy to understand.\n\n- The proposed idea of matching the 3D structure of a query point to its neighbors in world space, rather than camera space, is well-motivated. I believe this approach will become a standard in future 3D point tracking research.\n\n- An interesting result is that TAPIP-3D can outperform CoTracker3 in 2D tracking, especially when ground truth depth is provided.\n\n**Major Weaknesses**\n- As mentioned in the strengths, it's intriguing that TAPIP surpasses CoTracker3 in 2D tracking when ground truth depth is available. I am curious if this performance trend holds true for the TAPIP-3D dataset as well.\n\n- A key suggestion is to evaluate performance using sparse depth. In real-world scenarios, obtaining dense depth from sensors is almost impossible; most setups yield sparse depth. For instance, the Waymo dataset used in DriveTrack includes sparse depth ground truth from LiDAR. It would be valuable to investigate TAPIP-3D's performance with such sparse depth. One option would be to apply depth completion techniques to the sparse depth and then assess performance, which would make the paper more comprehensive and address the practical demand for using TAPIP-3D with real-world sensor data.\n\n**Minor Weaknesses**\n- Before Line 143, the notation 'T' is used to denote the number of frames in a video. However, in Lines 143-144 and Line 176, it is used to represent the size of a local window. This dual usage can mislead readers, and I recommend introducing a distinct notation for the local window size.\n\n- It appears the paper exclusively uses APD with a fixed threshold. It would be beneficial to see the depth-adaptive APD, as introduced in TAP-Vid-3D, also evaluated.\n\n- It would be helpful to include an intuition regarding the failure cases of DELTA, as depicted in Figure 4, within the main text. Specifically, why do the predicted points diverge significantly from the MegaSAM depth prediction? Does this stem from depth prediction failures or 2D tracking errors? If it's a depth error, why does such drifting from MegaSAM occur? In my opinion, simply unprojecting accurate 2D tracking with MegaSAM depth should not lead to such errors.\n\n**Overall, I strongly support this paper. I identify two main novelties:**\n\n- **Minor**: The finding that performing 3D point tracking in world space is advantageous.\n\n- **Major**: The technical novelty of establishing correspondence using 3D geometry features. In previous 3D point tracking works such as Seurat (CVPR'25), most showed marginal improvement over baselines that simply unprojected 2D tracking using depth. I believe the primary reason for this limited improvement was an over-reliance on 2D features, which prevented them from fully leveraging the 3D setup. However, TAPIP-3D matches the geometry around one point to other sets of points, similar to how point cloud registration works, thus benefiting from accurate 3D geometry.\n\nThese ideas, I believe, will significantly advance the understanding of 3D point tracking task.", "was_revised": true, "final_justification": "I've read the authors' rebuttal and all the reviews carefully.\n\nWhile there are concerns about the model's robustness with noisy depth or pose, and its 2D tracking performance, I believe this criticism is mainly based on the assumption of a 2D monocular-video application, which is a problem that existing 2D trackers have already widely addressed. However, this assumption may not be valid when considering the future application of 3D point tracking. Point tracking is already widely used in robotics and could be used more to extract motion from video [1,2,3,4]. In such scenarios, it is easy to obtain near-ground-truth depth from sensors or multi-view setups, which TAPIP-3D can greatly benefit from, as demonstrated in the TAPVid-3D-ADT experiment from the rebuttal. This property would highly beneficial for learning precise manipulation. Considering this trend, I find this paper extremely useful for these applications and believe this model could become a solid standard for them. Therefore, I am raising my score and strongly recommend this paper for an oral or spotlight presentation.\n\n[1] Bharadhwaj et al., \"Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation\", ECCV'24\n[2] Kim et al., \"UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations\", CoRL'25\n[3] Li et al., \"HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation\", ICLR'25\n[4] Levy et al., \"P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies\", ICRA'25", "rebuttal": "### Response to Reviewer Sbrq\n\nThank you for your encouraging feedback and for recognizing both the conceptual ideas and technical contributions of our work.\n\n**1. Performance Comparison on TAPVid-3D with Ground Truth Depth**\n\nWe appreciate the reviewer’s interest in whether TAPIP-3D’s superior 2D performance when using groundtruth depth generalizes to the TAPVid-3D dataset. However, TAPVid-3D does not provide ground truth depth annotations. \n\nTo partially address this question, we investigated the Aria Digital Twin (ADT) sub-dataset of TAPVid3D benchmark, where all objects in the 3D scenes are modeled with provided rendered depth. As in the below table, our TAPIP3D still outperforms CoTracker3 in 2D tracking on ADT dataset with GT depth.\n\n| Methods               | AJ-2D | APD-2D | AJ-3D | APD-3D |\n|-----------------------|-------|--------|-------|--------|\n| Cotracker3 + GT        | 55.3  | 73.1   | 49.6  | 64.0   |\n| DELTA + GT        | 58.9  | 72.4   | 46.9  | 58.8   |\n| TAPIP3D-camera + GT    | **63.1**  | **79.5**   | **61.2**  | **76.5**   |\n\n**2. Evaluation with Sparse Depth on the Waymo Dataset**\n\n\nWe created a benchmark consisting of 50 sequences from the Waymo Open Dataset [3] (a subset of TAPVid-3D), paired with depth map annotations obtained by densifying sparse LiDAR depth using a simple nearest-neighbor approach, as proposed in the DriveTrack [4] paper. We report the performance of TAPIP3D-camera, DELTA, and CoTracker3 on this dataset.\n\n| Methods               | AJ-2D | APD-2D | AJ-3D | APD-3D |\n|-----------------------|-------|--------|-------|--------|\n| Cotracker3 + LiDAR       |  63.7     |   78.8     |   16.5     |   24.1    |\n| DELTA + LiDAR       |  **71.4**     |     **85.0**   |   21.0      | 30.8     |\n| TAPIP3D-camera + LiDAR   |  67.7     |     82.2  |   **21.8**    |   **31.6**     |\n\n\n**3. Explanation for Results on LSFOdyssey**\n\n (1) ***Substantial performance gap between TAPIP-3D and its competitors in LSFOdyssey*** \nThe LSFOdyssey dataset features challenging scenarios with many objects rotating and occluding each other, where knowing the accurate local 3d geometry help substantially.\n\n\n(2) ***Why does SpatialTracker's performance in LSFOdyssey drop when using ground truth depth compared to MegaSAM predictions?*** \nIn the provided GT depth, the background has large depth values, and SpatialTracker normalizes all valid depths to the range [0,1], which results in compressing the scene to a small region in SpatialTracker’s tri-planes. Our model is more robust to large depth ranges because it does not require depth values to fall within a fixed interval.\n\nWe further fixed the depth value issues of the background regions in SpatialTracker's code by setting it to zero (previously is a large value) to mark it as an invalid region and re-evaluate SpatialTrack using GT depth. The result on the LSFOdyssey benchmark is below. Although the AJ-3D performance of SpaTracker got improved, it still has a significant gap to our method.\n\n\n| Methods                | AJ-2D | APD-2D | AJ-3D | APD-3D |\n|------------------------|-------|--------|-------|--------|\n| SpaTracker + M-SaM     | 73.8  | 88.5   | 14.8  | 22.8   |\n| SpaTracker + GT        | 71.2  | 87.7   | 24.1  | 34.5   |\n| TAPIP3D-world + GT     | **78.5**  | **92.8**   | **72.2**  | **85.8**   |\n\n**4. Clarification on Evaluation Metrics (definition of APD)**\n\n***Use depth-adaptive APD as a tracking metric, as introduced in TAP-Vid-3D***\n\nThis is a misunderstanding. APD_3D stands for a depth-adaptive variant that scales the threshold based on object distance, precisely as introduced in TAPVid-3D [2]. So, we do use the metric you are suggesting. APD_2D is a fixed-threshold metric computed in pixel space, following [1].\n\nWe apologize for the confusion regarding APD metrics. We will include the formal definitions in the text to make this distinction clear. \n\n***The notation 'T' is used to denote the number of frames in a video and  to represent the size of a local window.***\n\nWe agree the notation could be clearer. In the final version, we will modify $T$ (temporal window length) to $W$ to avoid confusion with the total number of video frames.\n\n\n**5. DELTA Failure Cases (Figure 4)**\n\nDELTA is trained with depth augmentation. As a result, it sometimes “corrects” geometry even when it conflicts slightly with the input depth. By contrast, our model always respects the given geometry.\n\n\n[1] TAP-Vid: A Benchmark for Tracking Any Point in a Video. NeurIPS, 2022.\n\n\n[2] TAPVid-3D: A Benchmark for Tracking Any Point in 3D. NeurIPS, 2024.\n\n[3] Scalability in Perception for Autonomous Driving: Waymo Open Dataset. CVPR, 2020.\n\n\n[4] DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos. CVPR, 2024."}, {"review_id": "OJNuqStAS9", "reviewer": "Reviewer_pGLy", "summary": "The paper proposes a new approach for 3D point tracking using RGB or RGB-D data. It uses the off-the-shelf method MegaSAM to recover camera motions and depth maps.\nThen, the per-frame point cloud features are lifted and transformed to the world frame for better stability against camera motion.\nThe 3D points are tracked by a 3D Neighborhood-to-Neighborhood attention mechanism.\nExperiments are conducted on TAPVid3D, DexYCB-Pt, LSFOdyssey, and DynamicReplica datasets and compared with camera frame 3D tracking methods, SpatialTracker, DELTA, and 2D tracking method CoTracker3.", "questions": "(1) The average performance of DELTA in Table 1 is 22.3, which is significantly lower than Cotrackerv3 and the proposed approach. However, its results on separately reported datasets are pretty competitive. The reviewer is curious about the reason for this performance gap. Is it a mistake?\n\n(2) In Table 2' GT depth version, the performance of the proposed approach (camera frame) is slightly (<5%) outperformed by the strong 2D tracker CoTracker3 on 2D tracking metrics, but the performance of CoTracker3 is significantly worse on 3D metrics (>40%). Since both of these approaches use GT depth maps, the reviewer is curious about the reason for this performance gap.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n(1) The experiments are sufficient, which are conducted on several large datasets with thousands of sequences and cover various scenarios.\n(2) The strategy of using estimated camera motions to convert each frame's point cloud to the world frame for tracking is reasonable and effective, which can reduce the influence of camera motion.\n(3) The overall writing is clear, and the method is well explained.\n\n# Weaknesses\n\n(1) The reviewer's major concern is the advantage of the proposed 3D tracking architecture over existing approaches (DELTA and directly lifting 2D tracked points of CoTracker3 to 3D). Firstly, the idea of transforming the point cloud to the world frame for tracking is straightforward as long as the camera motion is estimated. Ideally, this approach can be applied to other methods, where a 3D tracker may need retraining on such a type of world frame data. Without transforming to the world frame strategy, the reviewer finds that the performance gain of the proposed approach over strong baseline methods DELTA and CoTracker3 is very marginal, as shown in Table 1 and Table 2 (MegaSAM version)\nSpecifically, the performance improvement in Table 1 is generally around 1%, which is not significant.\nIn Table 2's MegaSAM version, the performance improvement over state-of-the-art CoTracker3 on 3D tracking metrics is around 1% and 0.5% on LSFOdyssey and Dynamic Replica datasets, respectively.\n\n(2) Since the approach performs tracking on point clouds, its by nature sensitive to the quality of depth maps and camera motions: In Table 2's MegaSAM version with depth noise, the performance of the proposed method on 2D tracking metrics is significantly lower than 3D tracker DELTA using same depth maps on several metrcis.\n\n(3) The sensitivity of TAPIP3D-world version on camera motion estimation is not well studied and discussed.", "was_revised": true, "final_justification": "The authors have addressed the major concerns of the reviewer. The reviewer is glad to raise the score.", "rebuttal": "### Response to Reviewer pGLy\n**1. On the Contribution and Necessity of TAPIP3D’s Architecture**\n\nWe agree with the reviewer that the performance improvements of our camera space approach using estimated depth may appear modest in Table 1. However, we respectfully disagree with the suggestion that operating in a stabilized world frame is a straightforward extension of existing baselines or that our architectural contributions are incremental.\n\nMost prior methods, including DELTA and CoTracker3, are fundamentally designed around 2D-centric architectures. They operate in UVD space, which is 2D coordinates with depth appended, and rely on grid-based correlation volumes defined in pixel space. These components are not naturally extendable to the unordered and irregular structure of world-frame 3D data, where no global 2D grid exists.\n\nIn TAPIP3D, we address this challenge with our 3D Neighborhood-to-Neighborhood (N2N) attention, which enables local feature matching and aggregation directly in a shared world coordinate space. To the best of our knowledge, there is no simple way to adapt 2D-based models like DELTA or CoTracker3 to this setting without significant architectural changes.\n\nWe also note that the performance gap becomes substantially larger when higher-quality geometric input is available. In Table 2 (GT depth), TAPIP3D outperforms prior methods by over 30% on the 3D APD metric. Similarly, Table 3 (DexYCB) shows that our architecture achieves significantly better performance than prior baselines when using sensor depth, which is often available in robotic scenario. This demonstrates that TAPIP3D is well-suited to leverage accurate geometric inputs, which is an increasingly important advantage as 3D foundation models for depth and pose estimation continue to improve.\n\n**2. On the Sensitivity of the method to Depth Input**\n\nWhile the reviewer notes that our 2D tracking performance using MegaSaM depth appears weaker, our primary contribution lies in robust 3D tracking. As shown in Table 2, TAPIP3D consistently outperforms all baselines in 3D metrics (AJ_3D and APD_3D) under both MegaSaM and GT depth, demonstrating the method’s robustness to depth quality. Moreover, the 2D performance of TAPIP3D improves significantly when higher-quality depth is available (e.g., GT depth), indicating that the 2D results are naturally coupled with depth accuracy rather than a limitation of the model itself.\n\nIn many real-world scenarios such as robotics or sensor-based capture, accurate depth maps are readily available from RGB-D cameras. As evidenced by the sensor depth results in Table 3, TAPIP3D achieves state-of-the-art 3D tracking in such settings, which highlights a key advantage of our approach. To further validate this, we provide evaluation results on a subset of DriveTrack [1] using sparse LiDAR depth, densified via a simple nearest neighbor approach. Despite the presence of noise and significant discontinuities in the unprojected geometry, our model achieves robust 2D and 3D tracking:\n\n| Methods               | AJ-2D | APD-2D | AJ-3D | APD-3D |\n|-----------------------|-------|--------|-------|--------|\n| Cotracker3 + LiDAR       |  63.7     |   78.8     |   16.5     |   24.1    |\n| DELTA + LiDAR       |  **71.4**     |     **85.0**   |   21.0      | 30.8     |\n| TAPIP3D-camera + LiDAR   |  67.7     |     82.2  |   **21.8**    |   **31.6**     |\n\n\nWe additionally include results using UniDepth [a], a weaker depth estimation model with lower geometric and pose accuracy than MegaSAM. While the overall performance drops, our model still produces reasonable results.\n\n\n| Method                             | AJ-3D | APD-3D | AJ-2D | APD-2D | OA   |\n|----------------------------------|-------|--------|-------|--------|------|\n| **Using UniDepth**                |       |        |       |        |      |\n| CoTracker3 + UniDepth             | 19.2  | 29.9   | **61.4** | **80.0** | **86.2** |\n| SpatialTracker + UniDepth         | 7.6   | 13.5   | 53.8  | 72.6   | 81.3 |\n| DELTA + UniDepth                  | 18.8  | 28.4   | 61.0  | 77.5   | 81.0 |\n| **TAPIP3D-camera** + UniDepth | **20.6** | **30.3** | 55.3  | 77.1   | 80.2 |\n|                                  |       |        |       |        |      |\n| **Using MegaSaM**                 |       |        |       |        |      |\n| CoTracker3 + MegaSaM              | 20.0  | 30.2   | 61.4  | 80.0   | **86.2** |\n| SpatialTracker + MegaSaM          | 6.6   | 12.0   | 54.1  | 72.8   | 81.7 |\n| DELTA + MegaSaM                  | 18.0  | 27.2   | 61.3  | 77.3   | 81.0 |\n| **TAPIP3D-camera** + MegaSaM | **20.6** | **30.3** | 55.3  | 77.1   | 80.2 |\n| **TAPIP3D-world** + MegaSaM  | **20.6** | 30.2   | **56.1** | **77.7** | 78.6 |\n|                                  |       |        |       |        |      |\n| **Using GT Camera Poses and Depth**                 |       |        |       |        |      |\n| CoTracker3 + GT             | 27.4  | 38.0   | 61.4  | 80.0   | **86.2** |\n| SpatialTracker + GT        | 7.6   | 13.6   | 56.4  | 74.7   | 83.8 |\n| DELTA + GT                   | 27.4  | 37.7   | 65.6  | 80.5   | 83.8 |\n| **TAPIP3D-camera** + GT | 53.7  | 70.8   | 64.6  | 84.7   | 84.7 |\n| **TAPIP3D-world** + GT | **55.5** | **72.8** | **66.2** | **85.7** | 85.3 |\n\n[a] UniDepth: Universal Monocular Metric Depth Estimation. CVPR, 2024.\n\n**3. On the Sensitivity of the method to Camera Pose Estimation**\n\nAlthough TAPIP3D relies on estimated camera poses to transform points into a shared world frame, our training augmentations enable the model to generalize well under smooth camera motion. As a result, the model retains a degree of robustness even when the poses are inaccurate. As shown in Tables 1 and 2 of the paper, TAPIP3D-world and TAPIP3D-camera share the same model but use different pose inputs. TAPIP3D-camera effectively serves as a lower bound on the robustness of our approach, achieving competitive results even when camera extrinsics are highly unreliable, in which case the user can simply specify identity matrices as extrinsics.\n\n**4. Evaluation Clarifications**\n\n**3.1 Discrepancy on the reported  Average Performance of DELTA in Table 1**\n\nThis is a typo, thank you for pointing this out! We incorrectly computed the DELTA average in Table 1. The per-dataset values are correct, and we have updated this average to its correct value from 22.3 to 26.3 in the manuscript. Our TAPIP3D-world still outperforms DELTA for 1.1 APD$_{3D}$.\n\n**3.2 Why the proposed method has a much better 3D tracking performance that Cotracker3 despite worse 2D tracking performance, since both methods use GT depth?**\n\nThank you for the question. We would like to clarify a possible misunderstanding. In Table 2 (GT depth version), **TAPIP3D-camera actually outperforms CoTracker3 on both 2D and 3D tracking metrics**, not just in 3D.  \n- **LSFOdyssey**: *APD-3D*: 83.2 vs. 35.0; *APD-2D*: 91.2 vs. 88.0  \n- **Dynamic Replica**: *APD-3D*: 70.8 vs. 38.0; *APD-2D*: 84.7 vs. 80.0  \n\nThe improvement appears numerically larger for 3D metrics because when using the same depth map, even small 2D pixel offsets—particularly for points near object boundaries—can lead to significant depth variation (e.g., a point jumping from foreground to background). This amplifies the effect in 3D space, making gains in 2D accuracy translate into larger improvements in 3D error.\n\n[1] DriveTrack: A Benchmark for Long-Range Point Tracking in Real-World Videos. CVPR, 2024."}, {"review_id": "pugchNU7Jc", "reviewer": "Reviewer_6eUD", "summary": "This paper introduces TAPIP3D, a method for long-term point tracking in 3D by explicitly operating in a world-centric coordinate frame. Unlike prior approaches that append depth to 2D pixel coordinates (UVD), TAPIP3D lifts features directly into (XYZ) using estimated depth and camera poses. These features are organized into spatio-temporal 3D feature clouds, with camera motion compensated via extrinsics. To model correspondence across time, the authors propose a 3D Neighborhood-to-Neighborhood (N2N) Attention module, where query features attend to local neighborhoods in 3D, avoiding 2D grid biases. The model achieves strong performance on standard 3D tracking benchmarks.", "questions": "In addition to the concerns mentioned under weaknesses, the reviewer has the following additional questions:\n- How does the model behave when camera extrinsics are inaccurate? Is there any fallback or degradation analysis?  \n- How critical is the CoTracker initialization? What happens with random initialization?  \n- What if a different geometry estimator, such as MonST3R [4], was used instead of MegaSaM? An ablation would be useful here.\n\n--- \n## References\n[4] Zhang et al., Monst3r: A simple approach for estimating geometry in the presence of motion, ICLR 2025", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths  \n- Using camera extrinsics to eliminate camera motion is a simple yet effective idea for robust 3D tracking.  \n- The Neighborhood-to-Neighborhood Attention module is a meaningful architectural contribution, allowing the model to reason over 3D geometry directly rather than relying on fixed 2D neighborhoods or triplane projections. \n- The paper is clearly written, and figures are informative.  \n- Ablation studies are thorough and highlight where the improvements come from, especially in high-motion scenes.\n\n---\n\n## Weaknesses  \n- It is important to demonstrate the benefit of 3D tracking over 2D. The paper does not compare against strong 2D trackers like Track-On [1], CoTracker3 [2], or TAPNext [3], which is especially relevant in setups like Dynamic Replica where GT camera parameters are available. Comparisons on TAP-Vid 2D would also be informative.  \n- Despite the architectural similarity to CoTracker3, the occlusion accuracy is considerably lower (Table 1). Some explanation is needed.  \n- The paper claims 10 FPS runtime, but it is unclear whether this includes camera pose estimation. A breakdown of latency and memory use for each module would give a clearer picture of efficiency.\n\n---\n\n## Overall  \nThe paper shows clear improvements over existing 3D trackers, with a simple but effective idea of reasoning in world space. However, comparisons against strong 2D baselines are missing, which limits how broadly the results can be interpreted.\n\n---\n\n## References\n[1] Aydemir et al., Track-On: Transformer-based Online Point Tracking with Memory, ICLR 2025\n\n[2] Karaev et al., CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos, arXiv 2025\n\n[3] Zholus et al., TAPNext: Tracking Any Point (TAP) as Next Token Prediction, arXiv 2025", "was_revised": true, "final_justification": "The rebuttal addresses several important concerns, including the impact of different camera pose estimation methods and runtime analysis (as detailed in the discussion below). However, the conceptual advantages of 3D tracking over strong 2D baselines remain somewhat **underexplored**. It is encouraged that the final version of the paper further elaborates on this aspect, as understanding the trade-offs between 2D and 3D tracking is crucial for practical applications.\n\nGiven the productive discussion during the rebuttal period and the novel contributions presented in the paper, the score is increased, and acceptance is recommended.", "rebuttal": "### Response to Reviewer 6eUD\n**1. Lack of Comparisons with 2D Trackers (Track-On, CoTracker3, or BootsTAPNext) on Dynamic Replica**\n\n\nOur submission already includes quantitative comparisons with CoTracker3, including on the Dynamic Replica dataset, as in Table 2 of the paper. Per your request, we now include additional experiments with BootsTAPNext [a] on both Dynamic Replica and LSFOdyssey datasets as below. While BootsTapNext exhibits slightly better performance than CoTracker3 on certain metrics, our method still demonstrates superior results in both 2D and 3D metrics when ground truth depth is available.\n\n\n\n| Dataset           | Method              | AJ-2D | APD-2D | AJ-3D | APD-3D |\n|-------------------|---------------------|-------|--------|-------|--------|\n| LSFOdysey         | CoTracker3 + GT     | 75.0  | 88.0   | 28.4  | 35.0   |\n|                   | BootsTAPNext + GT   | 75.9  | 89.5   | 24.9  | 30.9   |\n|                   | TAPIP-world + GT    | **78.5**  | **92.8**   | **72.2**  | **85.8**   |\n| Dynamic Replica   | CoTracker3 + GT     | 61.4  | 80.0   | 27.4  | 38.0   |\n|                   | BootsTAPNext + GT   | 38.7  | 53.7   | 15.8  | 22.8   |\n|                   | TAPIP-world + GT    | **66.2**  | **85.7**   | **55.5**  | **72.8**   |\n\n> Note: BootsTAPNext's performance on Dynamic Replica is inferior in the table. We consider this as normal, as mentioned in Section 4.3 of [a]: \"We observe significant failure in long-term point tracking over the full length of videos greater than 150 frames.\", while sequences in Dynamic Replica have 300 frames.\n\nHowever, we wish to clarify that our goal is not 2D tracking. Accordingly, we focus evaluations on datasets that provide 3D point-track annotations, which allows us to compute 3D point tracking metrics. Note that [a] is public on arXiv only 1 month before the paper submission DDL. We will include more detailed comparison to BootsTAPNext [a] and Track-On in our final version.  \n\n[a] TAPNext: Tracking Any Point (TAP) as Next Token Prediction[J]. arXiv preprint arXiv:2504.05579, 2025.\n\n**2. Lower occlusion accuracy**\n\nWe acknowledge this limitation, which stems from TAPIP3D’s 3D-centric design. The model operates entirely in XYZ coordinates using KNN-based attention, without explicit knowledge of pixel-space proximity or depth ordering. As a result, occlusion reasoning relies solely on neighbor features and their spatial offsets, making it more challenging, especially when depth reconstruction is unreliable.\n\nThis issue is particularly noticeable in the DriveTrack dataset, where occlusions between distant vehicles lead to significant depth estimation errors. In some cases, depth of remote objects becomes visually fused due to the low resolution of the underlying depth model, which results in misleading geometry and unreliable neighbor associations. Incorporating auxiliary 2D cues, such as depth-difference maps used in SpaTracker and DELTA, could help address this, which we leave for future work.\n\n**3. Clarification on Latency and Memory Use**\n\nWe appreciate the reviewer’s request for a more detailed runtime and memory analysis. The reported 10 FPS refers to the tracking stage, assuming that depth and camera poses are available. It does not include the runtime of depth or pose estimation, as these are obtained from external sensors or off-the-shelf models (e.g., MegaSaM [9], CUT3R [10]) and thus can  vary dramatically. Importantly, our model is not trained with any specific estimation method and can be flexibly combined with any future advancements in depth or pose estimation.\n\nBelow, we provide a breakdown of latency during one pass of forward tracking on a sequence from the Dex-YCB dataset on a single L40S GPU. The forward pass consumes a total peak memory usage of 3.6GB, while feature encoding uses up to 2.8GB.\n\n\n| Module  | Feature Extraction |  Correlation Feature Encoding (KNN & N2N Attention) | Updating Transformer | Overall |\n|-----------------------|-----------------|-----------|------------------------|--------------|\n| Latency (s)           | 0.11          |    6.04    | 0.25                     | 6.4          |\n\n\nWe will include these in the revised manuscript.\n\n\n**4. On Question 2: Model Behavior and Fallback under Inaccurate Camera Extrinsics**\n\nAlthough TAPIP3D relies on estimated camera poses to transform points into a shared world frame, our training augmentations enable the model to generalize well under smooth camera motion. As a result, the model retains a degree of robustness even when the poses are inaccurate. As shown in Tables 1 and 2 of the paper, TAPIP3D-world and TAPIP3D-camera share the same model but use different pose inputs. TAPIP3D-camera effectively serves as a lower bound on the robustness of our approach, achieving competitive results even when camera extrinsics are highly unreliable, in which case the user can simply specify identity matrices as extrinsics.\n\n**5. Dependence on CoTracker Initialization**\n\nWe incorporated CoTracker3 initialization early in the project and observed that it accelerates convergence; however, it should have minimal impact on the final performance after full training. We will include an ablation study in the revised paper to substantiate this observation.\n\n**6. Ablation on Depth Estimator Choice**\n\nWe would like to clarify that our model is not specifically trained for any particular geometry estimation method. For evaluation, we selected MegaSAM due to its superior geometric accuracy and robust camera pose estimation.\n\nHere, we additionally include results using UniDepth [a], a weaker depth estimation model with lower geometric and pose accuracy than MegaSAM. While the overall performance drops, our model still produces reasonable results.\n\n\n| Method                             | AJ-3D | APD-3D | AJ-2D | APD-2D | OA   |\n|----------------------------------|-------|--------|-------|--------|------|\n| **Using UniDepth**                |       |        |       |        |      |\n| CoTracker3 + UniDepth             | 19.2  | 29.9   | **61.4** | **80.0** | **86.2** |\n| SpatialTracker + UniDepth         | 7.6   | 13.5   | 53.8  | 72.6   | 81.3 |\n| DELTA + UniDepth                  | 18.8  | 28.4   | 61.0  | 77.5   | 81.0 |\n| **TAPIP3D-camera** + UniDepth | **20.6** | **30.3** | 55.3  | 77.1   | 80.2 |\n|                                  |       |        |       |        |      |\n| **Using MegaSaM**                 |       |        |       |        |      |\n| CoTracker3 + MegaSaM              | 20.0  | 30.2   | 61.4  | 80.0   | **86.2** |\n| SpatialTracker + MegaSaM          | 6.6   | 12.0   | 54.1  | 72.8   | 81.7 |\n| DELTA + MegaSaM                  | 18.0  | 27.2   | 61.3  | 77.3   | 81.0 |\n| **TAPIP3D-camera** + MegaSaM | **20.6** | **30.3** | 55.3  | 77.1   | 80.2 |\n| **TAPIP3D-world** + MegaSaM  | **20.6** | 30.2   | **56.1** | **77.7** | 78.6 |\n|                                  |       |        |       |        |      |\n| **Using GT Camera Poses and Depth**                 |       |        |       |        |      |\n| CoTracker3 + GT             | 27.4  | 38.0   | 61.4  | 80.0   | **86.2** |\n| SpatialTracker + GT        | 7.6   | 13.6   | 56.4  | 74.7   | 83.8 |\n| DELTA + GT                   | 27.4  | 37.7   | 65.6  | 80.5   | 83.8 |\n| **TAPIP3D-camera** + GT | 53.7  | 70.8   | 64.6  | 84.7   | 84.7 |\n| **TAPIP3D-world** + GT | **55.5** | **72.8** | **66.2** | **85.7** | 85.3 |\n\n[a] UniDepth: Universal Monocular Metric Depth Estimation. CVPR, 2024."}]

Author rebuttal:

Meta-review: After the discussion phase, three reviewers recommend the paper, including one who does so very strongly.  The fourth	 reviewer still has reservations about the experimental validation but I believe the authors did a good job of addressing his concerns in their rebuttal. I therefore recommend acceptance.
Decision: Accept (poster)