Title: TAPIP3D: Tracking Any Point in Persistent 3D Geometry
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3d point tracking
Abstract: We introduce TAPIP3D,  a novel approach for long-term 3D point tracking in monocular RGB and RGB-D videos. TAPIP3D represents videos as camera-stabilized spatio-temporal feature clouds, leveraging depth and camera motion information to lift 2D video features into a 3D world space where camera movement is effectively canceled out. Within this stabilized 3D representation, TAPIP3D iteratively refines multi-frame motion estimates, enabling robust point tracking over long time horizons. To handle the irregular structure of 3D point distributions, we propose a 3D Neighborhood-to-Neighborhood (N2N) attention mechanism—a 3D-aware contextualization strategy that builds informative, spatially coherent feature neighborhoods to support precise trajectory estimation. Our 3D-centric formulation significantly improves performance over existing 3D point tracking methods and even surpasses state-of-the-art 2D pixel trackers in accuracy when reliable depth is available. The model supports inference in both camera-centric (unstabilized) and world-centric (stabilized) coordinates, with experiments showing that compensating for camera motion leads to substantial gains in tracking robustness. By replacing the conventional 2D square correlation windows used in prior 2D and 3D trackers with a spatially grounded 3D attention mechanism, TAPIP3D achieves strong and consistent results across multiple 3D point tracking benchmarks. Our code and trained checkpoints will be public.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TAPIP3D, a method for long-term point tracking in 3D by explicitly operating in a world-centric coordinate frame. Unlike prior approaches that append depth to 2D pixel coordinates (UVD), TAPIP3D lifts features directly into (XYZ) using estimated depth and camera poses. These features are organized into spatio-temporal 3D feature clouds, with camera motion compensated via extrinsics. To model correspondence across time, the authors propose a 3D Neighborhood-to-Neighborhood (N2N) Attention module, where query features attend to local neighborhoods in 3D, avoiding 2D grid biases. The model achieves strong performance on standard 3D tracking benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths  \n- Using camera extrinsics to eliminate camera motion is a simple yet effective idea for robust 3D tracking.  \n- The Neighborhood-to-Neighborhood Attention module is a meaningful architectural contribution, allowing the model to reason over 3D geometry directly rather than relying on fixed 2D neighborhoods or triplane projections. \n- The paper is clearly written, and figures are informative.  \n- Ablation studies are thorough and highlight where the improvements come from, especially in high-motion scenes.\n\n---\n\n## Weaknesses  \n- It is important to demonstrate the benefit of 3D tracking over 2D. The paper does not compare against strong 2D trackers like Track-On [1], CoTracker3 [2], or TAPNext [3], which is especially relevant in setups like Dynamic Replica where GT camera parameters are available. Comparisons on TAP-Vid 2D would also be informative.  \n- Despite the architectural similarity to CoTracker3, the occlusion accuracy is considerably lower (Table 1). Some explanation is needed.  \n- The paper claims 10 FPS runtime, but it is unclear whether this includes camera pose estimation. A breakdown of latency and memory use for each module would give a clearer picture of efficiency.\n\n---\n\n## Overall  \nThe paper shows clear improvements over existing 3D trackers, with a simple but effective idea of reasoning in world space. However, comparisons against strong 2D baselines are missing, which limits how broadly the results can be interpreted.\n\n---\n\n## References\n[1] Aydemir et al., Track-On: Transformer-based Online Point Tracking with Memory, ICLR 2025\n\n[2] Karaev et al., CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos, arXiv 2025\n\n[3] Zholus et al., TAPNext: Tracking Any Point (TAP) as Next Token Prediction, arXiv 2025\n\nQUESTIONS:\nIn addition to the concerns mentioned under weaknesses, the reviewer has the following additional questions:\n- How does the model behave when camera extrinsics are inaccurate? Is there any fallback or degradation analysis?  \n- How critical is the CoTracker initialization? What happens with random initialization?  \n- What if a different geometry estimator, such as MonST3R [4], was used instead of MegaSaM? An ablation would be useful here.\n\n--- \n## References\n[4] Zhang et al., Monst3r: A simple approach for estimating geometry in the presence of motion, ICLR 2025\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach for 3D point tracking using RGB or RGB-D data. It uses the off-the-shelf method MegaSAM to recover camera motions and depth maps.\nThen, the per-frame point cloud features are lifted and transformed to the world frame for better stability against camera motion.\nThe 3D points are tracked by a 3D Neighborhood-to-Neighborhood attention mechanism.\nExperiments are conducted on TAPVid3D, DexYCB-Pt, LSFOdyssey, and DynamicReplica datasets and compared with camera frame 3D tracking methods, SpatialTracker, DELTA, and 2D tracking method CoTracker3.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n(1) The experiments are sufficient, which are conducted on several large datasets with thousands of sequences and cover various scenarios.\n(2) The strategy of using estimated camera motions to convert each frame's point cloud to the world frame for tracking is reasonable and effective, which can reduce the influence of camera motion.\n(3) The overall writing is clear, and the method is well explained.\n\n# Weaknesses\n\n(1) The reviewer's major concern is the advantage of the proposed 3D tracking architecture over existing approaches (DELTA and directly lifting 2D tracked points of CoTracker3 to 3D). Firstly, the idea of transforming the point cloud to the world frame for tracking is straightforward as long as the camera motion is estimated. Ideally, this approach can be applied to other methods, where a 3D tracker may need retraining on such a type of world frame data. Without transforming to the world frame strategy, the reviewer finds that the performance gain of the proposed approach over strong baseline methods DELTA and CoTracker3 is very marginal, as shown in Table 1 and Table 2 (MegaSAM version)\nSpecifically, the performance improvement in Table 1 is generally around 1%, which is not significant.\nIn Table 2's MegaSAM version, the performance improvement over state-of-the-art CoTracker3 on 3D tracking metrics is around 1% and 0.5% on LSFOdyssey and Dynamic Replica datasets, respectively.\n\n(2) Since the approach performs tracking on point clouds, its by nature sensitive to the quality of depth maps and camera motions: In Table 2's MegaSAM version with depth noise, the performance of the proposed method on 2D tracking metrics is significantly lower than 3D tracker DELTA using same depth maps on several metrcis.\n\n(3) The sensitivity of TAPIP3D-world version on camera motion estimation is not well studied and discussed.\n\nQUESTIONS:\n(1) The average performance of DELTA in Table 1 is 22.3, which is significantly lower than Cotrackerv3 and the proposed approach. However, its results on separately reported datasets are pretty competitive. The reviewer is curious about the reason for this performance gap. Is it a mistake?\n\n(2) In Table 2' GT depth version, the performance of the proposed approach (camera frame) is slightly (<5%) outperformed by the strong 2D tracker CoTracker3 on 2D tracking metrics, but the performance of CoTracker3 is significantly worse on 3D metrics (>40%). Since both of these approaches use GT depth maps, the reviewer is curious about the reason for this performance gap.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of 3D point tracking. It begins by extracting depth and camera parameters from MegaSAM, with the flexibility to incorporate higher-quality depth data if available. Subsequently, it extracts hierarchical image features using CoTracker's feature backbone. These image features are then concatenated with point maps derived from MegaSAM. The core of the model involves a transformer-based mechanism that performs \"neighbors-to-neighbors\" attention. This approach can be conceptually understood as a 3D extension of the 4D local correlation found in LocoTrack, or, from another perspective, as a local-to-local point-cloud registration. A crucial distinction is that point cloud matching is performed in world space rather than camera space, a formulation that simplifies trajectory complexity, as illustrated in Figure 1.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The paper is well-written, well-structured, and easy to understand.\n\n- The proposed idea of matching the 3D structure of a query point to its neighbors in world space, rather than camera space, is well-motivated. I believe this approach will become a standard in future 3D point tracking research.\n\n- An interesting result is that TAPIP-3D can outperform CoTracker3 in 2D tracking, especially when ground truth depth is provided.\n\n**Major Weaknesses**\n- As mentioned in the strengths, it's intriguing that TAPIP surpasses CoTracker3 in 2D tracking when ground truth depth is available. I am curious if this performance trend holds true for the TAPIP-3D dataset as well.\n\n- A key suggestion is to evaluate performance using sparse depth. In real-world scenarios, obtaining dense depth from sensors is almost impossible; most setups yield sparse depth. For instance, the Waymo dataset used in DriveTrack includes sparse depth ground truth from LiDAR. It would be valuable to investigate TAPIP-3D's performance with such sparse depth. One option would be to apply depth completion techniques to the sparse depth and then assess performance, which would make the paper more comprehensive and address the practical demand for using TAPIP-3D with real-world sensor data.\n\n**Minor Weaknesses**\n- Before Line 143, the notation 'T' is used to denote the number of frames in a video. However, in Lines 143-144 and Line 176, it is used to represent the size of a local window. This dual usage can mislead readers, and I recommend introducing a distinct notation for the local window size.\n\n- It appears the paper exclusively uses APD with a fixed threshold. It would be beneficial to see the depth-adaptive APD, as introduced in TAP-Vid-3D, also evaluated.\n\n- It would be helpful to include an intuition regarding the failure cases of DELTA, as depicted in Figure 4, within the main text. Specifically, why do the predicted points diverge significantly from the MegaSAM depth prediction? Does this stem from depth prediction failures or 2D tracking errors? If it's a depth error, why does such drifting from MegaSAM occur? In my opinion, simply unprojecting accurate 2D tracking with MegaSAM depth should not lead to such errors.\n\n**Overall, I strongly support this paper. I identify two main novelties:**\n\n- **Minor**: The finding that performing 3D point tracking in world space is advantageous.\n\n- **Major**: The technical novelty of establishing correspondence using 3D geometry features. In previous 3D point tracking works such as Seurat (CVPR'25), most showed marginal improvement over baselines that simply unprojected 2D tracking using depth. I believe the primary reason for this limited improvement was an over-reliance on 2D features, which prevented them from fully leveraging the 3D setup. However, TAPIP-3D matches the geometry around one point to other sets of points, similar to how point cloud registration works, thus benefiting from accurate 3D geometry.\n\nThese ideas, I believe, will significantly advance the understanding of 3D point tracking task.\n\nQUESTIONS:\nI have a question regarding the substantial performance gap between TAPIP-3D and its competitors in LSFOdyssey; further explanation would be beneficial. Additionally, it's somewhat perplexing to observe that SpatialTracker's performance in LSFOdyssey drops when using ground truth depth compared to MegaSAM predictions.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper transforms the 3D tracking problem from a frame-to-frame 2D problem into a frame-to-model 3D tracking approach. This method effectively decouples object motion and camera motion in the scene to simplify the problem. It employs a novel neighborhood attention mechanism instead of traditional matching methods to optimize the entire 3D-based tracking approach. Through extensive qualitative and quantitative analysis on datasets, each designed component significantly enhances the overall system while maintaining good real-time performance.\n\nSTRENGTHS AND WEAKNESSES:\nPaper Strengths:\n\n1. The paper achieves more efficient optimization by decoupling the representation of tracking processing from camera motion, while simplifying the complexity of the target function in high-dimensional spaces.\n\n2. The paper enhances tracking consistency in 3D space through cross-attention in 3D neighborhoods.\n\n3. The proposed system exhibits certain robustness to occlusions and can benefit from the quality of depth information.\n\nMajor Weaknesses:\n\n1. The paper mentions that systems like NeRF incur significant computational overhead but does not provide a detailed comparison. To my knowledge, 4DGS [1] can achieve real-time performance at 82 FPS with good results.\n\n2. Regarding robustness, the paper does not explicitly investigate the impact of blurry input images on the overall system. It only explores the system’s usability under different depth conditions and low-light scenarios.\n\n[1] G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, “4D Gaussian splatting for real-time dynamic scene rendering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2024, pp. 20310–20320.\n\nQUESTIONS:\nHave you investigated the impact of varying degrees of input image blur on the system? For example, motion blur caused by camera or object movement, or severe blur due to defocus.\n\nMost of the qualitative results focus on objects with significant motion in the foreground. Have you explored whether robust 3D tracking can be achieved for objects at a far distance with significant blur, such as people in the audience seats in a football field?\n\nLIMITATIONS:\nNo, I believe that, beyond the illumination flickering and certain depth map dependency discussed by the authors, this work also has the limitation of struggling to achieve accurate trackig under severe motion blur. Additionally, depth maps estimated from monocular inputs are affected by blur, leading to erroneous edge estimation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}