DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO
Authors: Tuan Duc Ngo
Peiye Zhuang
Evangelos Kalogerakis
Chuang Gan
Sergey Tulyakov
Hsin-Ying Lee
Chaoyang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Computer Vision
Point Tracking
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Capture dense, long-range, 3D trajectories from casual videos in a feed-forward manner.

Abstract: Tracking dense 3D motion from monocular videos remains challenging, particularly when aiming for pixel-level precision over long sequences. We introduce DELTA, a novel method that efficiently tracks every pixel in 3D space, enabling accurate motion estimation across entire videos. Our approach leverages a joint global-local attention mechanism for reduced-resolution tracking, followed by a transformer-based upsampler to achieve high-resolution predictions. Unlike existing methods, which are limited by computational inefficiency or sparse tracking, DELTA delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy. Furthermore, we explore the impact of depth representation on tracking performance and identify log-depth as the optimal choice. Extensive experiments demonstrate the superiority of DELTA on multiple benchmarks, achieving new state-of-the-art results in both 2D and 3D dense tracking tasks. Our method provides a robust solution for applications requiring fine-grained, long-term motion tracking in 3D space.

Published as a conference paper at ICLR 2025
DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO
Tuan Duc Ngo" UMass Amherst
Peiye Zhuang Snap Inc.
Chuang Gan
Evangelos Kalogerakis UMass Amherst & TU Crete
UMass Amherst
Sergey Tulyakov Snap Inc.
Hsin- Hsin-Ying Lee Snap Inc.
Chaoyang Wang Snap Inc.
wew.gggtthh... github. .ttthb...o/ELTT. io/DELTA/
Track every pixel from menocular videos
b) Consistent in 3D space
c) SoTA performance and speed
Ours
TAP-V1450
faster
APDse
18
faster
Spatial Fracker
SecneTracker
10x Iaster
130
200
500
1100
2300
densoly tracking (200K pixels), on an A100.
Figure 1: DELTA is dense 3D tracking approach that (a) tracks every pixel from monocular video, (b) provides consistent trajectories 3D space, and (c) achieves state-of-the-an accuracy on 3D tracking benchmarks while being significantly faster than previous methods in the dense setting.
ABSTRACT
Tracking dense 3D motion from monocular videos remains challenging, partic- ularly when aiming for pixel-level precision long sequences. We introduce DELTA, novel method that efficiently tracks every pixel in 3D space, enabling accurate motion estimation across entire videos. Our approach leverages a joint global-local attention mechanism for reduced-ressolution tracking, followed by a transformer-based upsampler achieve high-resolution predictions. Unlike exist- ing methods, which are limited by computational inefficiency or sparse tracking, DELTA delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy. Furthermore, impact of depth representation on tracking performance and identify log-depth as the optimal choice. Extensive experiments demonstrate the superiority of DELTA on multiple benchmarks, achieving new state-of-the-art results in both 2D and 3D dense tracking tasks. Our method provides robust solution for applications requiring fine-graine -rainrdi,,,, long-term motion tracking in 3D space.
explore the
INTRODUCTION
Accurately estimating motion and determining point correspondences in dynamic 3D environments is a longstanding challenge in computer vision. In this work, we aim to achieve dense 3D tracking
Work done during internship Snap Inc.

---

Published as a conference paper at ICLR 2025
Method
Dense
3D
Long-term
Feed-forward
RAFT (Teed & Deng, 2020)
TAPIR (Doersch et al., 2023)
CoTracker (Karaev et al., 2023)
A
SpatialTracker (Xiao et al., 2024)
A
Scene SceneTracker (Wang et al., 2024a)
A
DOT (Le Moing al., 2024)
OmniMotion (Wang et al., 2023a)
DELTA (Ours)
Table 1: Comparison of different types of motion estimation methods. A denotes that the method is technically applicable dense tracking but will be extremely time-consuming.
by establishing correspondences for every pixel from a given monocular video. 3D tracking pro- vides richer insights into object trajectories, depth, and scene interactions than 2D tracking, while dense tracking captures subtle, fine-grained motions often missed by sparse methods. The task is particularly challenging due to the need simultaneously address ill-posed 3D-to-2D projections, occlusions, camera motion, and dynamic scene changes.
The ultimate goal of tracking is to ensure both dense coverage and long-ferm consistency. Early ef- forts focused on predicting dense motion for adjacent frames or short-term sequences using optical flow (1lg et al., 2017; Sun al., 2018: 8; Teed & Deng, 2020; Xu al., 2022; Dong et al., 2023; Huang et al., 2022) and scene flow (Vogel et al., 2015; Liu et al., 2019; Yang & Ramanan, 2020), but these contrast, point-tracking methods (Doer-
approaches usually struggle to capture long-term motion. sch et al., 2022; Harley et al., 2022; Doersch et al., 2023; Li et al., 2024b) built correspondences over hundreds of frames but are limited to sparse pixels. Recently, hybrid approaches have emerged that attempt to combine both paradigms, yet they either rely on per-frame optical flow predictions and lack strong temporal correlation (Le Moing et al., 2024), or adopt suboptimal attention designs and cannot perform dense tracking efficiently (Karaev et al., 2023). More, advancements in depth esti- mation (Bhat et al., 2023; Piccinelli et al., 2024) have allowed for lifting 2D tracking to 3D (Wang al., 2024a; Xiao et al., 2024). but these pipelines remain computationally prohibitive for dense tracking due to cross-track attention. We summarize the characteristic of these methods in Table 1.
In this paper, introduce DELTA, Dense Efficient Long-range 3D Tracking for Any video. To knowledge, the first method capable of efficiently tracking every pixel in 3D space over hundreds of frames. We achieve efficient dense tracking using a coarse-to-fine strategy, starting with coarse tracking via a spatio-temporal attention mechanism at reduced resolution, followed by an attention- based upsampler for resolutoon predictions. Our key design choices include:
An efficient spatial attention architecture that captures both global and local spatial struc- tures of the dense tracks, with low computational complexity, enabling end-to-end learning for dense tracking.
An attention- based upsampler, carefully designed provide high-resolution, accurate tracking with sharp motion boundaries.
A comprehensive empirical analysis various depth representations, showing that the log- depth representation yields the best 3D tracking performance
These designs enable DELTA to capture hundreds of thousands 3D trajectories in long video sequences within single forward pass, completing the process in under two minutes for 100 frames-over 8x faster than the fastest existing methods, as shown in figure 1. DELTA is extensively evaluated on both 2D and 3D dense tracking tasks, achieving -f the-art results the CVO (Wu et al., 2023; Le Moing et al., 2024) and Kubric3D (Greff et al., 2022) datasets both with more than 10% improvement in AJ and APDan- Additionally, it performs competitively conventional 3DD point tracking benchmarks, including TAP-Vid3D (Koppula et al., 2024) and LSFOdyssey (Wang al., 2024a; Zheng et al., 2023).
RELATED WORK
Optical Flow estimates motion by providing dense pixel-wise correspondences between consecu- tive frames. Early variational approaches (Mémin & Pérez, 1998; Horn & Schunck, 1980; Brox
2

---

Published as a conference paper at ICLR 2025
et al., 2004) struggled with robustness in complex scenes with rapid motion, occlusions, and large displacements. The introduction of CNN-based methods (Ilg et al., 2017; Ranjan & Black, 2017: Xu et al., 2017; Sun et al., 2018) improved motion estimation between adjacent frames. RAFT (Teed & Deng, 2020) marked a breakthrough by leveraging 4D correlation volumes for all pairs of pixels. Follow-up works advanced this by incorporating transformers for tokenizing 4D correlation vol- umes (Huang et al., 2022), adopting global motion feature aggregation to improve prediction in oc- cluded regions (Jiang et al., 2021), and framing optical flow as a matching problem with correlation softmax operations (Xu et al., 2022). While some efforts propose to apply optical flow to long-tern sequences with multi-frame optical flow (Teed & Deng, 2020; Godet et al., 2021; Shi et al., 2023) or integration of point-tracking techniques (Le Moing et al., 2024; Cho et al., 2024a), they often suffer from drifting and occlusion challenges, limiting their reliability for long-term tracking.
Scene Flow generalizes optical flow into 3D, estimating dense 3D motion. One line of work uses RGB-D data (Hadfield & Bowden, 2011; Hornacek et al., 2014; Quiroga et al., 2014; Teed & Deng, 2021b), while others estimates 3D motion from point clouds (Liu et al., 2019; Wang et al., 2020; Gu et al., 2019; Niemeyer et al., 2019). Recent methods have improved robustness by using rigid motion priors, either explicitly (Teed & Deng, Ib) or implicitly (Yang & Ramanan, 2021). Nonetheless, integrating scene flow methods for long sequences is exder-explored.
Point Tracking estimates global motion trajectories in videos. Particle Video (Sand & Teller, 2008) introduced particle trajectories for long-range video motion. TAP- Vid (Doersch et al., 2022) pro- vided comprchensive benchmark to evaluate point tracking and TAPNet, bascline that predicts tracking locations using correlation features. PIPs (Harley et al., 2022) revisited the concept of particle video and proposed feedforward network that updates motion iteratively poral windows, but ignored spatial context with independent point tracking and struggle with occlu- sion. Subsequent efforts addressed these limitations by relaxing the fixed-length window to variable lengths (Doersch et al., 2023) and jointly tracking multiple points and strengthening correlations be- tween tracking points with temporal attention for temporal smoothness and spatial attention (Karaev et al., 2023). Recent approaches like SceneTrracke (Wang et al., 2024a) and SpatialTracker (Xiao et al., 2024) extend point tracking to 3D by incorporating depth information, but remain inefficient for dense tracking due to computationally expensive cross-track attention. Our model builds on the strengths of these methods, but scales to full-resolution tracking.
fixed tem-
Tracking by Reconstructing estimates long-range motion by reconstructing a deformation field. OmniMotion (Wang et al., 2023b) optimizes a NeRF (Mildenhall et al., 2020) representation with bijective deformation field (Dinh et al., 2016), then extracts 2D trajectories using this bijective map- ping, but suffers from instability and requires hours to optimize. Recent work with DINOv2 (Oquab et al., 2023) uses its superior semantic features to establish long-range correspondences, either with an improved invertible deformation field (Song et al., 2024) in self-supervised manner (Tu- manyan et al., 2024). While these approaches can produce dense motion trajectories, they require per-video optimization, which is computationally expensive, and their performance on tracking benchmarks lags behind data-driven tracking methods.
We are the first feed-forward approach that performs dense 3D tracking efficiently from a long-term video. Table 1 provides a brief comparison of existing methods alongside our approach.
3 METHOD
Problem setup. We propose a method to track every pixel of a video in 3D space. Specifically, our method takes an RGB-D video as input, where the RGB frames are denoted as E T*xxxWxx with T, H, and W representing the temporal and spatial resolution of the video, and the depth maps D E R Xx* are obtained from an off-the-shelf monocular depth estimation method. Our method then estimates dense, occlusion-awan
3D trajectories P Tx x4, where each 4D slice, Pt,,. = (ut, Vt. de, ot), represents the tracking result for a pixel located at (u, v) in the first frame as it moves to its corresponding 3D position in the t-th frame. Specifically, (u4, 01) are the pixel coordinates in the f-th frame, d, is the depth estimate, and o E {0, 1) is the visibility prediction.
3.1 PRELIMINARY: POINT TRACKING
Our method is inspired by recent advances in 2D point tracking, most notably CoTracker (Karaev et al., 2023), which uses transformer architecture that takes as input set of trajectories within
3

---

Published as a conference paper at ICLR 2025
Trameformer
Local window of I'I frames
Feature Extraction
Track Initializatii lizatiin
Sec.
Donse trccss
Spattal Aftecttin
Temppraa Allention
Pinal donse tracks
Fine-remilttin deave tracks
Upsampler
Exl7
Tx
Fx W)
Sec. 3.3
Anrativel spdats
18
Figure 2: Overview of DELTA. DELTA takes RGB-D videos as input and achieves efficient dense 3D tracking using a coarse-to- to-finet-nneeeeee strategy, beginning with coarse tracking through a spatio-tempont temporal attention mechanism at reduced resolution (Sec. 3.1, 3.2), followed by crention-baseee upsampler for high-resolution predictions (Sec. 3.3).
fixed temporal window and iteratively predicts the position offsets and visibilities of points based on features extracted around their current locations. It is extended by SceneTrracer Tracker (Wang et al., 2024a) and SpatialTracker (Xiao et al., 2024) to include 3D-aware features for 3D point tracking.
Specifically, the transformer processes a set of initial trajectories, denoted as (P:). where is the trajectory index, and P. Ph. Plr], with P = (ut. of. di, of) being the 3D location and visibility of the point associated with the i-th trajectory at the f-th frame. The initial values (ut, of, di, of) are typically initialized as (ui, vi, di, 1), assuming that each point starts from the same location in the first frame and is visible at the beginning. We iteratively repeat the following: Extract token features. Each input trajectory represented by a list of tokens G' [G], G22, ... Cipl, each token G encodes position, visibility, appearance and correlation of the trajectory at 1-th frame:
=Fi Dj, (a x=)] + Ppos (xx) + Yeimm (t),
where each term represents:
Track features Fi represents the appearance of the point to be tracked. It is initialized by sampling from the feature map at the starting location of the trajectory in the first frame, and will be updated by the transformer network.
Correlation features C₁ are computed by comparing track features to image features around the current estimated track location, similar to previous optical flow and point tracking methods (Teed & Deng, 2020; Harley et al., 2022; Karaev al., 2023). Additionally, we follow LocoTrack (Cho et al., 2024b) by including local 4D correlation, which utilizes all all-pair correspondences to establish more precise and bidirectional correspondences, enhancing robustness against ambiguities.
Depth correlation D is calculated as the difference between the current estimated depth and the depth queried from the depth map around the estimated track location.
Spacetime positions "poos and Ttime are the positional embedding of the input position æi
(uf, vř, d:) and time f, respectively.
also beneficial to separately encode the relative displacement of the points by computing (x=] - zi) where y represents positional embedding.
Relative displacement.
Iteratively apply transformer. The trajectory tokens will then be iteratively updated by applying a transformer Q. Each iteration computes updates for point positions and track features, i.e.
{Aaj), (DF = (((
Visibility of predicted only in the last iterative step when the accurate location has been estimated.
Spatial-temporal transformer architecture. The architecture consists of temporal attention (self- attending within the same track) and spatial attention (cross-track within the same frame).
Limitations in dense tracking settings. By default, CoTracker is trained and tested in sparse tracking settings, where the number of tracks is kept low (< 103). In dense settings, where the total number of tracks is N H x W (the order of 10--10") the spatial attention becomes bottleneck due to limitations w.r.t. its computational cost and spatial granularity explained below.

---

Published as a conference paper at ICLR 2025
Sparze
Firsal madte
Global Spatial Altention
Self Attention
Cross Artentiu
Crose Altentiin
Jxy
Train Infor: N
Ancker ducks
CaTracker's Spatial Artention
Firnel
Closs Attennltoon
Cross Attoniio
Preecte
Ausettion
TK(N + 2M) TK* KT TEN, K,
Train & Infer: 2TKN + +TK KT N
Train: TK(N + 2M) +MM++T*--- KT TLN, where N N
Wio Spotial Attn
+ Attn with virtual tokens (back-ant-furl)
Attn with virtual tokens (1-way)
Arto with virtual tokens
Groundtruth
Local Atti
Spatial atfention architectures. Top: Illustration of different spatial attention architec- tures. Compared to prior methods, our proposed architecture @ incorporates both global and local spatial attention and can be efficiently learned using patch-by-patch strategy. Bottom: Long-term optical flows predicted with different spatial attention designs. We find that both global and local attention are crucial for improving tracking accuracy, as highlighted by the red circles. Additionally, computationally efficient global attention design using anchor tracks (i.e., @ W/o Local Attn) CoTracker version @
Figure 3:
achieves similar accuracy to the more computationally-inee
Computational complexiry. As shown in Fig. 3, applying spatial self-attention across all to- kens in a frame (1) results a computational cost of T making impractical for dense tracking. To reduce complexity, CoTracker introduces virtual track tokens which are conceptually similar to learnable tokens introduced by DETR (Carion al., 2020). performs cross-attention back- and-forth between trajectory tokens and small number of virtual tracks, and self-attention is only applied within the virtual tracks (2). This reduces the computational cost to 27 KNN TT? KT", assuming that the number of virtual tracks K < N. However, this reduction is still not enough for end-to-end tracking be divided into disjoint groups and perform attention within each group separately, this strategy is sub-optimal due to the interaction between tokens from different groups are ignored.
every pixel in a high-resolution video. Although in practice, pixels could
Spatial granularity. The use of virtual tracks assumes small number of tokens and thus re- duces spatial attention granularity, limiting the capacity to represent fine spatial details for dense tracking. Increasing virtual tracks improves accuracy but negates the complexity reduction.
Method overview. To further reduce computation cost without sacrificing accuracy, we adopt strategy commonly used in optical flow (Teed & Deng, 2020); performing complex computations the reduced spatial resolution, followed by lighter layers for upsampling. As shown Fig. we first run dense tracking on 1/,22 of the original resolution, reducing the computation cost by 1//2. tracks are upsampled to full spatial resolution. the following section,
Then the use the notation N (H w)/r² to denote the number of dense tracks at reduced resolution.
In Sec 3.2, we first discuss our design for reduced-resolltion tracking with a new spatial attention architecture, which maintains linear complexity w.r.t. number of tracks while providing finer spatial granularity compared to CoTracker. More importantly, the new architecture can be learned end-to- end for pixel-wise dense tracking without test-train resolution discrepancies. Next in Sec 3.3 introduce a new transformer-based upsampler that effectively predicts high-res tracking. Finally in See 3.4, delve into details of depth representation that is crucial for accurate tracking in 3D.
3.2 JOINT GLOBAL AND LOCAL SPATIAL ATTENTION FOR EFFICIENT DENSE TRACKING
We design global-local spatial attention mechanism based three key criteria: (1) effi- ciency for both training and testing in dense settings, (2) ability to capture global motion across the image, and (3) ability capture fine-grained motion details in the neighboring region for each track.
5

---

Published as a conference paper at ICLR 2025
Platurr
eayp
weighte
CNN Upsampler
Cross Attention
neiphhars
Fixevtao Tx (8 W]
Ratrive
Attention Upsampler
Course
rup Fisara
Ix ( Wir)
Figure 4: Attention-based upsample module. Left: We apply multiple blocks of local cross- attention to learn the upsampling weights for each pixel in the fine resolution. Right: The red circles highlight regions in the long-term flow maps where our attention-based upsampler produces more accurate predictions compared to RAFT's convolution-based upsampler.
Global attention with sparse anchor tracks. The spatial attention architecture from previ- ous works, i.e. CoTracker (Karaev et al., 2023), which has linear complexity w.r.t. the number of tracks, can perform inference at reduced resolution. However, training remains computationally expensive, exceeding the 80GB GPU memory capacity even when tracking videos with 96 128 reduced resolution. To improve training efficiency, we employ the following two strategies.
First, patchwise training strategy employed to reduce computation in training. At each iteration, we randomly crop small patches of size h' x w' = N' from the reduced resolution image, then perform dense tracking and obtain supervision within these patches. One issue of patchwise training is that it only computes spatial attention within a local patch without considering the rest region of the first frame. To mitigate this limitation, we augment the patch by introducing a sparse set of M anchor tracks (M s 10 < N), with starting positions uniformly sampled across the first frame.
The second strategy involves computing virtual tracks by cross-attending only to the anchor tracks, instead of attending to all tracks in CoTracker As illustrated in @ of Fig. 3, this approach reduces the cross-attention cost for computing virtual tracks from TKN' to T TKM.. Consequently, the total cost for the new global attention becomes TK + 2M) + TKK + KT2, approximately halving the original cost of 2TK'" + TK² + KT2, assuming Т, K, M < N'. As shown in Table 6b, this reduction in computation has minimal impact tracking accuracy. An additional advantage of learning virtual tracks from anchor tracks is that the same set of anchor tracks is used during both patchwise training and testing the full image, eliminating any train-test resolution mismatch.
Dense local attention. To capture fine-grained representations of local relations among dense tracks, we apply self-attention within very small spatial patches containing pixels, prior to cross-attending to the virtual tracks. Since self-attention is applied only within these small patches, this approach adds a marginal complexity of OTTNL) during inference, where L < N is the number of tracks per patch. Experiments (Table 6b) show that incorporating dense local attention significantly improves dense tracking accuracy.
In summary, our joint Global-Local spatial attention has roughly the same computational cost CoTracker's global-only attention but (i) captures both global motion and fined-grained spatial relations, (ii) enables end-to end-to-end training, and (ii) achieves significantly better performance in dense tracking (see Table 2, Table 3 and the qualitative results in our supplementary).
3.3 TICH-RESSOLUTION TRACK UPSAMPLER
Given the dense tracks extracted reduced spatial resolution of H x W. the next step to upsample them to the full resolution H x W. In the context of optical flow, a common upsampling approach expresses the flow for each fine-resolution pixel as a convex combination of its nearest neighbor flows estimated in a coarse resolution (Teed & Deng, 2020). The weights for the combi- nation are learned via a convnet. In contrast, we propose an attention-based upsampling mechanism that more effectively captures the correlation between each fine-resolution pixel and its neighbors at the coarse resolution. We demonstrate the efficacy in our experiments (see Figure 4 and Table 6c).
Starting from the first input frame of the window (for clarity, we omit the frame subscript t in this section), the frame is initially processed through lightweight convolutional backbone

---

Published as a conference paper at ICLR 2025
# xD, where D is the channel dimension. This coarse feature map is then upsampled using a convolutional decoder to produce a fine-resolution feature map. Fiin E R "xxWX Each fine-resolution pixel (u,z), with the feature vector yine E RIID, cross-attends to a X k neighborhood centered on its corresponding coarse location (u', v²) in the coarse resolution, where n' u/r and r' = e/r, using subpixel accuracy for u' and v'. The neighboring coarse features is defined as the set (F (1( TAXX (((x))x Specifically, the fine-resolution feature map is extracted through cross-attention operation:
to extract coarse resolution feature map, Fcoarse
jo=
Fime = a(u, u) v v({Fraarse (14,2)) 15=1)
(3)
where the cross-attention scores a(u, u) are computed as:
a(u, v) = softmar ((F(())))
fine
k( k[[F Foarse }==1) + m 1|''', l'u'',
( (
of )
(4)
and q(-). k-),, v-)) are linear transformations for the queries, keys, and values respectively, The term added the above dot product represents a static, non-learned spatial bias inspired by Alibi (Press et al., 2022). In our case, we bias the query-key attention scores between pixels with penalty proportional to their distance between their positions (L1 distance in our implementation). We refine the fine-resolution feature map by applying a series of 7 multi-head, Alibi-modified cross-attention blocks, as described in Eq. 3. Finally, we use a MLP to predict the weight map W = MMP(FFFFrrt finee) where W E # x (xx*). This allows us to compute the high-resolution tracking by taking a weighted average of the cosrse-reoolution tracks using the predicted weight map. We found that more temporally consistent results are produced when the weights are estimated once for the first frame of the time window, and then the same weights are reused for the rest of the frames.
3.4 DELVING DEEPER INTO DEPTH REPRESENTATION
Prior works in 3D tracking have primarily focused exploring different designs of 3D features, such as depth correlation features (Wang et al., 2024a) and triplane features (Xiao et al., 2024). Our experiments reveal that the choice of depth representation, which is a previously overlooked factor, has a much more significant impact. In previous works, 3D features were typically computed in Euclidean space, with depth normalized to fixed range, and the network predicted the difference in normalized depth. We find that alternative depth representations, such as inverse depth 1/d and log depth log(d), improve 3D accuracy, with log depth offering the greatest boost (Table 6a).
This improvement can be intuitively explained: Euclidean depth evenly distributes granularity along the depth axis, which suboptimal since objects of interest are typically closer. Inverse or log depth enhances precision for nearby regions, where visual-based depth estimation methods tend to be more reliable, while tolerating higher uncertainty for distant areas. This reasoning also underlies why monocular depth estimation methods are often trained to output either inverse depth or log depth (Eigen et al., 2014; Wang et al., 2019; Ranftl et al., 2022).
More critically, switching the network output from Ad, de di to A log(de) log(d.) (og(d))= = log(d//d1), and similarly adjusting the depth correlation feature to log depth corre- lation feature, improves robustness to imperfections in input depth maps. The depth change ratio de/d1, being scale-invariant, effectively decouples the network from the arbitrary scale of the input depth maps. This ratio also aligns with the concept of optical expansion (Swanston & Gogel, 1986; Schrater et al., 2001). where objects appear larger as they approach the camera. Thus, estimating depth change ratios directly from visual features makes the network less dependent on depth map accuracy, a strategy similarly used in scene flow estimation (Yang & Ramanan, 2020).
4 EXPERIMENTS
4.1
IMPLEMENTATION DETAILS
Training data. We leverage the Kubric simulator (Greff et al., 2022) to generate 5,632 training RGB-D videos and 143 testing videos, featuring falling rigid objects against diverse backgrounds. Dense trajectories are annotated for every pixel in the first, middle, and last frames of each video. To augment the training set, we apply random geometric and color augmentations to the images and introduce noise to the depth maps.
Training loss. We supervise the model using both the low-res and the upsampled predictions.
The total loss is defined AzaL2D + >depth Edepth Auisith Cuisibh where Lan and Ledepth
7

---

Published as a conference paper at ICLR 2025
CVO-Clean(7 frames)
CVO-Final(7 frames)
CVO-Extended( frames)
Methods
EPRJ (atl/vis/occ)
InUt
(ollvvisoc))
loUt
EPEJ (all/vis/occ)
loUt
RAFT (Teed & Deng, 2020)
2.48/ /
57.6
2.3/1..777.500
56.7
21.80/
15.4 33,4
65.0
MFT (Neoral et al., 2024)
2.91 1.39 79.93
19.4
3,16/ 1.56 10.3
19.5
21.40/ 9920//920/// 41.8
37.6
TAPIR (Docrsch ct al., 2023)
3.80 1.49 14.7
73.5
4.19/
1,86 15.3
72.4
19.8
42.5
68.4
CoTrucker (Karaey al., 2023)
1.51/0.8 /4.57
75.5
/,99 /4
75.3
/ 7.70
70.4
DOT (Le Moing al., 2024)
/0.72//../43
80.4
/ 3.99
80,4
4.9
7.17
71.1
SceneTracker (Wang et al., 2024a)
4.61 / / /9.62
11.5 / 8,49 17.0
SpatialTracker (Xiao et al., 2024)
1.84/ /
68.5
1.88 / 1.37 4,66
68.1
5.53 / /4118/ 8.68
66.6
DOT-3D
1.33/ /4.66
79.0
1.38 0.88 /4.00
78.8
5.20 3.58 7.95
70.9
Ours (2D)
0.89 /0.46 / /299
78.3
0.97 / 0.55 / 2,96
77.7
3.63 2.67 5.24
71.6
Ours (3D)
/0.51 / 2,97
78.7
1.03 0.61 / 3.03
78.3
3.67 / 2.64 5.30
70.1
Table 2: Long-range optical flow results
CVO (Wu et al., 2023; Le Moing al., 2024).
LSFOdyssey APDao f
Methods
Kubric-3D (24 frames)
AJt
OAt
Methods
Time
Alt
APD2a
OAt
SpatialTracker Scene Scankerucker
5.7
9,9
84.0
SputialTracker
42.7
51.6
96.5
9mins
57.7
SceneTrracker
65.5
Smins
Ours
29.4
39.6
84.4
DOT-3D
72.3
77.5
88.7
0.15mins
Ours!
50.1
69.7
83.9
Ours
81.4
88.6
96.6
0.5mins
Table 4: 3D tracking results on the LSFOdyssey benchmark. $ denotes models trained with LS- FOdyssey training set.
Table 3: Dense 3D tracking results on the Kubric3D dataset.
the LI losses comparing the predicted 2D coordinates and inverse depth with the ground truth, and Luisib is the binary cross entropy loss for visibility prediction. We empirically set weightings A2d. Adepth: Aeisis to be 100.0, 1.0, 0.
Training details. Training details are included in the appendix.
4.2 COMPARISON TO PRIOR WORKS
Baselines. We evaluate our method against prior optical flow and point tracking methods. Partieu- larly, we perform close comparison against DOT (Le Moing et al., 2024), a recent SoTA method designed for dense 2D tracking. We implemented a 3D extension of DOT, named DOT-3D, where incorporate depth map input into its optical flow module and add a head output log (d /1).
Benchmark datasets. We evaluate the performance of our approach across multiple tracking sce- narios, including long-range 2D optical flow, dense 3D tracking, and 3D point tracking benchmarks. Long-range 2D optical flow: We use the CVO (Wu et al., 2023) test set, which originally in- cludes two subsets: CVO-Clean and CVO-Final, the latter incorporating motion blur. Each split contains approximately 500 videos with 7 frames captured at 60 FPS. Following the comparison in DOT(Le Moing et al., 2024), we introduce an additional split, CVO-Fxtended, which includes 500 videos of 48 frames rendered at 24 FPS. All videos in the CVO dataset are annotated with dense, long-range 2D optical flow and occlusion masks.
Dense 3D tracking: We use our generated Kubric test split with 143 videos, each with 24 frames.
* 3D point tracking: We use two benchmarks: (1) TAP-Vid3D includes videos from 3 datasets with different scenarios: DriveTrack (Balasingam et al., 2024), PStudio (Joo al., 2017), and Aria (Pan et al., 2023) with total 4569 videos for evaluation, where the number of frames varies from 25 to 300 per video. (2) LSFOdyssey contains 90 40-frame videos derived from the PointOdyssey dataset (Zheng et al., 2023). Both datasets provide sparse trajectories and occlusion annotations.
Metrics. For long- longerange optical flow benchmark, we follow Le Moing et al. (2024) and report the end-point-error (EPE) between the predicted flows and groundtruth flows for both visible, occluded points and the intersection over union (IoU) between predicted and ground-truth occluded regions in visibility masks. For dense 3D tracking and 3D point tracking benchmarks, we follow Koppula et al. (2024) and report APDsD (< dawg) which measures the average percent of points within 6, error threshold, occlusion accuracy OA measures the accuracy of visibility prediction and the average Jaccard (AJ) which evaluates both occlusion and position accuracy.
8

---

Published as a conference paper at ICLR 2025
Aria APDaD
DriveTrack FOA AJt APDaD OAt AJI APDaD 1 OAA AJf APDaD OAt
PStudio
Average
Methods
AJt
TAPIR' + COLMAP 7.1
11.9
72.6
8.9
14.7
80.4
6.1
10.7
75.2 7.4
12.4
76.1
CoTracker! COLMAP
8.0
12.3
11.6
11.7
19.1
81.7
8.1
13.5
77.2
9.3
15.0
79.1
BootsTAPIR + COLMAP 9.1
14.5
78.6 11.8
18.6
83.8
6.9
11.6
81.8 9.3
14.9
81,4
CoTracker' UniDepth 13.0
20.9
84.9.2.55
19,9
80.1
6.2
13.5
67.8 10.6
18.1
77.6
TAPTR + UniDepth 15.77
24.2
87.8
12.4
19.1
84.8
7.3
13.5
84.3 11.8
18.9
85.6
LocoTrack' UniDepth 15.1
24.0
83.5
13.0
19.8
82.8
7.2
13.1
80.1 11.8
19.0
82.3
SpatialTrackkr UniDepth 13.6
20.9
90.5
8.3
14.5
82.8
8.0
15.0
75.8 10.0
16.8
83.0
TceneTracker UniDepth
23.1
6.8
12.7
14.2
DOT-3D + UniDepth
13.8
22.1
85.5
11.8
17.9
82.3
5.3
52.5 9.6
15.1
73.4
w
Ours UniDepth 16.6
"
24.4
86.8
14.6
85.8
8.2
15.0
76.4
13.1
20.6
83.0
Table 5: 3D tracking results on the TAP-Vid3D Benchmark. We report the 3D average jaccard (AJ), average 3D position accuracy (APDaD), and occlusion accuracy (OA) across datasets Aria, DriveTrack, and PStudio using UniDepth and ZoeDepth for depth estimation." t denotes using depth to lift 2D tracks to 3D tracks. We re-evaluated SpatialTrackkr and SceneTracker using their publicly available code and checkpoints, following the same inference procedure as our method. We note that the results differ slightly from the numbers reported in the VAdDD paper.
Depth Repr.
Network Output
TAP-Vid3D (Avg.)
Global
Atin.
EPF.
(Estended)
OAT
Upsample Method
CVO (Exenda))
0A
Alt
APDao
531/4444799
de d
9.0
15.0
8.01/ 13.91
10.0/4884/ 18.1
65.7
69.0
KNN
534/4317798
66.9
log(d)
1/d.-
logldk/d.) /d,)
13.1
20.6
CaTracker
Ouns
3.73/ 2.76//777
0.72/2.7//55 5.44
5.34/
70.1
70.0
CowUUp
AtentoaUpp
Alientionipp + Albi
4,27/3.09
367/264/5.0
3.75/2.73/5.55
70.3
70.1
70.2
(a) Depth representation
(b) Spatial attention design
(c) Upsampler design
Table 6: Ablation studies (a) different depth representations on Vid3D (b) different spatial
attention designs on the CVO (Extended) (c) different upsampler designs on CVO (Extended).
Long-range optical flow results. We first compare our method against baseline approaches on the dense 2D tracking task (see Tab. 2). This experiment isolates the evaluation from additional 3D features and supervision, making it a straightforward assessment of our proposed network architec- ture for handling dense per-pixel tracking. We find that our method significantly outperforms all previous approaches, including the recent SOTA method, DOT, in terms of positional accuracy. The improvement is particularly noticeable when visualizing the results (see Appendix), where the tra- jectories predicted by DOT tend to become unstable once the tracked pixel is occluded or moves out of view. This highlights the importance of maintaining temporal attention in the tracking network, feature absent in DOT.
Additionally, we compare both DOT and our method with and without 3D supervision. We find that the variants are nearly equivalent, although the quantitative performance slightly decreases for the 3D-supervised versions. We also observe that our visibility mask accuracy is on par with CoTracker, from which our method is derived, though it is marginally lower than DOT. These discrepancies could potentially be addressed by adjusting the weightings of different terms in the training loss.
Dense 3D Tracking results. We report the results of dense 3D tracking on the Kubric synthetic test set, where our approach significantly outperforms other methods in both accuracy and runtime. We visualize of 3D dense tracking results in figure 5. Compared to SceneTracker and SpatialTracker, our method excels at accurately predicting the locations of moving objects and preserving object shapes throughout the video. Please find more qualitative results in the supplementary.
3D point tracking benchmarks results. To further evaluate the generalizability of our approach on in-the-wild videos, we assess its performance the TAPVid-3D dataset using depth maps estimated by either UniDepth (Piccinelli et al., 2024) or ZoeDepth (Bhat et al., 2023). The results, summa- rized in Table 5, show that our method consistently outperforms previous approaches, including SpatialTracker, SceneTracker, and 3D-lifted versions of tfate-of-the. 2D tracking methods. Our approach demonstrates improvements across most sub-datasets, well in the overall average.
We also evaluate our approach on the LSFOdyssey dataset (Wang et al., 2024a), as shown in Tab. 4. In this benchmark, Scene trained specifically the same domain, outperforms both Spa-

---

Published as a conference paper at ICLR 2025
Supphi
Swiar
[2 5Q)
ce (20.60)
10%0
OR
Figure 5: Qualitative results of dense 3D tracking on in-the-wild videos between CoTracker + UniDepth, SceneTracker, SpatialTracker and our method. We densely track every pixel from the first frame of the video 3D space, the moving objects are highlighted as rainbow color. Our method accurately tracks the motion of foreground objects while maintaining stable backgrounds.
tialTracker and our model, which were trained on the Kubric dataset. To ensure fair comparison, we fine-tuned our model for just one epoch on the LSFOdysseg ssey training set and observed substantial performance improvements, surpassing SceneTracker.
4.3 ABLATION STUDY
Study on the 3D representation presented Tab. 6a. We find that representing depth using log depth significantly improves 3D tracking accuracy compared to using depth and inverse depth.
Study on the design of spatial attention is shown in Tab. 6b. We evaluated different spatial atten- tion variants, as illustrated in Fig.3, comparing approaches with and without global or local attention. We also compared two versions of global attention: the used in CoTracker, which cross attends virtual tracks back and forth (illustrated @ of Fig.3), and our proposed method of cross-attending virtual tracks with anchor tracks (illustrated in @ of Fig. 3). Our results show that both global and local attention are crucial, and our design of global attention achieves comparable accuracy to CoTracker while being more computationally efficient.
Study on the design of upsampler is reported in Tab. 6c. We compared our approach against upsampling methods using non-learnable operators (bilinear, nearest neighbor, and 3D K-nearest neighbor) and the CNN-based upsampler from RAFT (Teed & Deng, 2020). Our method noticeably outperforms all of these approaches.
CONCLUSION
We presented method that efficiently tracks every pixel of frame throughout video, demonstrat- ing state-of-the-an art accuracy in dense 2D/3D tracking while running significantly faster than existing 3D tracking methods. Despite these successes, our method shares some common limitations with previous point-tracking approaches due to its relatively short temporal processing windows. It may fail to track points that remain occluded for extended periods, and currently performs best with videos of fewer than a few hundred frames. Additionally, our 3D tracking performance is closely tied the accuracy and temporal consistency of the off-the-shelf monocular depth estimation. We anticipate that our method will benefit from recent rapid advancements in monocular depth estima- tion research (Hu et al., 2024).
Acknowledgements Evangelos Kalogerakis has received funding from the European Research Council (ERC) under the Horizon research and innovation programme (Grant agreement No. 101 12474)..
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arjun Balasingam, Joseph Chandler, Chenning Li, Zhoutong Zhang, and Hari Balakrishnan, Drive- track: A benchmark for long-range point tracking in real-world videos. In CVPR, 2024.
Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero- shot transfer by combining relative and metric depth, 2023. URL https: https://aaxxx
abs/ 22022 12288. 2, 9, 17
Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In ECCV, 2004. 2
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael J Black. A naturalistic open source movie for optical flow evaluation. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pp. 61 611-625. Springer, 2012. 18, 20
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 5
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR, 2017. 16, 17
Weirong Chen, Le Chen, Rui Wang, and Mare Pollefeys. Leap-vo: Long-term effective any point tracking for visual odometry. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19844-19853, 2024. 20, 21
Seokju Cho, Jiahui Huang, Seungryong Kim, and Joon- Young Lee. Flowtrack: Revisiting optical flow for long-range dense tracking. CVPR, 2024a.
Seokju Cho, Jiahui Huang. Jisu Nam, Honggyu An, Seungryong Kim, and Joon- Joon-Young Young Lee. Local all-pair correspondence for point tracking. arXiv preprint arXiv:2407.154000 2024b. 4, 16
Laurent Dinh, Jascha Narain Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. ArXiv, abs/1605, 60.008003, 2016. URL https:/ /att://api. /api..p..../
https:/app..eematccccchooo www.ttcschooolrrcccoooo
CorpusID: 8768364. 3
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-vid: A benchmark for tracking any point
video. NeurlPS, 2022. 2, 3, 16, 17
Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In ICCV, 2023. 2, 3, 8, 16
Carl Doersch, Pauline Luc, Yi Yang. Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Hey- ward, Ignacio Rocco, Ross Goroshin, João Carreira, and Andrew Zisserman. BootsTAP: Boot- strapped training for tracking-any-point Asian Conference on Computer Vision, 2024. 16
Qiaole Dong, Chenjie Cao, and Yanwei Fu. Rethinking optical flow from geometric matching consistent perspective. In CVPR, 2023. 2
David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using a multi-scale deep network. NeurlPS, 2014. 7
Pierre Godet, Alexandre Boulch, Aurélien Plyer, and Guy Le Besnerais. Starflow: A spatiotemporal recurrent cell for lightweight multi-frame optical flow estimation. In ICPR, 2021.
Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf, Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang, Tianhao Wu, Kwang Moo Yi. Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: a scalable dataset generator. In CVPR, 2022. 2,7
11

---

Published as a conference paper at ICLR 2025
Xiuye Gu, Yijie Wang, Chongruo Wu, Yong Jae Lee, and Panqu Wang. Hplflownet: Hierarchical permutohedral lattice flownet for scene flow estimation on large-scale point clouds. In CVPR, 2019. 3
Simon Hadfield and Richard Bowden. Kinecting the dots: Particle based scene flow from depth sensors. In ICCV, 2011.
Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. ECCV, 2022. 2, 4,
BK Horn and Schunck. Determining optical flow (artificial intelligence laboratory). Mas- sachusetts Institute of Technology, Cambridge, MA, 1980. 2
Michael Hornacek, Andrew Fitzgibbon, and Carsten Rother. Sphereflow: 6 dof scene flow from rgb-d pairs. In CVPR, 2014. 3
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-world world videos, 2024. URL htthtts::/// / org/ atwp/209....
10
Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In ECCV, 2022.
2,
Eddy llg. Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In CVPR, 2017. 2.3
Shihao Jiang, Dylan Campbell, Yao Lu, Hongdong Li, and Richard Hartley. Learning estimate hidden motions with global motion aggregation. In ICCV, 2021. 3
Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, et al. Panoptic studio: A massively multiview system for social interaction. TPAMI, 2017. 8
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. vrXiv:2307.07633 2023. 2, 3, 4, 6, 8, 16
Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In Pro- ceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1611- 1621, 2021 21
Skanda Koppula, Ignacio Rocco, Yi Yang. Joe Heyward, João Carreira, Andrew Zisserman, Gabriel Brostow, and Carl Doersch. :ad A benchmark for tracking any point in 3d, 2024. URL https:/ //tr/vixvvv//////////////////avv org/ alos/ 2407. 05921. 2, 8,
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. In CVPR, 2024. 2, 3, 8, 16, 17, 18
Alex X Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost To- bias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond pick-and-place place: Tackling robotic stacking of diverse shapes. In CoRL, 2021. 16, 17
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Feng Li, Tianhe Ren, Bohan Li, and Lei Zhang. Taptrv2: Attention-based position update improves tracking any point. arXiv preprint arXiv:2407. 16291, 2024a. 16
Hongyang Li, Hao Zhang, Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, and Lei Zhang. Taptr: Tracking any point with transformers detection. arXiv preprint arXiv:2403.13042 2024b. 2, 16
Xingyu Liu, Charles R Qi, and Leonidas J Guibas. Flownet3d: Learning scene flow in 3d point clouds. In CVPR, 2019. 3
Etienne Mémin and Patrick Pérez. Dense estimation and object- based segmentation the optical flow with robust techniques. TIP, 1998. 2
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Michal Neoral, Jonávs vSerých, and Jivr Matas. Mft: Long-term tracking of every pixel. In WACV, 2024. 8, 16
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. In ICCV, 2019.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, PaoYao Huang, Hu Xu, Vasu Sharma, Shang-Wae Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. 3
Xiaqing Pan, Nicholas Charron, Yongqian Yang. Scott Peters, Thomas Whelan, Chen Kong, Omkar Parkhi, Richard Newcombe, and Yuheng Carl Ren. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In ICCV, 2023. 8
Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. In CVPR, 2024. 2, 9, 17
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Dorkine-Hornung and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 2017. 16, 17
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022. 7
Julian Quiroga, Thomas Brox, Frédéric Devernay, and James Crowley, Dense semi-rigid scene flow estimation from rgbd images. In ECCV, 2014. 3
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. TPAMI, 2022.
Anurag Ranjan and Michael Black. Optical flow estimation using a spatial pyramid network. In CVPR, 2017. 3
Peter Sand and Seth Teller. Particle video: Long-range motion estimation using point trajectories. IJCV, 2008. 3
Paul R Schrater, David C Knill, and Eero P Simoncelli. Perceiving visual expansion without optic flow. Nature, 2001. 7
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi- frame optical flow estimation. ICCV, 2023. 3
Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations
applicationns, 2019. 17
Yunzhou Song. Jiahui Lei, Ziyun Wang. Lingjie Liu, and Kostas Daniilidis. thing everywhere fast and robustly. ArXiv, abs/2403.17931, 2024. URL https://api.
Track every-
semanticschole
2686918533
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of rgb-d slam systems. 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 573-580. IEEE, 2012. 20
13

---

Published as a conference paper at ICLR 2025
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR, 2018. 2,
Michael T Swanston and Walter C Gogel. Perceived size and motion in depth from optical expan- sion. Perception & psychophysics. 1986.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020. 2, 5, 6, 8, 10, 18, 20
Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 16569, 2021a. 21
Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In CVPR, 2021b.
Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. Advances in Neural Information Processing Systems, 36, 2024. 21
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self- supervised point tracking in single video, 2024. 3
Christoph Vogel, Konrad Schindler, and Stefan Roth. 3d scene flow estimation with a piecewise rigid scene model. IJCV, 2015. 2
Bo Wang. Jian Li, Yang Yu, Li Liu, Zhenping Sun, and Dewen Hu. Scenetracker: Long-term scene flow estimation network. arXiv preprint arXiv:2403. (0t::2243..992444 19924, 2024a. 2, 3, 4, 7, 8, 9, 16, 17
Chaoyang Wang, Simon Lucey, Federico Perazzi, and Oliver Wang. Web stereo video supervision for depth prediction from dynamic scenes. In 3DV, 2019.
Qianqian Wang, Yen-Yo Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. ICCV, 2023a. 2, 16
Qianqian Wang. Yen- Yu Chang, Ruojin Cai, Zhenggi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. In ICCV, 2023b. 3
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709 2024b. 20, 21
Yihan Wang, Lahay Lipson, and Jia Deng. Sea-raft: Simple, efficient, accurate raft for optical flow. In European Conference Computer Vision, pp. 36-54. Springer, 2025. 18, 20
Zirui Wang, Shuda Li, Henry Howard-Jenkins, Victor Prisacariu, and Min Chen. Flownet3d+4: Geometric losses for deep scene flow estimation. In WACV, 2020.
Guangyang Wu, Xiaohong Liu, Kunming Luo, Xi Liu, Qingqing Zheng, Shuaicheng Liu, Xinyang Jiang, Guangtao Zhai, and Wenyi Wang. Accflow: Backward accumulation for long-range optical flow. In ICCV, 2023. 2.8
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng. Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In CVPR, 2024. 2, 3, 4, 7, 8, 16, 17
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, and Dacheng Tao. Gmflow: Learning optical flow via global matching. In CVPR, 2022. 2,3
Jia Xu, René Ranftl, and Vladlen Koltun. Accurate optical flow via direct cost volume processing. In CVPR, 2017. 3
Gengshan Yang and Deva Ramanan. Upgrading optical flow to 3d scene flow through optical ex- pansion, In CVPR, 2020. 2,7
Gengshan Yang and Deva Ramanan. Learning to segment rigid motions from two frames. CVPR, 2021. 3
14

---

Published as a conference paper at ICLR 2025
Junyi Zhang, Charles Herrmann, Junhwa Hur, Varun Jampani, Trevor Darrell, Forrester Cole, De- qing Sun, and Ming- Hsuan Yang. Monst3r: A simple approach for estimating geometry in the presence of motion. arXiv preprint 241.:22410 2024. 20, 21
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T Freeman. Structure and motion from casual videos. In European Conference on Computer Vision, pp. 20-37. Springer, 2022. 21
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. Particlesfm: Exploiting dense point trajectories for localizing moving cameras in the wild, In European Conference Computer Vision, pp. 523-542. Springer, 2022. 21
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas Guibas. Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023. 2,
15

---

Published as a conference paper at ICLR 2025
Aria
PStudio AJT APDaD TOAT AJ$ APDaD
DriveTrack
Average
Methods
AJt APDaD
1
AJT APDap 1 TAA
TAPIR ZoeDepth
9.0
14.3
79.7
5.2
8.8
81,6
10.7
18.2
78.7
8.3
13.8
80.0
CoTracker! 7nceepphh
10.0
15.9
87.8
5.0
9.1
82.6
11.2
19.4
80.0
8.7
14.8
83.4
BootsTAPIR' ZeneDepth
9.9
16.3
86.5
5.4
9.2
85.3
11.3
19.0
82.7
8.8
14.8
84.8
TAPTR! 77neDepth 9.1
15.3
87.8
7.4
12.4
84.8
10.0
17.8
84.3
8.8
15.2
85.6
LocoTrack' ZoeDepph
8.9
15.1
83.5
7.5
12.3
82.8
9.7
17.1
80.1
8.7
14.8
82.1
SpatialTracker + ZocDepth
9.2
15.1
89.9
5.8
10.2
82.0
9.8
17.7
78.0
8.3
14.3
83.3
SceneTracker ZocDepth
15.1
5.6
16.3
12.3
Ours + Znneeeph
10.1
16.2
84.7
7.8
12.8
87.2
10.2
17.8
74.5
9.4
15.6
82.1
Table Additional 3D tracking results on the TAP-Vid3) Benchmark. We report the 3D average jaccard (AJ), average 3D position accuracy (APD3n), and occlusion accuracy (OA) across datasets Aria, DriveTrack, and PStdio using ZoeDepth for depth estimation." denotes using depth to lift 2D
tracks to 3D tracks.
Kineties
DAVIS APD2n
RGB-Stacking
Methods
AJ1
APD2n
OAt
AJf
OAT
AJ1
APD2D 0A1
TAP-Net (Doersch al., 2022)
38.5
54.4
80.6
33.0
48.6
78.8
54.6
68.3
87.7
MPT (Ncoral et al., 2024)
39.6
60.4
72.7
47.3
77.8
PIPs (Harley et al., 2022)
31.7
53.7
72.9
42.2
64.8
77.7
15.7
28.4
77.1
OmniMotion (Wang et al., 2023a)
46.4
62.7
85.3
69.5
82.5
90.3
TAPIR (Doersch et al., 2023)
49.6
64.2
85.0
56.2
70.0
86.5
54.2
69.8
84.4
CoTracker (Karaev al., 2023)
48.7
64.3
86.5
60.6
75.4
89.3
63.1
77.0
87.8
DOT (Le Moing et al., 2024)
48.4
63.8
85.2
60.1
74.5
89.0
77.1
87.7
93.3
BootsTAPIR (Doersch et al., 2024)
54.6
68.4
86.5
61.4
73.6
88.7
70.8
83.0
89.9
TAPTR (Li et al., 2024b)
49.0
64.4
85.2
63.0
76.1
91.1
TAPTRv2 (Li et al., 2024a)
49.7
64.2
85.7
63.5
75.9
91.4
LocoTrack (Cho et al., 2024b)
52.9
66.8
85.3
75.3
87.2
69.0
83.2
89.5
SpatialTracker (Xiao al., 2024)
50.1
65.9
86.9
61.1
76.3
89.5
63.5
77.6
88.2
Scene TeenkTracker (Wang et al., 2024a)
66.5
71.8
73.3
DOT-3D
48.1
63.7
85.9
61.2
75.3
88.1
76.3
86.6
92.1
Ours (2D)
50.3
63.5
83.2
64.2
77.3
87.8
73.4
82.4
89.6
Ours 49.5
63.3
82.2
62.7
76.7
88.2
74.2
83.5
90.0
Table 8: 2D Tracking Results the TAP-Vid Benchmark (Doersch al., 2022) (query-first mode). We report the average jaccard (AJ), average 2D position accuracy (APD2p). and occlusion accuracy (OA) on the Kineties (Carreira & Zisserman, 2017), DAVIS PoottTuseet et al., 2017) and RGB- Stacking (Lee et al., 2021) datasets.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Implementation details. We use the same backbone as Karaev al. (2023); Harley et al. (2022), which consists of 6 residual blocks, outputting feature maps with a dimension of 256. Unlike Co- Tracker, which extracts a single-scale feature map and applies pooling later in the correlation mod- ule, we directly generate pyramid of feature maps at scales 2, 4. and 8. We perform dense pixel tracking at a resolution of H/4 W/4 (with r 4). The transformer network 6 spatial and temporal attention blocks, utilizing 8 attention heads and 384 hidden channels. The number iteration step is set to 6. The number of anchor tracks is set to x 12 during training. In the patch-wise dense local attention, the high-resolution track upsampler, we use 9 neighbors (with λ. = 3) and 2 cross-attention blocks.
composed of
patch size of 6, resulting in L 36 tracks per patch. In
Training details. We first pretrain the model with 2D loss and visibility loss for 100k iterations, then train with the full loss for another 100k iterations. All stages are conducted on a machine with A 1100 GPUs. We use the AdamW optimizer and the batch size set to for each GPU. The learning
16

---

Published as a conference paper at ICLR 2025
DOT
Ours
0- 23
0 24
0 0 25
0 26
27
0 0 28
DOT
Ours
37
38
40
Figure 6: Comparison of long-range optical flow predictions: We predict optical flows from the first frame to subsequent frames the video. DOT (Le Moing et al., 2024), which lacks strong temporal correlation, suffers from noticeable "flickering" effect (green circle), particularly the boundaries between foreground and background objects. In contrast, our method ensures a smooth and consistent transition time, effectively reducing artifacts at object boundaries.
rate initialized to 10-1 and scheduled by a linear one cycle (Smith & Topin, 2019). During training. save the GPU memory consumption, we randomly sample patch of size N' 30 40 from the dense 3D trajectory map as supervision. The input video is resized to 384 x 512 in both training and testing. For in-the-wild wild video, we leverage ZoeDepth (Bhat al., 2023) and UniDepth (Piccinelli et al., 2024) to obtain video depth.
A.2 3D POINT TRACKING
We additionally report the performance on the TAPVid-3D dataset using depth maps estimated by ZoeDepth (Bhat et al., 2023) in Table 7.
A.3 2D POINT TRACKING
Sparse Tracking Setting. In the sparse tracking scenario, where there fewer than 10K points and they are sparsely distributed across the image, our model can seamlessly switch to a sparse mode by disabling the local attention in the Transformer and the Upsampler. This adjustment is made without requiring any changes to the overall architecture. This mode enables efficient evaluation sparse tracking benchmarks such as TAP- Vid (Doersch al., 2022) and TAP-Vid3D (Koppula al., 2024).
2D point tracking benchmark results. We also evaluate the performance of our method on 2D point tracking on the TAP- Vid dataset, containing videos from 3 datasets: DAVIS (Pont-Tuset al., 2017) with 30 in-the-wild videos, RGB- CGB-Sttcking (Carreira & Zisserman, 2017) with 50 synthetic sequences, and Kinetics (Lee et al., 2021) with 1 1144 real videos. The results are reported in Ta- ble 8. Our approach outperforms 3D tracking methods, including SpatialTracker (Xiao al., 2024), SceneTracke (Wang et al., 2024a), and DOT-3D, in most of the sub-datasets while having competi- tive performance with other SoTA approaches designed for sparse 2D tracking only.
17

---

Published as a conference paper at ICLR 2025
1m5
Dur (ilte-pp
0-+ n-
6-
RAST
9-
Figure 7: Comparison of long-range optical flow predictions on Sintel dataset between our approach (with CNN-based/attentioo atteee--teet upsampler) and SEA-RAFT. We predict optical flows from the first frame to subsequent frames of the video. While SEA-RAFT excels short-range flow prediction (first column), it fails to predict flow between far-away frames. In contrast, our approach, with an attention- based upsampler, achieves smooth and consistent predictions across frames, outperforming the baseline using convolutional upsampler.
CVO (Extended) EPE 1
OA f
W/o anchor tracks
4.50 3. 10/ 6.95
69.7 71.6
With anchor tracks
3.63 / 2.67 5.24
Table 9: Ablation the role of anchor tracks.
A.4 MORE RESULTS
Qualitative results of long-range 2D flows are visualized in figure 6. Thanks to the strong tem- poral consistency of our method, produces smooth predictions over time, effectively avoiding the 'flicker' artifacts that are commonly observed in per-frame optical flow predictions, such as those produced by DOT(Le Moing al., 2024).
Qualitative comparison of the design of upsampler on the Sintel dataset (Butler et al., 2012) visualized in figure Our iitention-based upsampler (last row) significantly outperforms the CNN-based upsampler from RAFT (Teed & Deng, 2020) (second row), with better detail and less artifacts. For reference, we include predictions from the state-of-the-art optical flow model SEA- RAFT (Wang al., 2025) (first row), which, despite being trained Sintel, excels short-range predictions (first column) but fails with long-range flows.
We also compare in-the-wild videos. The results are visualized in figure showing that our approach achieves better temporal smoothness and less artifact for long-range motion prediction.
rangerange flow predictions between SEA-RAFT, DOT, and our method on
18

---

Published as a conference paper at ICLR 2025
184
186
[-10 19
12
DOT
10
70
70 -0
12
10
10
POT
70
22
14-
14
Figure 8: Comparison of long-range optical flow predictions on in-the-wild videos between DOT, SEA-RAFT, and our approach.
Ablation on the anchor tracks is shown in Table 9, Removing the anchor tracks during training increases the EPE by nearly 1.0, meaning that the anchor tracks are important to avoid train-test mismatch.
A.5 FURTHER DISCUSSIONS ON 3D TRACKING PERFORMANCE
As shown Table 5 of main paper, baseline methods (2D tracking + depth estimation) significantly underperform compared to our end-to-end 3D tracking approach. Interestingly, better 2D tracking models do not always lead to better 3D tracking performance (e.g., BootsTAPIR versus CoTracker). This is because the performance of these baselines heavily relies on the quality and consssconni of the input video depth. However, frame-wise depth estimators like ZoeDepth and UniDepth lack scale-consistency (see webpage), where even small depth errors cause
19

---

Published as a conference paper at ICLR 2025
significant 3D location discrepancies.
Furthermore, 2D tracking depth estimation only predict motion for visible points within the viewpoint, restricting applications such as 3D/4D reconstruction, which require complete motion predictions, including occluded and out-of-frame regions.
These challenges underline the need for end-to-end 3D tracking models achieve reliable 3D performance. Our approach addresses these limitations and is also complementary to recent advances 2D point tracking, enabling us integrate their findings to further enhance both 2D and 3D tracking performance.
A.6 APPLICATION
Dynamic video pose estimation. DUSt3R (Wang et al., 2024b) introduced a paradigm for estimating static 3D scene geometry and camera poses from image sets. By training model on large-scale data, it predicts aligned 3D point maps for image pairs, followed by lightweight optimization obtain scale-consistent depth maps and camera poses. However, this approach fails on videos with dynamic objects. MonST3R (Zhang et al., 2024) extended DUSt3R to dynamic videos by fine-tuning the model on dynamic datasets. Our method aligns with this approach.
We first perform dense pixel tracking from the initial frame, producing dense 3D trajectories p((.) X For any destination frame v. the 3D points p(u,,) plu [p] IR HR
represent the 3D positions of all pixels from the initial frame in the camera coordinate system at frame v. This naturally combines the camera motion from frame 0 to v with any non-rigid scene motion over the interval. This setup aligns with the methodology of MonST3R (Zhang et al., 2024). We then apply the global alignment approach from Wang et al. (2024b); Zhang al. (2024) to estimate camera poses across dynamic video sequences.
In detail, uniformly sample keyframes with stride of 2 from the input video and densely track all pixels these frames. For each keyframe k, tracking is performed both forward and backward 8. Following Zhang et al. (2024), model the
within window [k w. k + w]. where pose estimation as an optimization task, where the learnable parameters include per-keyfrume depth maps, as well as per-keyframe intrinsie and extrinsic camera parameters, and optimize with gradient descent. The objective function combines alignment loss, temporal smoothness loss, and 2D flow loss. For the 2D flow loss, we calculate pseudo optical flow ground truth using our dense tracking approach rather than relying on an off off-the-shelf model (Wang et al., 2025; Teed & Deng, 2020). For non-keyframe frames, camera poses are interpolated from the two nearest neighboring keyframes.
We evaluate the pose estimation performance on Sintel (Butler et al., 2012) and TUM-dynamics (Sturm et al., 2012) datasets. On Sintel, we follow the evaluation protocol in Zhang et al. (2024); Chen et al. (2024), which excludes static and easy scene, remaining 14 test sequences. For TUM-dynamies, sample the first 90 frames with the temporal stride of 3. We report the Absolute Translation Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot), after applying Sim(3) Umeyama alignment on prediction to the ground truth. The results are reported in Table 10. Our method demonstrates comparable performance to state-of-the-art approaches specifically tailored for visual odometry and SLAM tasks. We visualize the camera pose and dynamic 3D scene reconstruction on casual videos in figure 9.
20

---

Published as a conference paper at ICLR 2025
Sintel RPE trans
TUM-dynamics
Methods
ATEJ
RPE rot
ATEL
RPE trans
RPE rot.
DROID-SLAM (Teed Deng, 2021a)
0.175
0.084
1.912
DPVO (Teed et al., 2024)
0.115
0.072
1.975
ParticleSIM (Zhao et al., 2022)
0.129
0,031
0.535
LEAP-VO (Chen al., 2024)
0.089
0.066
1.250
0.068
0.008
1.686
Robust-CVD (Kopf al., 2021)
0.360
0.154
3.443
0.153
0.026
3.528
CasualSLAM (Zhang al., 2022)
0.141
0.035
0.615
0.071
0.010
1.712
DUSI3R (Wang et al., 2024b)
0.417
0.250
5.796
0.083
0.017
3.567
MonST3R (Zhang et al., 2024)
0.108
0.042
0.732
0.063
0.009
1.217
Ours
0.172
0.060
0.553
0.052
0.007
1.343
Table 10: Pose estimation results on Sintel and TUM datasets. The upper group (first four rows) includes methods that estimate camera poses only, without reconstructing scene geometry while the lower group and our approach provide both camera poses and per-frame depth maps. Our method achieves competitive results compared other approaches specifically designed for visual odometry or SLAM tasks.
21

---

Published as a conference paper at ICLR 2025
Figure 9: Qualitative results of jointly depth and pose estimation on in-the-wild videos (first two rows) and Al-generated video (last row) of our approach.
22


Reviews:
[{"review_id": "WpzLEz2qiB", "reviewer": "Reviewer_G77y", "summary": "This paper presents a novel method for dense 3D tracking in monocular video. By introducing the anchor tracks and attention based upsampling methods, the proposed method significantly improves the inference speed by ~10x. By modifying the depth representation and supervision signal, the method significantluy improves the performance.", "questions": "1. Why the 2D version consistently better than 2D version in Table2? Is there any analysis on this unexpected results?\n2. How the key parameters such as M, r, N' and L are chosen?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method indeed remedy the efficiency issue of previous SOTA methods significantly.\n2. The experiments are comprehensive and convincing.", "weaknesses": "Some key concepts and design choices are not well explained. \n \n1. Why do we need certain anchor tracks? What if we sample some certain points in the dense tracks?\n2. The difference of sparse tracks and dense tracks are not well explained. For the comparisons in subfigure 2 and 3 in Fig.3, the dimension of sparse tracks and dense tracks are both T x N, then why it is called dense tracks? What is N' and L, the meaning of these two parameters are not well defined and explained.  \n3. If the depth is predicted from monocular video, the scale is not specified. How can we ensure their consistency across different frames?\n4. The attention upsampling method is quite related to the well known guided filter work. It is better to discuss the relationship with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pN0tSdt1fy", "reviewer": "Reviewer_n65b", "summary": "The paper proposes a model for tracking points in 3D densely, whose architecture is inspired by 2D sparse tracking CoTracker model, but with a number of modifications to 1) lift the tracks to 3D and 2) improve the model efficiency to allow for dense tracking with reasonable computational cost. \n\nFor lifting the tracks to 3D, the authors propose that the model takes RGB-D frames as input, and add depth tokens to the model's transformer framework (eq 1) to refine the depth predictions jointly with the 2D tracking predictions. This seems to be similar to the approach proposed in SceneTracker.\n\nFor tracking densely with reasonable computational cost, the authors adopt two different strategies. First, they perform most of the tracking prediction at a downsampled resolution (H/r, W/r) and propose a learnt cross-attention based upsampler to upsample the predictions to (H, W) resolution. Second, authors introduce some modifications to the spatial attention module proposed in CoTracker to increase efficiency. As in CoTracker, they perform cross-attention between virtual tracks and sparse tracks (here sampled on a uniform grid on the first frame and called \"anchor tracks\"), and then perform most of the computation by self-attention layers on the virtual tracks. However, they introduce a third tensor of \"dense tracks\", which they process by local attention and then fuse with the anchor tracks before decoding back the final refined dense tracks predictions by performing cross-attention with the virtual tracks.\n\nThe model is trained using a fully supervised setting by using synthetic videos generated with the Kubric simulator. The model is first pre-trained on the 2D tracking setting for 100k iterations, and then further trained on the 3D setting for an additional 100k iterations.\n\nTheir model is evaluated both for the 2D and 3D tracking tasks using standard benchmarks such as CVO, LSFOdyssey and TAPVid-3D. The results show the model obtains SOTA results on the CVO benchmark as well as in the TAPVid-3D benchmark. On the LSFOdyssey benchmark, the model trained on Kubric does not attain SOTA performance, but the one finetuned on LSFOdyseey does.\n\nThe authors conduct a series of ablation studies to show the impact of the design decisions regarding the depth respresentation, spatial attention module and upsampler.", "questions": "Could you please describe which elements in the model design are novel and which are borrowed from prior work?\n\nHave you tried evaluating your method on TAPVid-DAVIS (first)? Does it obtain SOTA results on 2D tracking compared to CoTracker and BootsTAPIR?\n\nPlease clarify how the dense tracks in Fig. 3 (3) relate to the explanations in Sec. 3 and in particular to the tokens G^i_t of eq. (1).\n\nPlease clarify how the proposed attention-based upsampling works and provide a corrected eq. (3) if possible. Please explain how the \\tau cross attention blocks from Fig. 4 operate. Are these multiple heads in parallel?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is one of the first papers addressing the TAP-3D problem with a feedforward model (not requiring test-time optimization), along with SceneTracker and SpatialTracker, which makes it a valuable contribution in this newly developing field.\n\nIn addition, the authors present quantitative results on both 2D tracking and 3D tracking which demonstrate the good performance of the model. The proposed model obtains SOTA results on the CVO benchmark as well as in the TAPVid-3D benchmark.\n\nThe visualizations included in the supplementary material help to further assess the quality of the 3D tracks produced by the model and compare it to the SceneTracker and SpatialTracker baselines. The results show a cleary superiority of TAPE3D over SceneTracker, and  slightly superior results compared to SpatialTracker. These observations are roughly in line to the reported quantitative results on TAPVid-3d.\n\nThe design choices proposed by the paper are thoroughly ablated in section 4.3 and Table 6, demonstrating the value of each of these design decisions.\n\nFinally, Fig. 1 shows that in the dense-tracking scenario, TAPE3D is at least 8x faster than the baselines, while obtaining superior performance on TAPVid-3d.", "weaknesses": "While the paper is a valuable contribution with good results, the model design seems to be mainly a combination of ideas from SceneTracker and CoTracker. The paper does not clearly state which ideas are novel, and which are borrowed from these previous methods. \n\nFurthermore, the experimental section only presents 2D tracking results on CVO, which is not the most widespread 2D tracking benchmark. Results on TAPVid-DAVIS would help in demonstrating the SOTA status of this model for 2D tracking. \n\nHowever, the main weakness of the method lies in that some of technical contributions are not presented with sufficient clarity or mathematical rigor. First, it is not clear how the dense tracks from Fig. 3, (3), are computed and how they connect to the tokens G^i_t of eq (1). In addition, the eq. (3) seems mathematically incorrect or at least too vague in notation and lacking explanation. It is not explained whether the functions q and k are linear layers and on what they are applied. Also, it's not clear what the softmax function is applied over. Finally, this is referred to a as cross-attention, but the values seems to be missing from the equation. While Figure 4 helps in understanding the general idea of the proposed attention-based upsampling, it is not exactly clear how to connect it to eq. (3) and therefore to fully understand the proposed module. \n\nRegarding the training procedure, it's not clear why the authors train on such small number of videos (5.6K) if these are synthetic videos that can be generated at a larger scale at ease. It's also not clear adding other data sources, such as PointOddysey would improve performance on the  TAPVid-3D benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J9gWxZfw5H", "reviewer": "Reviewer_EecY", "summary": "This paper presented a method, TAPE3D, that efficiently tracks every pixel of a frame throughout a video, demonstrating state-of-the-art accuracy in dense 2D/3D tracking while running significantly faster than existing 3D tracking methods.", "questions": "See weakness. If questions addressd, I will increase the rating.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. TAPE3D introduces the spatial-temporal transformer to extract more visual features and uses the Upsampler module to obtain high-resolution results. \n2. TAPE3D delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy.", "weaknesses": "1. The idea of using the spatial-temporal transformer is not new, such as [1][2][3].\n2. The authors are suggested to provide the compilation cost of each module to verify the efficiency of TAPE3D.\n3. Are all inference experiments testing on a same machine? The information of the machine including GPU and CPU are suggested to provide.\n\n[1] Hu M, Zhu X, Wang H, et al. Stdformer: Spatial-temporal motion transformer for multiple object tracking[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(11): 6571-6594.\n[2] Zhang T, Jiao Q, Zhang Q, et al. Exploring Multi-modal Spatial-Temporal Contexts for High-performance RGB-T Tracking[J]. IEEE Transactions on Image Processing, 2024.\n[3] Zhang T, Jin Z, Debattista K, et al. Enhancing visual tracking with a unified temporal Transformer framework[J]. IEEE Transactions on Intelligent Vehicles, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r4k3cCg3pT", "reviewer": "Reviewer_oj5m", "summary": "This paper proposes a method for dense 3D motion prediction, named TAPE3D. This paper finds that there is a redundant design in the CoTracker's global attention, and proposes their local-global attention, which reduces calculation costs to both accelerate the speed and reduce the resource requirement. After that, the paper proposes a transformer-based upsampler, which shows better performance than previous upsamplers, such as the ConvNet-based used in optical flow methods. This paper also finds that using log-depth as the depth representation is an optimal choice. Finally, TAPE3D achieves SoTA performance on both 2D/3D point tracking with the fastest speed.\n\nOverall, TAPE3D is an interesting work that contributes some insights to the field, and this must be acknowledged. However, these contributions are a little lack depth, and feel somewhat scattered, like a collection of bundled findings. Considering that there are some parts not clearly written and are difficult to understand, I'm giving 'marginally below the acceptance threshold' for now.", "questions": "For the questions, please refer to the Weaknesses part. If the author can address my concerns, I will raise my score.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper proposes a method, named TAPE3D, which not only obtains SoTA performance but also achieves the fastest speed.\n2. This paper proposes the local-global cross-attenton, which reduces the calculation consumption while achieving better performance.\n3. This paper proposes a transformer-based upsampler to upsample the sparse predictions to dense ones, achieving better performance than the ConvNet-based ones in previous optical flow methods.\n4. This paper finds that using log-depth as the depth representation can obtain better performance.", "weaknesses": "1. Although many contributions are provided, they are a little lack depth, and feel somewhat scattered, like a collection of bundled findings. (but we have to acknowledge the contributions' value)\n2. There are some parts that are not clearly written. In local-global attention, what N' (Fig 3) means, and why N' is different between training and evaluation? And it seems that there is no description of what M means (number of anchors if I understand correctly). \n3. It seems that the differences between the proposed Local-Global attention and the global one in CoTracker are\n    a) use anchor tracks to attend virtual tracks.\n    b) add local attention.\nAlthough a) can reduce the calculation consumption, b) needs additional calculation consumption. During evaluation, local-global even requires O(TKM + TLN^2) more calculation.\n4. In L290-L295, I'm confused why the virtual tracks should \"cross-attent to a local random patch of tracks\". What is \"local random patch of tracks\"? In Fig.3, it seems that the virtual points only attend to anchor tracks in cross-attention and themselves in self-attention.\n5. In Table.5, why BootsTAPIR+ZoeDepth is worse than CoTracker+ZoeDepth? As far as I know, BootsTAPTIR have much better performance than CoTracker in 2D point tracking, maybe more comparisons are required to show the reasonability (LocoTrack and TAPTRv2 for example).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)