DELTA: DENSE EFFICIENT LONG-RANGE 3D TRACKING FOR ANY VIDEO

Authors: Tuan Duc Ngo, Peiye Zhuang, Evangelos Kalogerakis, Chuang Gan, Sergey Tulyakov, Hsin-Ying Lee, Chaoyang Wang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Computer Vision, Point Tracking

Abstract:
Tracking dense 3D motion from monocular videos remains challenging, particularly when aiming for pixel-level precision over long sequences. We introduce DELTA, a novel method that efficiently tracks every pixel in 3D space, enabling accurate motion estimation across entire videos. Our approach leverages a joint global-local attention mechanism for reduced-resolution tracking, followed by a transformer-based upsampler to achieve high-resolution predictions. Unlike existing methods, which are limited by computational inefficiency or sparse tracking, DELTA delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy. Furthermore, we explore the impact of depth representation on tracking performance and identify log-depth as the optimal choice. Extensive experiments demonstrate the superiority of DELTA on multiple benchmarks, achieving new state-of-the-art results in both 2D and 3D dense tracking tasks. Our method provides a robust solution for applications requiring fine-grained, long-term motion tracking in 3D space.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729481459353

thread_type:
official_review

<thread_object>
creator: reviewer_oj5m

<content>
summary:
This paper proposes a method for dense 3D motion prediction, named TAPE3D. This paper finds that there is a redundant design in the CoTracker's global attention, and proposes their local-global attention, which reduces calculation costs to both accelerate the speed and reduce the resource requirement. After that, the paper proposes a transformer-based upsampler, which shows better performance than previous upsamplers, such as the ConvNet-based used in optical flow methods. This paper also finds that using log-depth as the depth representation is an optimal choice. Finally, TAPE3D achieves SoTA performance on both 2D/3D point tracking with the fastest speed.

Overall, TAPE3D is an interesting work that contributes some insights to the field, and this must be acknowledged. However, these contributions are a little lack depth, and feel somewhat scattered, like a collection of bundled findings. Considering that there are some parts not clearly written and are difficult to understand, I'm giving 'marginally below the acceptance threshold' for now.

strengths:
1. This paper proposes a method, named TAPE3D, which not only obtains SoTA performance but also achieves the fastest speed.
2. This paper proposes the local-global cross-attenton, which reduces the calculation consumption while achieving better performance.
3. This paper proposes a transformer-based upsampler to upsample the sparse predictions to dense ones, achieving better performance than the ConvNet-based ones in previous optical flow methods.
4. This paper finds that using log-depth as the depth representation can obtain better performance.

weaknesses:
1. Although many contributions are provided, they are a little lack depth, and feel somewhat scattered, like a collection of bundled findings. (but we have to acknowledge the contributions' value)
2. There are some parts that are not clearly written. In local-global attention, what N' (Fig 3) means, and why N' is different between training and evaluation? And it seems that there is no description of what M means (number of anchors if I understand correctly). 
3. It seems that the differences between the proposed Local-Global attention and the global one in CoTracker are
    a) use anchor tracks to attend virtual tracks.
    b) add local attention.
Although a) can reduce the calculation consumption, b) needs additional calculation consumption. During evaluation, local-global even requires O(TKM + TLN^2) more calculation.
4. In L290-L295, I'm confused why the virtual tracks should "cross-attent to a local random patch of tracks". What is "local random patch of tracks"? In Fig.3, it seems that the virtual points only attend to anchor tracks in cross-attention and themselves in self-attention.
5. In Table.5, why BootsTAPIR+ZoeDepth is worse than CoTracker+ZoeDepth? As far as I know, BootsTAPTIR have much better performance than CoTracker in 2D point tracking, maybe more comparisons are required to show the reasonability (LocoTrack and TAPTRv2 for example).

questions:
For the questions, please refer to the Weaknesses part. If the author can address my concerns, I will raise my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570463429

thread_type:
official_review

<thread_object>
creator: reviewer_eecy

<content>
summary:
This paper presented a method, TAPE3D, that efficiently tracks every pixel of a frame throughout a video, demonstrating state-of-the-art accuracy in dense 2D/3D tracking while running significantly faster than existing 3D tracking methods.

strengths:
1. TAPE3D introduces the spatial-temporal transformer to extract more visual features and uses the Upsampler module to obtain high-resolution results. 
2. TAPE3D delivers dense 3D tracking at scale, running over 8x faster than previous methods while achieving state-of-the-art accuracy.

weaknesses:
1. The idea of using the spatial-temporal transformer is not new, such as [1][2][3].
2. The authors are suggested to provide the compilation cost of each module to verify the efficiency of TAPE3D.
3. Are all inference experiments testing on a same machine? The information of the machine including GPU and CPU are suggested to provide.

[1] Hu M, Zhu X, Wang H, et al. Stdformer: Spatial-temporal motion transformer for multiple object tracking[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(11): 6571-6594.
[2] Zhang T, Jiao Q, Zhang Q, et al. Exploring Multi-modal Spatial-Temporal Contexts for High-performance RGB-T Tracking[J]. IEEE Transactions on Image Processing, 2024.
[3] Zhang T, Jin Z, Debattista K, et al. Enhancing visual tracking with a unified temporal Transformer framework[J]. IEEE Transactions on Intelligent Vehicles, 2024.

questions:
See weakness. If questions addressd, I will increase the rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649831831

thread_type:
official_review

<thread_object>
creator: reviewer_n65b

<content>
summary:
The paper proposes a model for tracking points in 3D densely, whose architecture is inspired by 2D sparse tracking CoTracker model, but with a number of modifications to 1) lift the tracks to 3D and 2) improve the model efficiency to allow for dense tracking with reasonable computational cost. 

For lifting the tracks to 3D, the authors propose that the model takes RGB-D frames as input, and add depth tokens to the model's transformer framework (eq 1) to refine the depth predictions jointly with the 2D tracking predictions. This seems to be similar to the approach proposed in SceneTracker.

For tracking densely with reasonable computational cost, the authors adopt two different strategies. First, they perform most of the tracking prediction at a downsampled resolution (H/r, W/r) and propose a learnt cross-attention based upsampler to upsample the predictions to (H, W) resolution. Second, authors introduce some modifications to the spatial attention module proposed in CoTracker to increase efficiency. As in CoTracker, they perform cross-attention between virtual tracks and sparse tracks (here sampled on a uniform grid on the first frame and called "anchor tracks"), and then perform most of the computation by self-attention layers on the virtual tracks. However, they introduce a third tensor of "dense tracks", which they process by local attention and then fuse with the anchor tracks before decoding back the final refined dense tracks predictions by performing cross-attention with the virtual tracks.

The model is trained using a fully supervised setting by using synthetic videos generated with the Kubric simulator. The model is first pre-trained on the 2D tracking setting for 100k iterations, and then further trained on the 3D setting for an additional 100k iterations.

Their model is evaluated both for the 2D and 3D tracking tasks using standard benchmarks such as CVO, LSFOdyssey and TAPVid-3D. The results show the model obtains SOTA results on the CVO benchmark as well as in the TAPVid-3D benchmark. On the LSFOdyssey benchmark, the model trained on Kubric does not attain SOTA performance, but the one finetuned on LSFOdyseey does.

The authors conduct a series of ablation studies to show the impact of the design decisions regarding the depth respresentation, spatial attention module and upsampler.

strengths:
This is one of the first papers addressing the TAP-3D problem with a feedforward model (not requiring test-time optimization), along with SceneTracker and SpatialTracker, which makes it a valuable contribution in this newly developing field.

In addition, the authors present quantitative results on both 2D tracking and 3D tracking which demonstrate the good performance of the model. The proposed model obtains SOTA results on the CVO benchmark as well as in the TAPVid-3D benchmark.

The visualizations included in the supplementary material help to further assess the quality of the 3D tracks produced by the model and compare it to the SceneTracker and SpatialTracker baselines. The results show a cleary superiority of TAPE3D over SceneTracker, and  slightly superior results compared to SpatialTracker. These observations are roughly in line to the reported quantitative results on TAPVid-3d.

The design choices proposed by the paper are thoroughly ablated in section 4.3 and Table 6, demonstrating the value of each of these design decisions.

Finally, Fig. 1 shows that in the dense-tracking scenario, TAPE3D is at least 8x faster than the baselines, while obtaining superior performance on TAPVid-3d.

weaknesses:
While the paper is a valuable contribution with good results, the model design seems to be mainly a combination of ideas from SceneTracker and CoTracker. The paper does not clearly state which ideas are novel, and which are borrowed from these previous methods. 

Furthermore, the experimental section only presents 2D tracking results on CVO, which is not the most widespread 2D tracking benchmark. Results on TAPVid-DAVIS would help in demonstrating the SOTA status of this model for 2D tracking. 

However, the main weakness of the method lies in that some of technical contributions are not presented with sufficient clarity or mathematical rigor. First, it is not clear how the dense tracks from Fig. 3, (3), are computed and how they connect to the tokens G^i_t of eq (1). In addition, the eq. (3) seems mathematically incorrect or at least too vague in notation and lacking explanation. It is not explained whether the functions q and k are linear layers and on what they are applied. Also, it's not clear what the softmax function is applied over. Finally, this is referred to a as cross-attention, but the values seems to be missing from the equation. While Figure 4 helps in understanding the general idea of the proposed attention-based upsampling, it is not exactly clear how to connect it to eq. (3) and therefore to fully understand the proposed module. 

Regarding the training procedure, it's not clear why the authors train on such small number of videos (5.6K) if these are synthetic videos that can be generated at a larger scale at ease. It's also not clear adding other data sources, such as PointOddysey would improve performance on the  TAPVid-3D benchmark.

questions:
Could you please describe which elements in the model design are novel and which are borrowed from prior work?

Have you tried evaluating your method on TAPVid-DAVIS (first)? Does it obtain SOTA results on 2D tracking compared to CoTracker and BootsTAPIR?

Please clarify how the dense tracks in Fig. 3 (3) relate to the explanations in Sec. 3 and in particular to the tokens G^i_t of eq. (1).

Please clarify how the proposed attention-based upsampling works and provide a corrected eq. (3) if possible. Please explain how the \tau cross attention blocks from Fig. 4 operate. Are these multiple heads in parallel?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731231239928

thread_type:
official_review

<thread_object>
creator: reviewer_g77y

<content>
summary:
This paper presents a novel method for dense 3D tracking in monocular video. By introducing the anchor tracks and attention based upsampling methods, the proposed method significantly improves the inference speed by ~10x. By modifying the depth representation and supervision signal, the method significantluy improves the performance.

strengths:
1. The proposed method indeed remedy the efficiency issue of previous SOTA methods significantly.
2. The experiments are comprehensive and convincing.

weaknesses:
Some key concepts and design choices are not well explained. 
 
1. Why do we need certain anchor tracks? What if we sample some certain points in the dense tracks?
2. The difference of sparse tracks and dense tracks are not well explained. For the comparisons in subfigure 2 and 3 in Fig.3, the dimension of sparse tracks and dense tracks are both T x N, then why it is called dense tracks? What is N' and L, the meaning of these two parameters are not well defined and explained.  
3. If the depth is predicted from monocular video, the scale is not specified. How can we ensure their consistency across different frames?
4. The attention upsampling method is quite related to the well known guided filter work. It is better to discuss the relationship with it.

questions:
1. Why the 2D version consistently better than 2D version in Table2? Is there any analysis on this unexpected results?
2. How the key parameters such as M, r, N' and L are chosen?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732059994521

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
The General Response to Reviewers (1/2)

comment:
We would like to thank all reviewers for their feedback. We are pleased that the reviewers acknowledged that: (i) our contributions to the 3D tracking are **"valuable and insightful for the field"** (Reviewer oj5m, n65b); (ii) our method achieves **significant improvements over recent approaches across multiple benchmarks while being 8x more efficient** (all reviewers); (iii) our paper includes **extensive experiments to justify our design choices** (Reviewer G77y).

**In the updated version of the paper, changes are marked in blue.** The changes to the paper are summarized as follows:
1. Sections 3.2 and 3.3 are modified to improve clarity.
2. Ablation on the anchor tracks (See Tab. 8 in the Appendix). 
3. Qualitative comparisons of long-range optical flow prediction between our proposed attention-based upsampler and the CNN-based upsampler used in RAFT[4] are included (see Fig. 7 in the Appendix, and L908-912).
4. Qualitative comparisons of long-range optical flow prediction between our approach, DOT, and the SoTA optical flow method SEA-RAFT[3] are also included (See Fig. 8 in the Appendix, and L914-916).
5. A downstream application is added: non-rigid structure from motion from in-the-wild dynamic videos (See Fig. 9 in the Appendix, and L917).
6. Quantitative results of pose estimation on Sintel and TUM-dynamic dataset are added (see Tab. 9 in the Appendix).

**We also provide video demonstrations of our method on this [anonymous website](https://tape3d.github.io/)**, including:

1. More qualitative results on dense 3D tracking for in-the-wild videos.
2. More qualitative comparisons with 3D point tracking approaches (SpaTracker, SceneTracker, DOT3D) and lifted-3D versions of 2D point tracking methods (CoTracker and LocoTrack) on in-the-wild videos.
3. Demonstrations of the limitations of lifted-3D versions of 2D point tracking methods, particularly the "jitter" effect caused by inconsistent video depth and occlusion issues.
4. Qualitative results of non-rigid structure-from-motion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives comments from 4 reviewers which are all positive for the publication of this paper. The paper is well-written and contributes to the tracking community.