RealTracker: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos

Authors: Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, Christian Rupprecht
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Point tracking, Optical flow, Motion estimation, Pseudo labelling

Abstract:
Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task.
However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce RealTracker, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers.
The new model eliminates or simplifies components from previous trackers, resulting in a simpler and smaller architecture.
This training scheme is much simpler than prior work and achieves better results using 1,000 times less data.
We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking.
The model is available in online and offline variants and reliably tracks visible and occluded points.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730562867835

thread_type:
official_review

<thread_object>
creator: reviewer_4ezj

<content>
summary:
This paper introduces the RealTracker, a point tracker that combines several ideas from other related trackers but eliminates some components and simplifies others. RealTracker also designes a semi-supervised training protocol, where real videos are annotated utilizing several off-the-shelf trackers. With this protocol, RealTracker can achieve encouraging results on the Kinetics, RGB-S, and DAVIS datasets.

strengths:
1. RealTracker combines valuable ideas from the recent state-of-the-art point trackers and eliminates some unimportant modules.
2. RealTracker proposes a simple semi-supervised training protocol and achieves better results on several public datasets compared to state-of-the-art trackers.
3. RealTracker explores the training scaling low via its proposed training protocol.

weaknesses:
1. The idea of using trackers to annotate unlabeled datasets, such as [1], is not new.
2. The authors should use the Kub+15M data to train the CoTracker and TAPTR and verify the proposed method's effectiveness.
3. To prove the effectiveness of the RealTracker, it is suggested that confidence and visibility be visualized.
4. More ablation studies are suggested to verify that eliminating some modules in the listed trackers and simplifying some modules is useful, including the computation cost and tracking performance.

[1] Muller M, Bibi A, Giancola S, et al. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild[C]//Proceedings of the European conference on computer vision (ECCV). 2018: 300-317.

questions:
Please follow the weakness. If the issues are addressed,  I will improve the rating.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660441132

thread_type:
official_review

<thread_object>
creator: reviewer_mnzp

<content>
summary:
the approach leverages other point trackers to produce training data  for their point tracker.  Supposedly less additional training data is required compared to other point trackers.   The biggest contribution is that the other trackers use real data and not synthetic data for training.   Other approaches in the past have typically used point data for tracking.

strengths:
The paper incrementally builds upon point trackers by producing a better approach that leverages other point trackers to produce supervised training data.  In the past other trackers have used synthesized data however this is all based on real data.  The results seem to better than other point trackers.

weaknesses:
It is not clear on what type of motions were tested, if parallax for motion is required, what about zooming like motions with no parallax, does the method work.
What % of occlusion in terms of coverage of the object and in terms of time occluded were not clearly tested.
The limitations and failure cases of the algorithm were not explored.

questions:
From Table 2, it appears that the training set does matter in the results, The methods training with Kub+15M performed on average better than the methods trained with Kub, please explain and elaborate.  What is the difference?
Why does the offline method perform better than the online method, Intuitively I would assume the opposite?
What are the limitations and failure cases?
Table 6, why does SIFT turn on the best results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730875638388

thread_type:
official_review

<thread_object>
creator: reviewer_pqwv

<content>
summary:
1. The authors address the redundancy in modules of various existing point tracking models and propose RealTracker, a network with simplified architecture that achieves better performance and faster processing speed.

2. The authors leverage existing models to generate pseudo-labels for real video data, enabling effective utilization of unlabeled videos for network fine-tuning, which further enhances performance.

3. The authors analyze the impact of real data scale on the network model's performance, providing insights into the relationship between dataset size and tracking effectiveness.

strengths:
1. The paper's motivation is well-justified, particularly in its approach to eliminate model redundancies, resulting in a more lightweight yet powerful architecture.
2. The paper demonstrates effective utilization of unlabeled real-world datasets for training, achieving significant performance improvements through this approach.
3. The experimental analysis is comprehensive, and the visualization results are particularly impressive in demonstrating the model's capabilities.

weaknesses:
1. The methodology appears to be more engineering-oriented rather than theoretically innovative, primarily consisting of combinations and modifications of existing methods. The pseudo-label fine-tuning approach is relatively common. Given this is a **deep learning conference**, the technical contributions seem somewhat limited.
2. As acknowledged in the limitations section, the model's improvement of performance is heavily dependent on the teacher model's capabilities. This strong reliance on existing methods' performance creates a ceiling effect where the training results are constrained by the teacher model's performance limits, potentially reducing the method's generalizability.
3. The authors aim to bridge the domain gap using real-world dataset training. However, the paper lacks substantial technical innovation in terms of cross-domain adaptation techniques. The approach merely relies on real-data fine-tuning and teacher model voting effects for enhanced robustness, neither of which represents a significant contribution to the field of domain adaptation. More sophisticated cross-domain strategies or novel technical approaches would have strengthened the paper's contribution in addressing the domain gap problem.

questions:
1. The terminology "self-supervised fine-tuning" is indeed questionable in this context. Using state-of-the-art models from the same domain to generate pseudo-labels for supervision is more aligned with teacher-student learning or pseudo-labeling approaches rather than traditional self-supervised learning, where the supervision signals are typically derived from the data itself without external models.

2. The incorporation of domain adaptation strategies during the fine-tuning process would have significantly enhanced the paper's contribution. This could have included techniques specifically designed to address domain shift and better align feature distributions between source and target domains.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731120345225

thread_type:
official_review

<thread_object>
creator: reviewer_gpaf

<content>
summary:
This paper proposes a simpler and better point tracking approach by pseudo-learning real videos. Specifically, the proposed approach allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The proposed approach explores to use real video for training point tracking models w/o annotations. Moreover, the authors also study the scaling law to understand the impact of using more real training videos.

strengths:
- The paper focuses on an interesting problem in the community, i.e., aiming to explore to train TAP models w/ real videos w/o annotations, since the previous approaches mainly focus on learning w/ synthetic datasets;
- The proposed RealTracker shows that a simpler architecture and training protocols can outperform SOTA trackers like BootsTAPIR and LocoTrack;
- The paper is well written and organized;

weaknesses:
- Using pseudo-labels for training trackers is well explored, e.g., for some online learning-based trackers like Dino-Tracker, it uses pre-computed optical flow which provides the pseudo ground truth pixel-level correspondences for online training the tracker. For DinoTracker3, pseudo-labelling is explored. Please illustrate more differences with these trackers for better highlighting the contributions;
- Are there any specific concerns for choosing a teacher model for pseudo label generation? Does the better teacher model with higher tracking performance commonly lead to better tracking performance? Can a single teacher model well support the tracker learning?
-  In Table 2, the time of the per frame and per tracked point is shown. For the online variant, what’s the overall tracking speed (i.e., fps) given an online testing video?
- Missing Refs for discussion. For completeness, please include more pseudo-label based tracker training approaches [1,2,3,4] for discussion in the related work.

[1] Progressive Unsupervised Learning for Visual Object Tracking;

[2] Unsupervised Learning of Accurate Siamese Tracking;

[2] DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video;

[3] CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos;

questions:
Overall, I think this is an interesting paper that focuses on an essential problem in the community, i.g., enabling existing TAP trackers to leverage real videos w/o annotations for training. The idea is somewhat incremental but effectively addresses an essential problem in a simple yet effective way. Thus my current rating is ``accept''. I would like to see more author rebuttal in terms of differences w/ existing pseudo label based approaches as mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732576021259

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their thoughtful feedback. The reviewers find that RealTracker [R1]`is an interesting paper that focuses on an essential problem in the community` with a [R2]`well-justified motivation`. RealTracker's  [R2] `visualization results are particularly impressive in demonstrating the model's capabilities`, it [R3,R4] `achieves better results on several public datasets compared to state-of-the-art trackers`. We address reviewers' comments below and have already incorporated their feedback in the uploaded revised version.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The article has received feedback from four reviewers, to which the authors have provided partial responses. However, the lack of intense discussion among the reviewers suggests that the paper may have limited novelty and topicality. It is hoped that the authors will take the reviewers' comments into consideration for further refinement of the manuscript and wish them success in future submissions. The current version of the paper will be rejected.