Self-Supervised Any-Point Tracking by Contrastive Random Walks
Authors: name: Ayush Shrivastava; name: Andrew Owens
Published: 2024-09-24 17:59:56

Abstract: We present a simple, self-supervised approach to the Tracking Any Point (TAP)
problem. We train a global matching transformer to find cycle consistent tracks
through video via contrastive random walks, using the transformer's
attention-based global matching to define the transition matrices for a random
walk on a space-time graph. The ability to perform "all pairs" comparisons
between points allows the model to obtain high spatial precision and to obtain
a strong contrastive learning signal, while avoiding many of the complexities
of recent approaches (such as coarse-to-fine matching). To do this, we propose
a number of design decisions that allow global matching architectures to be
trained through self-supervision using cycle consistency. For example, we
identify that transformer-based methods are sensitive to shortcut solutions,
and propose a data augmentation scheme to address them. Our method achieves
strong performance on the TapVid benchmarks, outperforming previous
self-supervised tracking methods, such as DIFT, and is competitive with several
supervised methods.