TAPNext: Tracking Any Point (TAP) as Next Token Prediction
Authors: name: Artem Zholus, name: Carl Doersch, name: Yi Yang, name: Skanda Koppula, name: Viorica Patraucean, name: Xu Owen He, name: Ignacio Rocco, name: Mehdi S. M. Sajjadi, name: Sarath Chandar, name: Ross Goroshin

Tracking Any Point (TAP) in a video is a challenging computer vision problem
with many demonstrated applications in robotics, video editing, and 3D
reconstruction. Existing methods for TAP rely heavily on complex
tracking-specific inductive biases and heuristics, limiting their generality
and potential for scaling. To address these challenges, we present TAPNext, a
new approach that casts TAP as sequential masked token decoding. Our model is
causal, tracks in a purely online fashion, and removes tracking-specific
inductive biases. This enables TAPNext to run with minimal latency, and removes
the temporal windowing required by many existing state of art trackers. Despite
its simplicity, TAPNext achieves a new state-of-the-art tracking performance
among both online and offline trackers. Finally, we present evidence that many
widely used tracking heuristics emerge naturally in TAPNext through end-to-end
training.