CoTracker: It is Better to Track Together
Authors: name: Nikita Karaev; name: Ignacio Rocco; name: Benjamin Graham; name: Natalia Neverova; name: Andrea Vedaldi; name: Christian Rupprecht
Published: 2023-07-14 21:13:04

Abstract: Methods for video motion prediction either estimate jointly the instantaneous
motion of all points in a given video frame using optical flow or independently
track the motion of individual points throughout the video. The latter is true
even for powerful deep-learning methods that can track points through
occlusions. Tracking points individually ignores the strong correlation that
can exist between the points, for instance, because they belong to the same
physical object, potentially harming performance. In this paper, we thus
propose CoTracker, an architecture that jointly tracks multiple points
throughout an entire video. This architecture combines several ideas from the
optical flow and tracking literature in a new, flexible and powerful design. It
is based on a transformer network that models the correlation of different
points in time via specialised attention layers. The transformer iteratively
updates an estimate of several trajectories. It can be applied in a
sliding-window manner to very long videos, for which we engineer an unrolled
training loop. It can track from one to several points jointly and supports
adding new points to track at any time. The result is a flexible and powerful
tracking algorithm that outperforms state-of-the-art methods in almost all
benchmarks.