TRecViT: A Recurrent Video Transformer
Authors: name: Viorica Pătrăucean; name: Xu Owen He; name: Joseph Heyward; name: Chuhan Zhang; name: Mehdi S. M. Sajjadi; name: George-Cristian Muraru; name: Artem Zholus; name: Mahdi Karami; name: Ross Goroshin; name: Yutian Chen; name: Simon Osindero; name: João Carreira; name: Razvan Pascanu
Published: 2024-12-18 19:44:30

Abstract: We propose a novel block for video modelling. It relies on a
time-space-channel factorisation with dedicated blocks for each dimension:
gated linear recurrent units (LRUs) perform information mixing over time,
self-attention layers perform mixing over space, and MLPs over channels. The
resulting architecture TRecViT performs well on sparse and dense tasks, trained
in supervised or self-supervised regimes. Notably, our model is causal and
outperforms or is on par with a pure attention model ViViT-L on large scale
video datasets (SSv2, Kinetics400), while having 3times less parameters,
12times smaller memory footprint, and 5times lower FLOPs count. Code and
checkpoints will be made available online at
https://github.com/google-deepmind/trecvit.