TRecViT: A Recurrent Video Transformer
Authors: name: Viorica Pătrăucean, name: Xu Owen He, name: Joseph Heyward, name: Chuhan Zhang, name: Mehdi S. M. Sajjadi, name: George-Cristian Muraru, name: Artem Zholus, name: Mahdi Karami, name: Ross Goroshin, name: Yutian Chen, name: Simon Osindero, name: João Carreira, name: Razvan Pascanu

We propose a novel block for video modelling. It relies on a
time-space-channel factorisation with dedicated blocks for each dimension:
gated linear recurrent units (LRUs) perform information mixing over time,
self-attention layers perform mixing over space, and MLPs over channels. The
resulting architecture TRecViT performs well on sparse and dense tasks, trained
in supervised or self-supervised regimes. Notably, our model is causal and
outperforms or is on par with a pure attention model ViViT-L on large scale
video datasets (SSv2, Kinetics400), while having 3times less parameters,
12times smaller memory footprint, and 5times lower FLOPs count. Code and
checkpoints will be made available online at
https://github.com/google-deepmind/trecvit.