text: 
  
  TITLE
  Add Video Vision Transformer
  
  COMMENTS
  4
  
  REACTIONS
  +1: 8
  -1: 0
  laugh: 0
  hooray: 6
  heart: 6
  rocket: 0
  eyes: 0
  
  BODY
  # 🌟 New model addition
  
  ## Model description
  
  The ViViT is a transformer based model similar to Vision Transformer but operating on videos. In their paper, the authors present 4 types of possible architecture, and 2 are considered successful (unfactorised and factorised encoder models). Paper available at https://arxiv.org/abs/2103.15691.
  
  The original research uses flax, and mentions usage of weights from large pretrained image models (ViT). I currently have an implementation of model 1, analogous to existing ViTModel. I have also written a flax-to-pytorch weights conversion (to use weights from the original repo) and I have written the adaptation of ViT-based weights to ViViT weights (based on the methodology in the paper).
  
  ## Open source status
  
  * [ ] the model implementation is available: Original implementation in Scenic (flax) https://github.com/google-research/scenic/tree/main/scenic/projects/vivit
  * [ ] the model weights are available: Flax checkpoints and TF SavedModel present here https://github.com/google-research/scenic/blob/main/scenic/projects/vivit/README.md
  * [ ] who are the authors: Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid @MostafaDehghani @anuragarnab