Add Video Vision Transformer
Author: jegork
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

The ViViT is a transformer based model similar to Vision Transformer but operating on videos. In their paper, the authors present 4 types of possible architecture, and 2 are considered successful (unfactorised and factorised encoder models). Paper available at https://arxiv.org/abs/2103.15691.

The original research uses flax, and mentions usage of weights from large pretrained image models (ViT). I currently have an implementation of model 1, analogous to existing ViTModel. I have also written a flax-to-pytorch weights conversion (to use weights from the original repo) and I have written the adaptation of ViT-based weights to ViViT weights (based on the methodology in the paper).

## Open source status

* [ ] the model implementation is available: Original implementation in Scenic (flax) https://github.com/google-research/scenic/tree/main/scenic/projects/vivit
* [ ] the model weights are available: Flax checkpoints and TF SavedModel present here https://github.com/google-research/scenic/blob/main/scenic/projects/vivit/README.md
* [ ] who are the authors: Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, Cordelia Schmid @MostafaDehghani @anuragarnab

Comments:
Would be super cool!

@jegork I am looking to use the Flax weights provided in the original ViViT repo in PyTorch too. Can you guide me how to convert the weights?

@akshatshah21 I still can't finish writing the tests for my pull request to add ViViT, but I have created a repo with my converter for you.
https://github.com/jegork/vivit-weights-converter

Thanks a lot @jegork !

@jegork is there any progress on adding vivit to huggingface/transformers? I need this model for my Ph.D. research and it would be super cool if I could use the hugging face version of it :)

Hello @fcakyon !
You're just on time! I have been really busy finishing my bachelor's and working, but I have just started to work on the pull request two days ago, and I hope to finish it today or tomorrow. Although I cannot give you an estimation of when it will be actually merged in the main branch, but in the worst case you would be able to use my fork until it is merged. I will tag you as soon as I have add a PR with a working version of the model (I will also add the model with the weights to the huggingface repo) 

Amazing news @jegork! Will be following your progress :+1: 

@fcakyon 
I have added a pull request to add the model, however, I am still in the process of writing tests so it might take some time to get it into the release, but you could already try the model out using my fork. 
You can install it by using
```sh
pip install git+https://github.com/jegork/transformers@add_vivit
```

To try it out, use the following code:
```python
from transformers import ViViTFeatureExtractor, ViViTForVideoClassification
feature_extractor = ViViTFeatureExtractor.from_pretrained("jegormeister/vivit-b-16x2-kinetics400")

model = ViViTForVideoClassification.from_pretrained("jegormeister/vivit-b-16x2-kinetics400")
```

Will really appreciate your feedback!