Add ViViT
Author: jegork
Pull request: yes

# What does this PR do?

Fixes #15666

Add Video Vision Transformer to transformers. This PR implements a spacetime version of the Video Vision Transformer from the original paper. 

I have provided the model weights here https://huggingface.co/jegormeister/vivit-b-16x2-kinetics400 
I will try to add Factorised Encoder version later on (these are the two versions that authors provide weight for).


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/15666
- [x] Did you make sure to update the documentation with your changes? I have added the documentation, but I have troubles testing it as I couldn't run the preview command of the doc-builder, so if someone has the possibility to run and check it, I will be really grateful!
- [x] Did you write any new necessary tests? WIP


## Who can review?

@LysandreJik answered to the original issue so would be great if you could assist with the PR or suggesting who could. Thanks!

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Hello, @jegork thanks for the PR! Having some experience in adding tests and fixing repo-consistency/style, I can help you with these aspects if you need any :+1:  Feel free to tag me when needed.

cc @alaradirik and @NielsRogge 

Hi @jegork, thanks for working on this! 

I saw that it's not possible to import and test ViViT on your PR branch yet, this is because a lot of files need to be edited to properly  import the new modules (ViViTConfig, ViViTModel, etc.). You can refer to this [PR](https://github.com/huggingface/transformers/pull/20459) to see what files need to be edited. 

You can either make these changes manually or run the `transformers-cli add-new-model` command, which automatically takes care of a lot of these changes and initializes the model-specific files (modelling_vivit.py, etc.). You can learn more about this over [here.](https://github.com/huggingface/transformers/blob/main/docs/source/en/add_new_model.mdx)

Once you are done, you can run the `make fixup` command to make sure your code passes the [style, quality and repo consistency CI tests](https://huggingface.co/docs/transformers/contributing).

cc @sgugger @NielsRogge 

Hey @alaradirik, thanks for your reply and guidance! 

I will address what you suggested and add tests by the end of the week.


@alaradirik I've fixed the structuring via the suggested `transformers-cli add-new-model` and have run `make fixup`. 

All the relevant imports seem to work now (via `from transformers import ViViTModel, ViViTConfig, ViViTImageProcessor, ViViTFeatureExtractor, ViViTLayer, ViViTPreTrainedModel, ViViTForVideoClassification`)

Thanks again! Will add tests next.

@jegork really cool work, as a next step could you try to make the CI as green as possible? Currently there are many failing checks (10 failing and 9 successful). You can click on "details" -> "artifacts" -> "failures long" to see why exactly a check has failed.

You will also need to rebase on the main branch due to some issues with TF versions which are fixed on the upstream branch:
```
git remote add upstream https://github.com/huggingface/transformers.git
git fetch upstream
git rebase upstream/main
```



@NielsRogge thanks for your reply! I was indeed looking at the CI and not picking up the issue because most of them display `No module named 'keras.saving.hdf5_format'` as the error, but it seems that is exactly what you mentioned - the recent update to the TF version in the main branch. I will look into that today!

Also please use `Vivit` to prefix all the classes (instead of `ViViT`), so `VivitConfig`, `VivitModel` etc (liek `BertConfig`, `BertModel` etc.). Users have complained a lot about the casing we use for our models as it makes them harder to find in the lib :-)

@NielsRogge apparently I have run what you provided incorrectly, sorry for that! 

Okay np, fixed 👍

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20441). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

 @NielsRogge would be great if you could take a look at the changes I made. The CI also seems to be all good. 
Thanks!

hey @NielsRogge, thank you for your comments!

I have a question regarding the conversion script. I am using `restore_checkpoint(flax_model_path, None)` from `flax.training.checkpoints`, however, using the current jax version 0.3.6 (it was installed on my machine automatically when installing dev transformers version as per documentation). Upgrading the version of flax to 0.3.25 fixes the issue (haven't tested lower versions). Is there any other way I could load a flax checkpoint? 


@NielsRogge i think I've addressed all of your points, though I have doubt whether I implemented correctly the transformations check in the conversion script (I couldn't come up with any better approach as the preprocessing code in the original implementation is kinda scattered among multiple files, but I have tried to leave as much as possible of notes there). Would be great if you could check that and, if applicable, suggest any improvements.

Thanks again!

seems like the test job fails with
```
FAILED tests/models/bridgetower/test_modeling_bridgetower.py::BridgeTowerModelTest::test_save_load_fast_init_to_base - AssertionError: 0.08835485577583313 not less than or equal to 0.001 : vision_model.visual.transformer.resblocks.11.attn.in_proj_weight not identical
```

after i rebased from the main branch. Seems like has nothing to do with this PR. Was it already fixed and shall I rebase again?

Hi @jegork! You're right, the failing test is unrelated to the PR. Could you wait for a day and rebase again?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Sorry for the late reply here, I've assigned @amyeroberts to review the PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.