Add sdpa for Vivit
Author: RUFFY-369
Pull request: yes
Labels: name: Vision

name: SDPA

name: run-slow

# What does this PR do?

Towards #28005 

Adds sdpa for the ViViT model.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 


Comments:
@amyeroberts All green 🟢 :smiley: 

Hi @amyeroberts , I think in the `ViViT` model, originally `interpolate_pos_encoding` method [here](https://github.com/huggingface/transformers/blob/4196590aa0fe6ed735b68e126c2125a5d806acb7/src/transformers/models/vivit/modeling_vivit.py#L110) hasn't been correctly implemented and has just been copied from `ViT` because I made a commit in this current PR regarding `patch_size` [here](https://github.com/huggingface/transformers/pull/33757/commits/9c46bf2d4df7137a686dc8b55a58ee4280fc1c61) as `patch_size` itself wasn't declared in the method. I gave benefit of doubt to the work done during the port of this model and just fixed it. But when I tested the slow tests with `RUN_SLOW`, `test_inference_interpolate_pos_encoding` failed because the implementation hasn't been done properly. So, should I raise an issue separately or fix it here.

One thing can be that apart from the `test_inference_interpolate_pos_encoding` slow test rest can tested by `[run_slow] vivit` commit message and I will push the training and inference benchmark too just in some time. And when this PR gets merged, I will open a separate issue for `test_inference_interpolate_pos_encoding` test fix.

Or the other thing can be that I fix it here in this PR.

Please suggest what works well for you?!

> Hi @amyeroberts , I think in the `ViViT` model, originally `interpolate_pos_encoding` method [here](https://github.com/huggingface/transformers/blob/4196590aa0fe6ed735b68e126c2125a5d806acb7/src/transformers/models/vivit/modeling_vivit.py#L110) hasn't been correctly implemented and has just been copied from `ViT` because I made a commit in this current PR regarding `patch_size` [here](https://github.com/huggingface/transformers/pull/33757/commits/9c46bf2d4df7137a686dc8b55a58ee4280fc1c61) as `patch_size` itself wasn't declared in the method. I gave benefit of doubt to the work done during the port of this model and just fixed it. But when I tested the slow tests with `RUN_SLOW`, `test_inference_interpolate_pos_encoding` failed because the implementation hasn't been done properly. So, should I raise an issue separately or fix it here.
> 
> One thing can be that apart from the `test_inference_interpolate_pos_encoding` slow test rest can tested by `[run_slow] vivit` commit message and I will push the training and inference benchmark too just in some time. And when this PR gets merged, I will open a separate issue for `test_inference_interpolate_pos_encoding` test fix.
> 
> Or the other thing can be that I fix it here in this PR.
> 
> Please suggest what works well for you?!

Update:  [This](https://github.com/huggingface/transformers/pull/33757/commits/9c46bf2d4df7137a686dc8b55a58ee4280fc1c61) and [this commit](https://github.com/huggingface/transformers/pull/33757/commits/50b9f1a54bcee7a8a4e8f1ff37a69424e54e7cd3) was enough to fix the interpolate method and the only thing required right now is a nit in `test_inference_interpolate_pos_encoding` for it to pass, basically a correction in the input being passed. Right now, I'm keeping that nit change in hold because I need a confirmation from your side that to fix `test_inference_interpolate_pos_encoding` failure in this issue or simply open a separate issue.

And regarding this PR, 

> Thanks for adding!
> 
> Two things before merge:
> 
> * Running the slow tests for the model. Could you push an empty commit with the message `[run_slow] vivit`?
> * Adding info and benchmarks to the model's doc page. A good example is [here](https://huggingface.co/docs/transformers/v4.44.2/en/model_doc/gpt_neox#using-scaled-dot-product-attention-sdpa). Benchmarking scripts from @fxmarty for [training](https://gist.github.com/fxmarty/7e75cc3942d6974e4849093ebea0a331) and [inference](https://gist.github.com/fxmarty/5113e4304fbdd38c9c3702ce44683f6a).

Both of the things asked has been completed and pushed. So, this PR is ready for merge.

Hi @amyeroberts , I just created a separate PR where `interpolate_pos_encoding` for ViViT has been fixed here #33815 , fixing the slow test with it. 
Also, your suggestions have already been addressed for this PR so, its ready to be merged.

Please review both of them when you get the time.
Thank you :smile: 

@amyeroberts Can you please add `run_slow` label here as well so that the empty commit can work.
Thanks

@RUFFY-369 You should be able to add the label yourself, I think. Could you try? It would be useful to know if external collaborators can or can't add this, as we'd like to make this so that contributors can do as much as possible independently 

> @RUFFY-369 You should be able to add the label yourself, I think. Could you try? It would be useful to know if external collaborators can or can't add this, as we'd like to make this so that contributors can do as much as possible independently


@amyeroberts No, I can't. On my side I don't have any gear/settings button near label:
![Screenshot from 2024-09-30 18-23-20](https://github.com/user-attachments/assets/7cf2d4df-d54a-4e75-9ce1-7205c9f30140)

Usually if I can add the label, I would see something like this:

![Screenshot from 2024-09-30 18-24-06](https://github.com/user-attachments/assets/c97ba843-0587-4ff8-b87c-0d0b53a90191)

So, yes, I guess external collaborators can't add this


@RUFFY-369 OK, thanks for checking! @ydshieh This means the message for the slow model tests bot might need to be changed as contributors can't add the label as instructed

@RUFFY-369 I've added the label. Could you push another `[run-slow] vivit` commit to trigger the tests? 

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).


> @RUFFY-369 I've added the label. Could you push another `[run-slow] vivit` commit to trigger the tests?

@amyeroberts Done :+1: 


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33757). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Failing tests are unrelated

@RUFFY-369 Could you rebase? A fix has been merged in 

> @RUFFY-369 Could you rebase? A fix has been merged in

@amyeroberts Done

@amyeroberts Fallback has been added for `output_attentions`

Cheers!

All green, slow tests are left to trigger
If a maintainer can trigger them before merge
cc @amyeroberts 

soft ping @amyeroberts 
Thank you

soft ping @ArthurZucker 
Thank you :smile: 

🤗