CvT: Introducing Convolutions to Vision Transformers
Author: AnugunjNaman
Pull request: yes

# What does this PR do?
Add CvT Model for Vision Classification

Fixes #13158


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@NielsRogge @FrancescoSaverioZuppichini 


Comments:
Well I will need to update all weights again. Since they were in contracted form. Okay will take time.

@FrancescoSaverioZuppichini I have made most of the changes. The changes left are:
1. Configuration: For num_stages, it would be better for understanding, inferring from depth might make it little difficult to understand IMO. But if you feel the need to remove then I can do that
2. Configuration: I will update the projection after reading paper once more.
3. Tuple one: Tell me how to update that part
4. CVTLayer, Intermediate and Stage part: I followed the convention used in ViT and Segfomer as @NielsRogge said. He can give the input.

<b>I especially need clarification on point 4, the changes will be heavy if I need to add Stage layer as well. Then renaming all the weights again and upload it.<b>

@NielsRogge waiting for update!

@NielsRogge  I will be free on weekend, I will make the suggested changes then 

1. I think it’s okay, more in line with repo and paper. More clear I guess.
2. I talked with Francesco, he said no need for that since it’s similar to ConvNextFeatureExteactor use AutoFeatureExtractor.


I was hoping to have a meeting were we can quickly finish this. As you suggested changes are minor. Maybe an hour or two it can be done. If your free any day this week we can do that then too.


> I was hoping to have a meeting were we can quickly finish this. As you suggested changes are minor. Maybe an hour or two it can be done. If your free any day this week we can do that then too.

Is it possible for you to apply the suggestions from the review?

@NielsRogge I made the suggested changes. The shifting of models to Microsoft is left. You can review it. 

Some remarks:

* PR is in a good state, but please don't close comments that aren't resolved yet. I would like the `BaseModelOutputWithCLSToken` to be removed, and just return the CLS token by default in the last hidden states.
* can you also add CvT to the doc tests? This will make sure all code examples in the docstrings are tested as well. All details found here: https://github.com/huggingface/transformers/tree/main/docs#testing-documentation-examples

@NielsRogge Can you work on the doc part. Every time something or the other mistake takes place. If the doc part is done. 

Then I can make changes in `BaseModelOutputCLSToken` and inference of stages using depth and length check on patch sizes etc.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16575). All of your documentation changes will be reflected on that endpoint.

@NielsRogge can you review it and suggest any further changes.
I'm not sure how you want to modify `cls_token` part. Since modifying in `CvtStage` section (stopping the split) will change shape of hidden states (stored in all hidden states and passing of 4D shape for CNN in next layer). I leave that part to you on how you want to change it to pass it over different classes further.

I have run the `make fix-copies` and done docstrings part. I think everything is done apart from change you wanted to make for `cls_token`.

P.S. Can we sit for an hour and so to finish this PR. I can then finally move on from it. I know you're busy but I hope you can find some time.

Hi,

It seems that there was an issue with rebasing as the "files changed" tab now shows that 800+ files have changed.

Could you open a new, clean PR?

@NielsRogge How should I do that. Create a new branch and copy all the codes?