Adding CvT Model : Convolution based Image Transformers
Author: AnugunjNaman
Pull request: yes

# What does this PR do?
Adding CvT Model : Convolution based Image Transformers
A  new architecture, named Convolutional vision Transformers (CvT), that improves Vision Transformers (ViT) in performance and efficiently by introducing convolutions into ViT to yield the best of both designes. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (e.g. shift, scale, and distortion invariance) while maintaining the merits of Transformers (e.g. dynamic attention, global context, and better generalization).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [NO ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [YES ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [https://github.com/huggingface/transformers/issues/13158 ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/13158
      to it if that's the case.
- [No ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ No] Did you write any new necessary tests?


## Who can review?
@NielsRogge
I have few queries and doubts and need help for further addition of pretrained models and adaption to respective base classes

Models:
-cvt @@NielsRogge

Library:



Comments:
@NielsRogge. Currently I have loaded model using another script for testing. The model works fine on samples images I have tested. But I need help at few steps:
1 . Adaption to base classes, especially for pretrained models
2. How to upload to hugging face archive
3. I also need to understand few thing in feature extractor part too.

So, yeah further guidance needed from here

```python
from transformers import CvTConfig, CvTModel, BeitFeatureExtractor, BeitForImageClassification
import torch
from PIL import Image
import requests


url = 'http://images.cocodataset.org/val2017/000000039769.jpg'

# tabby cat image

image = Image.open(requests.get(url, stream=True).raw)
feature_extractor = BeitFeatureExtractor.from_pretrained('microsoft/beit-base-patch16-224-pt22k')

model1 = BeitForImageClassification.from_pretrained('microsoft/beit-base-patch16-224')
inputs = feature_extractor(images=image, return_tensors="pt")
out1 = model1(**inputs)
logit1 = out1.logits
print(model1.config.id2label[logit1.argmax(-1).item()])


config = CvTConfig()
model2 = CvTModel(config)
model_file = '/home/naman/CvT/models/CvT-21-384x384-IN-1k.pth'

state_dict = torch.load(model_file, map_location="cpu")
model2.load_state_dict(state_dict, strict=False)

logit2 = model2(inputs['pixel_values'])
pred = logit2.argmax(-1).item()
print(model1.config.id2label[pred])
````

You can test it here. I have used BeITFeatureExtractor which is similar to CvT I think.

Hi,

Thanks for your PR. 

> Adaption to base classes, especially for pretrained models

I've seen that currently, the modeling file is a copy from the original repository. However, to add CvT to this library, we need the follow the same implementation as other models like ViT and BEiT (i.e. the HuggingFace API). Therefore, the `Block` class for example (which is used in the original timm-based implementation) will have to be translated to a `CvtLayer` class (similar to `ViTLayer`). I also opt to use `CvtModel` instead of `CvTModel`, as it will be more difficult for people to type ;) we should have done this for ViT too actually, and we've done it for BEiT now (`BeitModel` instead of `BEiTModel`).

Looking at the modeling file, the main difference between ViT and CvT seems to happen in the attention layer. So probably, you can just copy everything from `modeling_vit.py`, rename every Vit from that file to Cvt, and then update the attention layer.

The code example looks great already! Does it predict a reasonable class (like cat or remote)?

Do you have an email address? Then I set up a Slack channel to further guide you.

@NielsRogge Yeah prediction is good. I tested it on a small set of 20 images it okay there. Yeah I will need little guidance. 😅. Yeah email is anugunjjha@gmail.com

It's midnight here. I will work on it tomorrow. 

@NielsRogge I have done code as hugging face api. I have problems in tests. I need help there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@AnugunjNaman Do you have any updates? I'd like to help contribute if need be.

Yup, sorry yeah you can help. I got busy in job search since it was my final year. We can contact and continue from there. Can you write me your email? We can set up a time to discuss it.

Hey @AnugunjNaman, my email is jaesungtae@gmail.com. Happy to discuss more!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.