ViT tensorflow Implementation
Author: elk-cloner
Pull request: no
Labels: name: Feature request

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

I was reading about ViT in the HuggingFace document and noticed there is no TF implementation of it. It would be great to have it in HuggingFace repo.

## Motivation

I have seen [this](https://keras.io/examples/vision/image_classification_with_vision_transformer/) and think it wouldn't be so hard. We can convert pytorch pretrain weights and use it for tensroflow model. 
 
<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Comments:
Hi, I've contributed ViT in PyTorch and wanted to also add ViT in Tensorflow, but there's currently a limitation to adding TF models that don't expect `input_ids` as an input (ViT only requires `pixel_values`). Any TF model in HuggingFace currently relies on a `input_processing` function (defined [here](https://github.com/huggingface/transformers/blob/77f4c46b501322e9bffb5416dfbf0397deefd7d8/src/transformers/modeling_tf_utils.py#L315)), and this function needs to be updated to also support models that don't expect input_ids as input.

cc @Rocketknight1 

My current implementation can be found [here](https://github.com/NielsRogge/transformers/blob/modeling_vit_tf_v2/src/transformers/models/vit/modeling_tf_vit.py).

> Any TF model in HuggingFace currently relies on a `input_processing` function (defined [here](https://github.com/huggingface/transformers/blob/77f4c46b501322e9bffb5416dfbf0397deefd7d8/src/transformers/modeling_tf_utils.py#L315)), and this function needs to be updated to also support models that don't expect input_ids as input.

nice work @NielsRogge. Could you answer these two questions, please?

1.  Can't /Should we use something  like `ViTFeatureExtractor` that was defined [here](https://github.com/huggingface/transformers/blob/fe3576488ad122b12364c66ef09dee38b3763f5f/src/transformers/models/vit/feature_extraction_vit.py#L31)??
2. What's the problem of current implementation of `input_processing ` ? if we feed `input_ids` tensor of shape `[batch_size, w, h, c]` to it what would be the problems ? 

> 1. Can't /Should we use something  like `ViTFeatureExtractor` that was defined [here](https://github.com/huggingface/transformers/blob/fe3576488ad122b12364c66ef09dee38b3763f5f/src/transformers/models/vit/feature_extraction_vit.py#L31)??

If `TFViTModel` and `TFViTForImageClassification` will be available, you can indeed use `ViTFeatureExtractor` to prepare images for the model (you only need to update the `return_tensors` parameter value to `"tf"` instead of `"pt"`).

> 2\. What's the problem of current implementation of `input_processing ` ? if we feed `input_ids` tensor of shape `[batch_size, w, h, c]` to it what would be the problems ?

Currently it only works as follows:

```
inputs = {"input_ids": None, "pixel_values": pixel_values}
outputs = model(inputs)
```



Hey! TF maintainer here - we're definitely aware of the issues with `input_processing`, but we're still working on the ways to fix it without breaking other things! If your model works when passing a null `input_ids`, it's fine to use that for now - you could possibly insert a shim into your `call()` method to avoid the user having to do it themselves?

I see a TF version of Wav2Vec2 has just been added, and they overwrote the `input_processing` function with a custom `input_values_processing` function as seen [here](https://github.com/huggingface/transformers/blob/040283170cd559b59b8eb37fe9fe8e99ff7edcbc/src/transformers/models/wav2vec2/modeling_tf_wav2vec2.py#L61). So I might do the same for ViT.