TensorFlow MobileViT
Author: sayakpaul
Pull request: yes

This PR implements the MobileViT model in TensorFlow.

## Interesting points

* The classification and segmentation models provided with MobileViT are fully compatible with TensorFlow Lite. Therefore, I have included sample code in the model documentation showing how to perform the TensorFlow Lite conversion (~4 lines of code). 
* TFLlite versions of the smallest checkpoints for classification and semantic segmentation are 1MB and 2MBs, respectively. I believe this will be quite beneficial to the TinyML community.

## TODOs

- [x] Hosting of the TF checkpoints on the Hub. (Can I do it now? If so, I need resources that show how to do that.)
- [x] Remove `from_pt` wherever needed. 

@amyeroberts @gante @sgugger up for review!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Didn't realize that re-requesting a review from @gante would result in removing @amyeroberts and @sgugger from the reviewer list. Please know that it was completely unintentional. 

@sayakpaul no worries :)

Thanks for another great model addition @sayakpaul ! 

@sayakpaul assuming it is passing the slow tests, it is ready for the TF weights.

The super complex instructions to do it are as follows:
1. Make sure you have the latest version of the hub installed (`pip install huggingface_hub -U`) and that you are logged in to HF with a write token (`huggingface-cli login`)
2. Run `transformers-cli pt-to-tf --model-name foo/bar` from this branch :D
3. In the Hub PR, tag `@joaogante, @lysandre` 

Super simple (complex?) question:

What is the format of `foo/bar`?

The same as the model name on the hub, e.g. [this model](https://huggingface.co/apple/mobilevit-small/tree/main) would be `apple/mobilevit-small`

P.S.: I edited the comment above with a 3rd step :D

The CLI might fail due to the conversion error being above the threshold -- let us know if that happens. There is a PR open with a flag to overwrite the error threshold.

> The CLI might fail due to the conversion error being above the threshold -- let us know if that happens. There is a PR open with a flag to overwrite the error threshold.

Failing due to this. Need that flag. 

@sayakpaul it is now merged (https://github.com/huggingface/transformers/pull/18752). You can use `--max-error` to change the limit.

This flag should be used with care. What are the differences you're seeing?

@gante, I think I have a clue as to why the `5e-5` threshold is being crossed during cross-loading. 

MobileViT model has these two components: unfolding and folding. They interpolate the intermediate feature maps. I checked ([Colab Notebook](https://colab.research.google.com/gist/sayakpaul/be24f152d91d0f1cbe95d5cea9ae8b14/scratchpad.ipynb)) the output consistency of `nn.functional.interpolate` and `tf.image.resize` with the same argument values. You'd notice that the outputs assert when `atol` is 1e-5, otherwise (higher `atol`) it fails. 

I suspect this inconsistency has a compounding effect and is the major reason the cross-loading fails with `5e-5`. 

I created PRs for adding the TF weights. Navigable from here: https://huggingface.co/apple. 

Cc: @amyeroberts 

@gante 

@hollance merged my PRs for the TF weights of MobileViT (thanks!). https://github.com/huggingface/transformers/pull/18555/commits/82079a74268c8b633e61064058362a0e6e53294c removes the `from_pt` argument. 

Nothing seems to be remaining now. Up to you (or anyone having merging privileges) to take the reigns. 

> [...] the output consistency of `nn.functional.interpolate` and `tf.image.resize` with the same argument values.

This might be due to the `align_corners` option. I once wrote a long blog post about this difference between PyTorch and TF. https://machinethink.net/blog/coreml-upsampling/ Not sure if that's the same issue but it seems likely.

> > [...] the output consistency of `nn.functional.interpolate` and `tf.image.resize` with the same argument values.
> 
> This might be due to the `align_corners` option. I once wrote a long blog post about this difference between PyTorch and TF. https://machinethink.net/blog/coreml-upsampling/ Not sure if that's the same issue but it seems likely.

Very well! If we need to deal with the inconsistencies between `tf.image.resize` and `nn.functional.interpolate` I suggest we do that in a separate PR 'cause various vision models would benefit from that (ViT for example). 

@gante WDYT?

@sayakpaul regarding the PR, all good on my end, but we still need approval from @sgugger :D

As for the `tf.image.resize` -- yeah, it would be nice to standardize for all models. Would you be interested in working on it? In any case, I'd like to ask you to open an issue, so we don't forget to track it! 

> As for the tf.image.resize -- yeah, it would be nice to standardize for all models. Would you be interested in working on it? In any case, I'd like to ask you to open an issue, so we don't forget to track it!

On it, sir!

@amyeroberts @gante 

Please take note of the changes in https://github.com/huggingface/transformers/pull/18555/commits/32cfd30cee185a090a80a6604b850c639b04203b. 

Initially, when I tested TFLite conversion it didn't require any spec for [SELECT operations](https://www.tensorflow.org/lite/guide/ops_select) but now they're failing with a specification for the SELECT ops. What is more surprising is that the TFLite interpreter is treating `tf.Conv2D` to be a SELECT op. Hence I have raised https://github.com/tensorflow/tensorflow/issues/57550. 

(retriggered failing job, seems like a spurious failure)

Yeah probably nothing related to the PR? 

The build doc job failure is not spurious. There seems to be a problem with an example bloc introduced by this PR.

Let me see if removing comments from the example block does the trick. Because when the job wasn't failing the example block didn't have any comments. 

No, it didn't help :( Any suggestions to try out? 

> The build doc job failure is not spurious. There seems to be a problem with an example bloc introduced by this PR.

My bad :D read the failure bottom to top, so I didn't notice the `mobilevit` errors