[CvT] Tensorflow implementation
Author: mathieujouffroy
Pull request: yes

# What does this PR do?
This PR adds the Cvt model implementation in Tensorflow.
This includes the base model and the model with an image classification head on top.

<!-- Remove if not applicable -->

## TODO
- [x] Write the fundamental components (Convolutional Token Embeddings & Convolutional Transformer Block)
- [x] Write base model & image classification model 
- [x] Modify related utilities 
- [x] Write relevant tests (in test suite)
- [x] Preview Tensorflow documentation for Cvt 

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


### Questions
- In the configuration file of the model CVT, ```layer_norm_eps``` is initialized at ```1e-12```.
However, it seems that in the original implementation, the authors use ```epsilon=1e-5```.
Moreover, the Cvt model in pytorch (HuggingFace), does not seem to use the configuration ```layer_norm_eps=1e-12``` for layer normalization throughout the model, instead using the default ```epsilon=1e-5```.
What is the use of layer_norm_eps in the configuration file (of the Cvt model) ? 

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for your PR @mathieujouffroy! Let me ping @amyeroberts for review :)

You're welcome. Cool thanks, should I create an Issue ? 

Thanks a lot for both of your reviews 🙏 ! 
I've corrected the issues :) 
Although, I kept using `shape_list` instead of `tf.shape` throughout the implementation of the model as `tf.shape` was breaking things while running the tests (see comment above).
Should I follow the instructions in this [PR comment](https://github.com/huggingface/transformers/pull/18678#issuecomment-1222244001) to upload to weights ?

@mathieujouffroy awesome, seems like we are ready to move on to the next stage. I'm adding @sgugger as the last reviewer.

Meanwhile, you can open the PR to the TF model weights on the hub as follows:
1. Make sure you have the latest version of the hub installed (`pip install huggingface_hub -U`) and that you are logged in to HF with a write token (`huggingface-cli login`)
2. Run `transformers-cli pt-to-tf --model-name foo/bar` from this branch :D
3. In the Hub PR, tag `@joaogante, @nielsr, @sgugger`

> @mathieujouffroy awesome, seems like we are ready to move on to the next stage. I'm adding @sgugger as the last reviewer.
> 
> Meanwhile, you can open the PR to the TF model weights on the hub as follows:
> 
> 1. Make sure you have the latest version of the hub installed (`pip install huggingface_hub -U`) and that you are logged in to HF with a write token (`huggingface-cli login`)
> 2. Run `transformers-cli pt-to-tf --model-name foo/bar` from this branch :D
> 3. In the Hub PR, tag `@joaogante, @nielsr, @sgugger`

I am getting an error when using `transformers-cli pt-to-tf --model-name microsoft/cvt-13` : 
```
File "/Users/MathieuJouffroy/transformers/src/transformers/commands/pt_to_tf.py", line 307, in run
    + "\n".join([f"{k}: {v:.3e}" for k, v in hidden_differences.items() if v > self._max_error])
ValueError: The cross-loaded TensorFlow model has different outputs, something went wrong!

List of maximum output differences above the threshold (5e-05):
logits: 1.190e-04

List of maximum hidden layer differences above the threshold (5e-05):
hidden_states[2]: 1.227e-02
```

It seems that both the `max_crossload_output_diff `and the `max_crossload_hidden_diff` are bigger than the `self._max_error` **(5e-5)**. 
Respectively I have `max_crossload_output_diff` = 0.00011897087 **(1.190e-04)** and  `max_crossload_hidden_diff` = 0.012268066 **(1.227e-02)**.

I am trying to figure out how to correct this error (WIP).

@mathieujouffroy ~1e-2 is quite large -- does this happen exclusively on `microsoft/cvt-13`, or across all CvT models?

> @mathieujouffroy ~1e-2 is quite large -- does this happen exclusively on `microsoft/cvt-13`, or across all CvT models?

Yess, unfortunately it happens across all CvT models.
When inspecting the difference between the hidden states of the pytorch model and the hidden states of the tensorflow model, I can see that the difference increases throughout the model (with the number of layers).
The CvT model is composed of 3 stages of encoder block, with their respective number of layers being 1, 2 and 10. In the last stage, the difference between the torch model's hidden state and the tensorflow model's hidden state increases from ~e-5 at layer[0] to ~1e-2 at layer[9].
For `microsoft/cvt-13`  : 
```
CvtLayer output vs TFCvtLayer output

diff pt-tf stage[0]/layer[0]: 1.77919864654541e-05

diff pt-tf stage[1]/layer[0]: 2.4199485778808594e-05
diff pt-tf stage[1]/layer[1]: 3.9249658584594727e-05

diff pt-tf stage[2]/layer[0]: 3.0934810638427734e-05
diff pt-tf stage[2]/layer[1]: 0.000102996826171875
diff pt-tf stage[2]/layer[2]: 0.0004825592041015625
diff pt-tf stage[2]/layer[3]: 0.0009307861328125
diff pt-tf stage[2]/layer[4]: 0.001621246337890625
diff pt-tf stage[2]/layer[5]: 0.0032196044921875
diff pt-tf stage[2]/layer[6]: 0.0064239501953125
diff pt-tf stage[2]/layer[7]: 0.0091705322265625
diff pt-tf stage[2]/layer[8]: 0.012481689453125
diff pt-tf stage[2]/layer[9]: 0.01226806640625

Hidden Differences:
hidden_states[0]:1.77919864654541e-05
hidden_states[1]:3.9249658584594727e-05
hidden_states[2]:0.01226806640625

Output Differences:
logits:0.00011897087097167969
```
I can't seem to correct this issue. I was wondering if this was due to floating points operations.
Do you have any advice ? 🙏


@mattchurgin in these cases, a deep dive has to be done -- place a pair of `breakpoint()` in the layer where the problems start, one in each framework, and see which operation causes the divergence. Then, confirm that the TF operation/layer is parametrized correctly and, if it is, one has to dig even deeper :D 

Hello @gante, sorry for the late response.
I've done a deep dive into both frameworks. It seems that the Batch Normalization is responsible for the divergence. The 2 residual connections further increase the divergence throughout the model. However, I have parameterized `tf.keras.layers.BatchNormalization` accordingly to the default parameters of pytorch (`epsilon=1e-5` and `momentum=0.1`). I have also set both models in inference mode when testing.

Is this divergence due to the **momentum** definition of Batch Normalization being different in tensorflow than in pytorch ?

When removing the Batch Normalization layers from both frameworks, the difference in the output tensors and the hidden states is greatly reduced. I get a `max_crossload_output_diff` of ~e-6 and a `max_crossload_hidden_diff` of ~e-4 for all Cvt models. However, the `max_crossload_hidden_diff` is still higher than 5e-5 (I have ~e-4). The 2 residual connections are responsible for this difference.

I'm a bit confused. Therefore I've inspected the ViT model (`google/vit-base-patch16-224`) which also has 2 residual connections. There is also a divergence in the hidden states between the tensorflow implementation and the pytorch implemention. This difference also increases throughout the layers (with the residual connections), until it reaches a `max_crossload_hidden_diff` of ~2e-2 at layer 12.

Is this behaviour normal/acceptable ?

@mathieujouffroy That's a great in-depth exploration! 

Previously we didn't have these checks in place, so it is possible that issues like the one you're seeing slipped through the cracks. It's not positive at all to have such large mismatches (it implies that TF users will have a poorer experience). I've had in my plans to go back and double-check the most popular models with the recently introduced checks, and you've just raised the priority of the task with your message :)

I think @amyeroberts has seen similar TF/PT mismatches due to the Batch Normalization layer. @amyeroberts do you mind pitching in?

@mathieujouffroy Thanks for all the work digging into this 🕵️  

As `momentum` is set for both the pytorch and TF models, I believe their behaviour (outputs and moving stats updates) _should_ be the same during both inference and training, given the same weights and params.  

@gante @mathieujouffroy Yes, I had similar issues with the TF ResNet port ([a weights PR for reference](https://huggingface.co/microsoft/resnet-152/discussions/1)). Like this model, the batch norm layer introduced differences which then got amplified through the forward pass. @ydshieh did some excellent detective work, and found that matching all of the parameters and inputs to produce an equivalent TF and PyTorch layer would still produce outputs with a difference on the order of `1e-7` (enough to start causing problems 😭)  

Ultimately, we decided to add the weights as the difference between the logits was small ~1e-5. I think the ~1e-4 absolute differences in this case are acceptable for adding the weights. @sgugger Is this OK? 

Yes, as long as it stays in the range of 1e-4, we can accept the difference between frameworks.

Thank you for pitching in @amyeroberts :D

@mathieujouffroy feel free to use `--max-error 1e-4` (or slightly higher) in the `pt-to-tf` CLI to ignore those errors and push the weights!

@gante @amyeroberts you're welcome and thanks a lot for your feedbacks 😊 !

@amyeroberts It seems that for the batch normalization, the update rule for the running statistics is slightly different in Tensorflow compared to Pytorch:
PT -> `running_mean = (1 - momentum) * running_mean + momentum * new_value`
TF -> `running_mean = momentum * running_mean  +  (1 - momentum) * new_value`
Therefore, I think I made a mistake in setting the momentum to 0.1 in TF Batch Norm. Considering the update rules, shouldn't the momentum be set to 0.9 (default) in TF when it is set to 0.1 (default) in PT ? 
However, even though I change the momentum, I still have the same difference in my outputs 😕.

@gante Ok thanks. Although, just as a reminder, when I keep the Batch Normalization layers, I have a `max_crossload_output_diff` of ~1e4 and a `max_crossload_hidden_diff` of  ~2e-2 for all CvT models except the `cvt-21-384-22k`. The `cvt-21-384-22k` has a `max_crossload_output_diff` of ~4e4 and a `max_crossload_hidden_diff` of  ~1e-1.
Therefore, should I use `--max-error 2e-2` for all CvT models and  `--max-error 1e-1` for `cvt-21-384-22k` ? 

I'll also be more than happy to help if you need any assistance regarding the mismatches between PT and TF (I'm a bit intrigued) 😊

Hello @gante @amyeroberts, as pointed out in this [PR](https://github.com/huggingface/transformers/pull/19341),  the dense layer weights for PT should be initialized with `nn.init.trunc_normal_` instead of `normal_` as in the original implementation of the Cvt model (which uses `trunc_normal_` from `timm` library). In TF `get_initializer` already returns `tf.keras.initializers.TruncatedNormal`.
Also, following the original implementation,  in both frameworks the `cls_token` should be initialized with `trunc_normal` as with the `config.initializer_range` (here 0.02).

Should I add the modifications regarding the `momentum` (setting it to 0.9 in TF) and the use of `trunc_normal` ? 

@mathieujouffroy regarding initialization yeah, update it if possible :) In any case, it has come to our attention that TF initialization is very wrong in some places, and we will be adding tests and updates in the coming weeks!

Regarding momentum, I will defer the decision to @amyeroberts, who has been working more closely with that layer.

@mathieujouffroy Thanks for the update. 

Regarding the initialisation, could you update the PyTorch model in a separate PR and do similar thing as [suggested here](https://github.com/huggingface/transformers/pull/19341#pullrequestreview-1131874527) - naming the PR with a 🚨🚨🚨 prefix so we can easily spot and flag in the release notes. 

For momentum in the batch norm layers, yes please use the (1 - pytorch momentum) value :)  

@gante @amyeroberts Okay thanks, I will update the changes concerning the TF model (cls_token initialization & momentum) in this PR and will create a new PR for the Pytorch model 😊.

What should I do regarding the use of `pt-to-tf` (and `--max-error`) to add the weights ?
@gante should I wait for the future tests and updates you were mentioning regarding the TF initialization ?
Or should I add the weights with the `--max-error` I mentioned in the comment above ? 
Thanks !

@mathieujouffroy the initialization shouldn't change the mismatch between the two frameworks for pre-trained models -- I'd recommend going forward with `--max-error` 💪 

Hi @amyeroberts, thanks for your mention !! I've added the [PR](https://github.com/huggingface/transformers/pull/19486) regarding the pytorch model. 

@gante following your recommendation I've added the weights on the hub 😊
As @amyeroberts had pointed out, I'll need to remove `from_pt` in the testing file once the weights are added. 

@mathieujouffroy weights merged 🙌 

> @mathieujouffroy weights merged 🙌

Cool thanks @gante 😊 !
I'll update the testing file & run the slow tests locally . 

@mathieujouffroy off-topic: are you working with `transformers` as part of École 42? I've been at the school once (like 5+ years ago) and I had a friend who participated -- I really liked the concept!

> @mathieujouffroy off-topic: are you working with `transformers` as part of École 42? I've been at the school once (like 5+ years ago) and I had a friend who participated -- I really liked the concept!

@gante Yess I was working with `transformers` 🤗 on my last project (computer vision) at École 42. The project was in partnership with Hectar, an agricultural campus. I was pretty excited to try out the vision transformers 😊.
I've also used `transformers` at 42 for my NLP projects and in my internship (in NLP). 
I think 42 is a very good training (I've just finished) 🚀 : project-based & peer to peer pedagogy ! 

All seems ready, merging as soon as CI turns green.

@mathieujouffroy on behalf of TF users, thank you for making the ecosystem richer 🧡 

@gante @amyeroberts thanks a lot for your help and feedbacks !! 💛
It was really interesting and cool to do this PR (1st in an open source project) and to get it merge 😊