[TensorFlow] Adding GroupViT
Author: ariG23498
Pull request: yes

# Adding TensorFlow version of GroupViT

This PR adds the TensorFlow version of [GroupViT](https://github.com/NVlabs/GroupViT).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


CC: @LysandreJik @Rocketknight1 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Tagging @NielsRogge @Rocketknight1 @gante for the PR review!

@ariG23498 The parsing issue seen in the `check_repository_consistency` tests is arising because of `@keras_serializable` decorators being below `# Copied from` statements. If I check out your branch, I can run `make fix-copies` successfully if I move the decorator to be above the comment. 

Hey @ariG23498 👋 

Regarding failing tests:
- The tests in `run_tests_torch_and_tf` seem to need a slightly higher tolerance. Let's leave those as their are for now (failing), as we might be able to fix them throughout the PR review process :)
- The other errors is because the attribute `base_model_prefix` does not match the name of the attribute that is used to hold the models. I.e. it should be changed to `base_model_prefix = "group_vit"` (notice the `_` :D )
[`base_model_prefix` wrong -> [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_tf_utils.py#L1084) does not get matched with the correct class -> raises the exception in the function]


Hey @gante thanks for the insights.

> The other errors is because the attribute base_model_prefix does not match the name of the attribute that is used to hold the models. I.e. it should be changed to base_model_prefix = "group_vit" (notice the _ :D )
[base_model_prefix wrong -> [this line](https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_tf_utils.py#L1084) does not get matched with the correct class -> raises the exception in the function]

The [PyTorch](https://github.com/huggingface/transformers/blob/8cb5ecd912e09301be126c6ce6e9a22ca7153da4/src/transformers/models/groupvit/modeling_groupvit.py#L769) implementation has `groupvit` as the `base_model_prefix`. Would this be a problem if I changed the name for the TensorFlow port?


@ariG23498 Good point -- `groupvit` (no underscore) should be used everywhere then!

Fixes #18543 

CC: @NielsRogge 

References
- hard_softmax: https://gist.github.com/ariG23498/08cdae21637b8b61bdd6d21d11719fb3
- resize_attention_map: https://gist.github.com/ariG23498/3777f8d9be25de8ae782256f5aacb2c5

@amyeroberts I have added the `shape_list` back to places which are copied from other parts of the repository. This was done to pass the `make fixup`.

Hey @amyeroberts and @gante I had to swap back in the `shape_list` and also the `axis=range(len(shape_list(logits)))[dim]` to pass the tests.

Perhaps relevant to the TF-PT mismatch: https://github.com/huggingface/transformers/pull/18555#issuecomment-1230066514

> Perhaps relevant to the TF-PT mismatch: [#18555 (comment)](https://github.com/huggingface/transformers/pull/18555#issuecomment-1230066514)

Hey @gante I do not think this might be the source of the problem.

`nn.functional.interpolate` has been used twice in the torch code:
- [`resize_attention_map`](https://github.com/ariG23498/transformers/blob/feat/groupvit-tf/src/transformers/models/groupvit/modeling_groupvit.py#L135)
- [`interpolate_pos_encoding`](https://github.com/ariG23498/transformers/blob/feat/groupvit-tf/src/transformers/models/groupvit/modeling_groupvit.py#L416)

I have a [colab notebook](https://gist.github.com/ariG23498/3777f8d9be25de8ae782256f5aacb2c5) that takes care of the tf and pt equivalence for the `resize_attention_map`. As for the `interpolate_pos_encoding`, it has been taken from the ViT implementation (which is assumed to have no equivalence problem?).

@ariG23498 With the selected seeds, it passes now on CircleCI. This depends also on hardware, so it may or may not work when running on different machines, sadly.

It passes on GCP CPU-only VM too. For GPU, we might need different seeds, but we can do that later.

`tests/models/groupvit/test_modeling_tf_groupvit.py::TFGroupViTModelTest::test_keras_fit` failed however. I will leave @ariG23498 to check this one 🙏 

Tagging @sgugger for a review (as core maintainer) 🎉 

After this PR has 3 approvals (including Sylvain's), these are the instructions to finalize the PR:
1. Add the TF weights for the model(s). Assuming you are logged in to the Hub in your terminal (if not, run `huggingface-cli login`) and are on this branch, run `transformers-cli pt-to-tf --model-name foo/bar`, where `foo/bar` is the target model repo. Assuming it passes the CLI's checks, it will open a PR on the model hub. Please tag `@_nielsr, @_sgugger, @_joaogante` (⚠️ removing the `_`) in the opened Hub PRs :)
2. After the TF model weights are merged, make sure there are no `from_pt=True` in the PR. Then, re-run the test suite for the model locally and confirm that all tests pass (command: `NVIDIA_TF32_OVERRIDE=0 RUN_SLOW=1 py.test -vv tests/models/groupvit/modeling_tf_groupvit.py`)

The [`run_tests_hub`](https://app.circleci.com/pipelines/github/huggingface/transformers/47547/workflows/26d9127c-5aac-43e9-99ff-8a42a25aa0e1/jobs/563161) fails for the time being. @ydshieh thinks it is irrelevant to this PR. All the relevant tests pass.

Pinging @Rocketknight1 for the serving outputs bit (as mentioned by @sgugger [here](https://github.com/huggingface/transformers/pull/18020#pullrequestreview-1112320572)).

@Rocketknight1 The changes you have suggested are taken care of! @ydshieh the tests pass as well!

@gante @amyeroberts should we talk about the deprecated code used [here](https://github.com/huggingface/transformers/pull/18020#discussion_r974312323), or should we have a different PR to handle this?

@gante over to you for saving the TF model to hub 🫂

FYI: The failing tests are not related to the `GroupViT` model.

The failure is unrelated. As long as all models have been moved this PR can be merged.

@gante over to you now! 🤗 

@ariG23498 adding TF weights as PRs to the hub is now available to everyone -- you can have a go at it, following [the instructions above](https://github.com/huggingface/transformers/pull/18020#issuecomment-1250738894)! 

I can also do it. Let me know how you want to proceed :)

Thanks @gante I will give it a try myself!

@gante while running `transformers-cli pt-to-tf --model-name nvidia/groupvit-gcc-yfcc` I get the following error.

```
List of maximum hidden layer differences above the threshold (5e-05):
text_model_outputhidden_states[1]: 9.155e-05
text_model_outputhidden_states[2]: 9.155e-05
text_model_outputhidden_states[3]: 9.155e-05
text_model_outputhidden_states[4]: 9.155e-05
text_model_outputhidden_states[5]: 9.155e-05
text_model_outputhidden_states[6]: 9.155e-05
text_model_outputhidden_states[7]: 9.155e-05
text_model_outputhidden_states[8]: 9.155e-05
text_model_outputhidden_states[9]: 9.155e-05
text_model_outputhidden_states[10]: 9.155e-05
text_model_outputhidden_states[11]: 9.155e-05
text_model_outputhidden_states[12]: 9.155e-05
vision_model_outputhidden_states[1]: 2.441e-04
vision_model_outputhidden_states[2]: 7.629e-05
vision_model_outputhidden_states[3]: 7.629e-05
```

I think this is because of the seed issue that @ydshieh had pointed out.

How would you want me to proceed in this case?

@gante thanks for the help.

The PR for the TF weights on HF Hub are [here](https://huggingface.co/nvidia/groupvit-gcc-yfcc/discussions/1).

It seems ready to be merged, except for an inconsistency that arises from a `# Copied from` statement -- lmk if you need a hand!

@gante I have taken care of the `copied from` inconsistency.

I think we are good to go.

Merged! Thank you for making the TF users happier, @ariG23498 🧡