Add Data2Vec for Vision in TF
Author: sayakpaul
Pull request: yes

This PR adds the data2vec [1] model for vision in TensorFlow. 

**Todo**:

~* Fix cross-loading.~
~* Add integration test.~
~* Add remaining tests.~
~* Rest of the files remaining for the PR.~
~* TF weight uploading to Hub (to be done by someone from the 🤗  team)~

## Notes

* This PR does not add `...ForSegmentation`. This can be done in a separate PR I think. 
* Locally, I ran the tests using: `RUN_SLOW=1 python -m pytest tests/data2vec/test_modeling_tf_data2vec_vision.py`.

## References

[1] data2vec: https://arxiv.org/abs/2202.03555

@sgugger @Rocketknight1 @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I used these steps for styling: https://github.com/huggingface/transformers/pull/16255#discussion_r830432539. 

On my end, when I am running `make style` I get the following:

```
...

doc-builder style src/transformers docs/source --max_len 119 --path_to_docs docs/source
Overwriting content of src/transformers/models/speech_to_text/modeling_tf_speech_to_text.py.
Overwriting content of src/transformers/models/luke/modeling_luke.py.
Overwriting content of src/transformers/models/xlm_prophetnet/modeling_xlm_prophetnet.py.
Overwriting content of src/transformers/models/tapas/modeling_tapas.py.
Overwriting content of src/transformers/models/tapas/modeling_tf_tapas.py.
Overwriting content of src/transformers/models/data2vec/modeling_tf_data2vec_vision.py.
Overwriting content of src/transformers/models/t5/modeling_flax_t5.py.
Overwriting content of src/transformers/models/t5/modeling_t5.py.
Overwriting content of src/transformers/models/t5/modeling_tf_t5.py.
Overwriting content of src/transformers/models/rag/modeling_rag.py.
Overwriting content of src/transformers/models/rag/retrieval_rag.py.
Overwriting content of src/transformers/models/wav2vec2/modeling_flax_wav2vec2.py.
Overwriting content of src/transformers/models/encoder_decoder/modeling_tf_encoder_decoder.py.
Overwriting content of src/transformers/models/encoder_decoder/modeling_encoder_decoder.py.
Overwriting content of src/transformers/models/xlm/modeling_xlm.py.
Overwriting content of src/transformers/models/vision_encoder_decoder/modeling_tf_vision_encoder_decoder.py.
Overwriting content of src/transformers/models/imagegpt/modeling_imagegpt.py.
Overwriting content of src/transformers/models/longformer/modeling_longformer.py.
Overwriting content of src/transformers/models/xlnet/modeling_xlnet.py.
Overwriting content of src/transformers/models/xlnet/modeling_tf_xlnet.py.
Overwriting content of src/transformers/models/gpt2/modeling_tf_gpt2.py.
Overwriting content of src/transformers/models/prophetnet/modeling_prophetnet.py.
Overwriting content of src/transformers/models/realm/modeling_realm.py.
Overwriting content of src/transformers/models/openai/modeling_tf_openai.py.
Overwriting content of src/transformers/models/openai/modeling_openai.py.
Overwriting content of docs/source/en/model_doc/luke.mdx.
Overwriting content of docs/source/en/model_doc/bert-generation.mdx.
Cleaned 27 files!
```

The [CI console](https://app.circleci.com/pipelines/github/huggingface/transformers/38997/workflows/f017efba-6409-4669-835d-e463043d3ea0/jobs/436193) is also suggestive of this change. **Should I add these cleaned files to the PR?**



Make sure you update `hf-doc-builder` to its latest version with `pip install hf-doc-builder -U`. We had a new release last week to fix some bugs in the example styling in our docs :-)

> Make sure you update `hf-doc-builder` to its latest version with `pip install hf-doc-builder -U`. We had a new release last week to fix some bugs in the example styling in our docs :-)

@sgugger I see that `hf-doc-builder` is already up to date on my end (`Version: 0.3.0`).

You'll probably need to rebase your PR on master to get the changes in the setup for the quality check to pass (otherwise the CI uses the cached installed libraries).

> You'll probably need to rebase your PR on master to get the changes in the setup for the quality check to pass (otherwise the CI uses the cached installed libraries).

Thanks, @sgugger! I first rebased my main with the upstream and then merged the main into the PR branch. And then I force-pushed. Let's see. 

Hi, @sayakpaul 

- You can ignore `Model templates runner / run_tests_templates (pull_request)`. (You can even cancel that workflow run)
- I have just merged a (big) PR that moved model test folders, like `tests/bert` to `tests/models/bert`. When  you have time, could you
  - pull the changes (from upstream main) to your main
  - **rebase** your working branch on the main (better to avoid using `merge` in this case, I believe)
  - move your new test file `tests/data2vec/test_modeling_tf_data2vec_vision.py` from `tests/data2vec` to `tests/models/data2vec`
    - You might need to fix a few lines of `import`. 
    - For example, `from ..test_configuration_common import ConfigTester` --> `from ...test_configuration_common import ConfigTester`

please? 🙏 Thank you!


@ydshieh after rebasing, won't I need to merge the main into my PR branch so that it has the full effect?

> @ydshieh after rebasing, won't I need to merge the main into my PR branch so that it has the full effect?

In order to incorporate the changes in main into your PR branch, you can either use `merge` or `rebase`. I am in favor of using `rebase` as it might be cleaner in some cases (won't introduce a lot of file changes).

Once you have latest changes from upstream main in your local main, you can **checkout to your PR branch**, and do something like

```
git rebase main
```
(sometimes there might be conflicts to fix, but I think there won't be conflict in this case)

Then you will have to force push.

@ydshieh oops looks like I have made things worse instead of making them work. I am not sure how I can revert to a mergeable state now. Any suggestion?

> @ydshieh oops looks like I have made things worse instead of making them work. I am not sure how I can revert to a mergeable state now. Any suggestion?

Let me give it a try - I am definitely NOT a Git Pro 😢 
(No guarantee though - hope 🤞 )

Could you let me know what steps you have done, please?

I just followed your suggestions:

* Rebased my main with the upstream main.
* Checked out to my PR branch and ran `git rebase main`. 
* Made the necessary changes you suggested regarding moving the test file. 

I think you mistakenly made a push to my PR branch which is what may have caused the unnecessary changes to reflect in this PR. 

![image](https://user-images.githubusercontent.com/22957388/166462882-8e80d89e-dda0-4a0a-9e4d-e3f4357d5613.png)

I am happy to work on the necessary steps per your suggestions too. 

> I just followed your suggestions:
> 
> * Rebased my main with the upstream main.
> * Checked out to my PR branch and ran `git rebase main`.
> * Made the necessary changes you suggested regarding moving the test file.
> 
> I think you mistakenly made a push to my PR branch which is what may have caused the unnecessary changes to reflect in this PR.
> 
> ![image](https://user-images.githubusercontent.com/22957388/166462882-8e80d89e-dda0-4a0a-9e4d-e3f4357d5613.png)
> 
> I am happy to work on the necessary steps per your suggestions too.

Hi. That is the merge of my PR into main. I didn't merge that one into your PR. I am not sure why it appears like this and also confused. (Maybe it's somehow related to the merges have done). Let me try to figure out a way. Sorry about this.

> Hi. That is the merge of my PR into main. I didn't merge that one into your PR. I am not sure why it appears like this and also confused. (Maybe it's somehow related to the merges have done). Let me try to figure out a way. Sorry about this.

@ydshieh here's what I am thinking:

* Revert to https://github.com/huggingface/transformers/pull/17008/commits/247a6c53dc6a64664ff58c862319116aff359d9c. 
* Follow [your suggestions](https://github.com/huggingface/transformers/pull/17008#issuecomment-1116059265) again. 
* Push the changes. 

I am going to force push and see if it works 🙏 

Force push where?

To this PR, if you are OK with it. Please let me know, thanks.

> > Hi. That is the merge of my PR into main. I didn't merge that one into your PR. I am not sure why it appears like this and also confused. (Maybe it's somehow related to the merges have done). Let me try to figure out a way. Sorry about this.
> 
> @ydshieh here's what I am thinking:
> 
> * Revert to [247a6c5](https://github.com/huggingface/transformers/commit/247a6c53dc6a64664ff58c862319116aff359d9c).
> * Follow [your suggestions](https://github.com/huggingface/transformers/pull/17008#issuecomment-1116059265) again.
> * Push the changes.

Hi, I think we need to get to

```
[fix: tests due to removal of to_2tuple().](https://github.com/huggingface/transformers/pull/17008/commits/a0714e210c4f7da0f1321e50259ccf4fb40020ef)
```

and see what we can do to incorporate the main, that's what I am trying now.

Actually the commit I was referring to, it had the bits and pieces (like styling nits of the upstream files). 

It may just work, let's see @ydshieh 

If we revert to `[247a6c5](https://github.com/huggingface/transformers/commit/247a6c53dc6a64664ff58c862319116aff359d9c).`, we will still get a lot of changed file showing up in your PR.

I am able to get something cleaner like


<img width="399" alt="Screenshot 2022-05-03 162316" src="https://user-images.githubusercontent.com/2521628/166472233-75ccdf82-c653-4512-9131-9708a0015962.png">

by just revert to `ddd6b1c`, which I think it is the close to your PR with the changes from main in the clean way.
Let me know if you want to try it by yourself, otherwise I can push to this PR.

Sounds good. Let me know the steps. 

Here is what I would try
(always a good idea to have a backup)
```
git checkout -b tf-data2vec-backup
```
Then
```
git checkout tf-data2vec
git reset --hard ddd6b1c3
git push --force-with-lease
```
Once the commit history is clean on PR page, we can see if there is any style issues to fix. By that time, things should be easy.

@ydshieh fingers crossed 🤞

@sgugger, this is the step I need someone from the 🤗. team to perform. After that, I will remove `from_pt=True` from the code and will test.

> TF weight uploading to Hub (to be done by someone from the 🤗 team)

Will look into this. It's just for the checkpoint `facebook/data2vec-vision-base-ft1k` right? Or is there another one?

> Will look into this. It's just for the checkpoint `facebook/data2vec-vision-base-ft1k` right? Or is there another one?

There are four in the Facebook organization: [data2vec-vision](https://huggingface.co/models?sort=downloads&search=data2vec-vision)

@ydshieh forgot to say: THANK YOU VERY MUCH.

Currently, the follow checkpoint crashes (after the two suggestions I have made on the PR):
```
from transformers import TFAutoModel

tf_model = TFAutoModel.from_pretrained("facebook/data2vec-vision-base", from_pt=True)
```
Same for "facebook/data2vec-vision-large", therefore I can't convert those checkpoints (and it looks like something needs fixing?)

Here is the traceback:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/tmp/ipykernel_2749758/3199004601.py in <module>
----> 1 tf_model = TFAutoModel.from_pretrained("facebook/data2vec-vision-large", from_pt=True)

~/git/transformers/src/transformers/models/auto/auto_factory.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    444         elif type(config) in cls._model_mapping.keys():
    445             model_class = _get_model_class(config, cls._model_mapping)
--> 446             return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    447         raise ValueError(
    448             f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"

~/git/transformers/src/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1794 
   1795             # Load from a PyTorch checkpoint
-> 1796             return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
   1797 
   1798         # we might need to extend the variable scope for composite models

~/git/transformers/src/transformers/modeling_tf_pytorch_utils.py in load_pytorch_checkpoint_in_tf2_model(tf_model, pytorch_checkpoint_path, tf_inputs, allow_missing_keys)
    122     logger.info(f"PyTorch checkpoint contains {sum(t.numel() for t in pt_state_dict.values()):,} parameters")
    123 
--> 124     return load_pytorch_weights_in_tf2_model(
    125         tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
    126     )

~/git/transformers/src/transformers/modeling_tf_pytorch_utils.py in load_pytorch_weights_in_tf2_model(tf_model, pt_state_dict, tf_inputs, allow_missing_keys)
    153 
    154     if tf_inputs is not None:
--> 155         tf_model(tf_inputs, training=False)  # Make sure model is built
    156     # Adapt state dict - TODO remove this and update the AWS weights files instead
    157     # Convert old format to new format if needed from a PyTorch state_dict

~/anaconda3/lib/python3.9/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     65     except Exception as e:  # pylint: disable=broad-except
     66       filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67       raise e.with_traceback(filtered_tb) from None
     68     finally:
     69       del filtered_tb

~/git/transformers/src/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    381         main_input = fn_args_and_kwargs.pop(main_input_name, None)
    382         unpacked_inputs = input_processing(func, self.config, main_input, **fn_args_and_kwargs)
--> 383         return func(self, **unpacked_inputs)
    384 
    385     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, pixel_values, bool_masked_pos, head_mask, output_attentions, output_hidden_states, return_dict, training)
    893     ) -> Union[tuple, TFData2VecVisionModelOutputWithPooling]:
    894 
--> 895         outputs = self.data2vec_vision(
    896             pixel_values=pixel_values,
    897             bool_masked_pos=bool_masked_pos,

~/git/transformers/src/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    381         main_input = fn_args_and_kwargs.pop(main_input_name, None)
    382         unpacked_inputs = input_processing(func, self.config, main_input, **fn_args_and_kwargs)
--> 383         return func(self, **unpacked_inputs)
    384 
    385     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, pixel_values, bool_masked_pos, head_mask, output_attentions, output_hidden_states, return_dict, training)
    712         embedding_output = self.embeddings(pixel_values, bool_masked_pos, training=training)
    713 
--> 714         encoder_outputs = self.encoder(
    715             embedding_output,
    716             head_mask=head_mask,

~/git/transformers/src/transformers/models/data2vec/modeling_tf_data2vec_vision.py in call(self, hidden_states, head_mask, output_attentions, output_hidden_states, return_dict)
    625             layer_head_mask = head_mask[i] if head_mask is not None else None
    626 
--> 627             relative_position_bias = self.relative_position_bias() if self.relative_position_bias is not None else None
    628             layer_outputs = layer_module(hidden_states, layer_head_mask, output_attentions, relative_position_bias)
    629 

ValueError: Exception encountered when calling layer "encoder" (type TFData2VecVisionEncoder).

The first argument to `Layer.call` must always be passed.

Call arguments received:
  • hidden_states=tf.Tensor(shape=(3, 197, 1024), dtype=float32)
  • head_mask=['None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None', 'None']
  • output_attentions=False
  • output_hidden_states=False
  • return_dict=True
```

I have converted `facebook/data2vec-vision-base-ft1k` and am doing `facebook/data2vec-vision-large-ft1k` now.

@sgugger thanks for providing the update. Let me check from my end once. 

@sgugger should be all good now. 

I have verified from my end too:

![Screenshot 2022-05-03 at 8 57 18 PM](https://user-images.githubusercontent.com/22957388/166485490-9e115e28-ed34-40ef-9248-c1dc4348e031.png)

![Screenshot 2022-05-03 at 9 01 10 PM](https://user-images.githubusercontent.com/22957388/166485506-3f205bb3-9eaa-4c2a-88ab-96d7846b2ee0.png)





Can confirm it works. TF weights added for all 4 facebook Data2Vec Image models.

@sgugger thanks! I have also run the tests on my end locally, they're passing.

Over to you. 

Repinging @Rocketknight1 to have another set of eyes on this :-)

Thanks! Looks like they are all green now. 

Thanks again for your contribution!