Add BridgeTower model
Author: abhiwand
Pull request: yes

# What does this PR do?

This PR implements a HuggingFace Transformers version of **BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning** from the paper https://arxiv.org/abs/2206.08657.pdf

This paper has been accepted to https://aaai.org/Conferences/AAAI-23/

The model's pre-trained checkpoints and configurations have been released here:
https://huggingface.co/BridgeTower under:

-  https://huggingface.co/BridgeTower/bridgetower-base-itm-mlm
-  https://huggingface.co/BridgeTower/bridgetower-base

The following heads have been implemented:

- BridgeTowerForMaskedLM
- BridgeTowerForImageAndTextRetrieval

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@amyeroberts @NielsRogge @ArthurZucker could you please assist with review and feedback.

@philschmid 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for the model here https://moon-ci-docs.huggingface.co/docs/transformers/pr_20775/en/model_doc/bridgetower
show up under Text Models. This needs to be under **Multimodal Models**. Can someone please assist?

Thanks a lot for your review @younesbelkada 
We have addressed your comments in our latest commits. We have cleaned up the code based on your feedback!
We do plan to upload some more to the hub once this PR is merged successfully. 

Please do let us know if you have any more suggestions and feedback.

Any help on the failing style checks and tests will be really appreciated.

> Thanks so much for adding this new model into `transformers` and introducing another multimodal model on the ecosystem! The PR is in a very good shape! Very strong efforts on the integration side, we should be close merging this once the main comments will be addressed. I left a couple of comments, my main comments being code readability / structure comments:
> 
> * Feature extractors are deprecated and should be replaced by Image Processors only. This would require a very minimal change. Check what is done in BLIP for instance: #20716
> * From what I have understood only 2 models are uploaded on the Hub, therefore I think that the model initialization and forward functions of `FusionHead` and `LinkTower` can be much more simplified
> * I understand that some layers needs to be freezed. You can wrap the freezing procedure in a class method, and call it only if the module is in a training model (i.e., if `self.training = True` )
> * Please avoid using variable names such as `x`, `x1`, or `x1_` as it makes harder to understand what the variable is meant to be. Consider calling these variables `hidden_states`, or any. Same comments for variables such as `image_embeds` and `image_embeds_`.
> * I think that there is no need to assert that the tokenizer in a Roberta tokenizer. Tokenization auto does it magically for you
> * Let's wrap all the weights initialization methods in the method `_init_weights` and call `self.post_init()` at the end of the init method for each module that inherits from `BridgeTowerPreTrainedModel`.
> * Regarding your question about documentation, you should add it together with CLIP, [here](https://github.com/huggingface/transformers/blob/1543cee7c8c95ef47f832b1f37625ba2923c4994/docs/source/en/_toctree.yml#L498) in the multi modal models section.
>   Again thank you very much!



@abhiwand thanks a lot for your PR! Are you also planning to add classes for the downstream tasks (like VQA)?

> @abhiwand thanks a lot for your PR! Are you also planning to add classes for the downstream tasks (like VQA)?

@NielsRogge We are hoping to add VQA/other downstream tasks in the coming months and also release some more models to the model-hub. In this PR however, we won't be doing so.
We have addressed most of the review feedback. Could you please help merge this PR if you it looks good to you?

@amyeroberts @NielsRogge @younesbelkada I think we've handled almost all your comments and simplified and streamlined the code significantly wherever possible.

If you'll approve, can you'll please merge this PR. You are welcome to make changes too. Thank you very much for your valuable feedback!

> @amyeroberts @NielsRogge @younesbelkada I think we've handled almost all your comments and simplified and streamlined the code significantly wherever possible.
> 
> If you'll approve, can you'll please merge this PR. You are welcome to make changes too. Thank you very much for your valuable feedback!

@amyeroberts @NielsRogge @younesbelkada Could you please help merge this PR :) Thanks! Happy Holidays.

@NielsRogge Thanks a lot for your review! We have addressed your review feedback as possible.  If it looks good to you, could you please help merge this PR.
Could you also please merge https://huggingface.co/datasets/huggingface/documentation-images/discussions/28 PR - @abhiwand moved it to the right folder.

Dear @NielsRogge and @sgugger,
Thanks a lot for your review! We have addressed your review feedback as possible. If it looks good to you, could you please help merge this PR.
Sincerely,

> Thanks for your work on adding this new model! There are still a few things to do before we can merge it:
> 
> * the model type should not be used to make tests inside modeling code (see comments below)
> * make sure all of the modules defined take the config for all arguments directly extracted from it
> * make sure all of the modules defined are prefixed with `BridgeTower`
> 
> I've added comments below.

@sgugger Thanks a lot for your review! We have updated our code to reflect your suggestions. If it looks good to you can you please help merge the PR. Thanks again!

> Just added last comments around the `device` use: there is no need to add a `device` property to some of the modules introduced in this PR, you should rely on the device of other tensors.

Thanks @sgugger, we have addressed your feedback in the latest commit!

> Nice work!
> 
> Left a few comments, mostly nits. Only major comments are about removing `pad` from the image processing file and allowing `eps` to be configurable for the layer norm layers. Otherwise looks good to go for me :)

Thanks for your suggestions @amyeroberts and approval. I have addressed your changes in the PR. Can you please help merge the PR?

@NielsRogge Our PR keeps failing at tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_return_timestamps_in_preprocess. Would you please help to see if it is because of BridgeTower or because of something else? 
Thanks a lot

@abhiwand @tileintel Thanks for address all of the comments! On Monday there were two PRs merged into main which added `test_image_processing_common.py` (#20785) and updated the feature extractor references in the `test_image_processing_xxx.py` files (#20768). Could you update `test_image_processing_bridgetower.py` to reflect these please? 



@amyeroberts We have updated test_image_processing_bridgetower.py as you suggested. Thanks for the suggestion.
@NielsRogge @amyeroberts @sgugger We have addressed all of the comments. Thanks a lot for helping us to review and approve this. We are very looking forward to having this PR merged into main soon. 

Thanks again for your contribution!

@sgugger Thank you for merging this PR. May I ask when BridgeTower model will go to HuggingFace's production and what release is that? 
Thanks

The next release will be in a month roughly (given the fast last release was yesterday).

Thank @sgugger for letting us know.