Add ALIGN to transformers
Author: alaradirik
Pull request: yes

# What does this PR do?
Adds [ALIGN](https://arxiv.org/abs/2102.05918) to transformers, a multi-modal model similar to CLIP. ALIGN uses EfficientNet as its vision encoder and BERT as its text encoder. No public implementation is available for this model, the code is adapted from Kakao Brain's tensorflow implementation shared with us.

To do:
- [x] Upload converted model
- [x] Create a model card
- [x] Fix processor tests
- [x] Fix model tests

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for adding this model! Most comments are just formatting nits.
> 
> Two main comments to address:
> 
> * Can you modify the model prefix to Align, rather than ALIGN?
> * All of the vision encoder architecture is, as far as I can tell, exactly the same as the EfficientNet model. And the text encoder is directly imported from Bert. Is there a reason for reimplementing EfficientNet but importing Bert?

Thank you! 

I can modify the model prefix to Align but then I will have to change README titles to Align in order to not get repo-consistency errors and I'd like to keep the model name in the documentation the same as the original work. Could we keep it as it is, similar to BERT and CLIP? cc @sgugger 

The Kakao Brain EfficientNet implementation is slightly different from the official one (final layers), hence I'm copy pasting the modules and changing the final layers.

> The Kakao Brain EfficientNet implementation is slightly different from the official one (final layers), hence I'm copy pasting the modules and changing the final layers.

OK, I understand better now. I realised there's something I overlooked in my initial review: why are we using a randomly initialized Bert model as the text encoder? My understanding of this model is a text and vision backbone are loaded in and then trained to align their embeddings i.e. the weights we should be loading are the respective BERT and EfficientNet weights post-training and the api would be something like `AlignModel.from_pretrained(google/align-efficientnetv2-bert-large')`. Is this correct? 

I agree that it'd make more sense to initialize BERT and EfficientNet from pretrained checkpoints but here is the experiment setup described in the paper:
> We train our ALIGN models from scratch, using the open sourced implementation of EfficientNet as the image encoder and BERT as the text encoder.

The Kakao Brain implementation also trains models from scratch, should I keep it in line with the paper or use the respective repos? Both are fine with me


> I can modify the model prefix to Align but then I will have to change README titles to Align in order to not get repo-consistency errors and I'd like to keep the model name in the documentation the same as the original work. Could we keep it as it is, similar to BERT and CLIP? cc @sgugger
> 
My mistake, I figured out how to keep the documentation title camel cased, disregard this please.



Ah, OK. Sorry for the confusion. Just to make sure I understand: Google haven't released their weights; kakao Brain have themselves trained the ALIGN model and these are the weights we're using. Is this right?

> Ah, OK. Sorry for the confusion. Just to make sure I understand: Google haven't released their weights; kakao Brain have themselves trained the ALIGN model and these are the weights we're using. Is this right?

No problem at all, Kakao Brain implemented and trained the ALIGN model themselves, Google haven't released checkpoints nor the code.

I think all comments are addressed, pinging @amyeroberts and @sgugger for the final review