Add support for XLM-R XL and XXL models by modeling_xlm_roberta_xl.py
Author: Soonhwan-Kwon
Pull request: yes

This PR adds support for the newly released XL and XXL models for XLM-R, and . These models are described in the "Larger-Scale Transformers for Multilingual Masked Language Modeling" paper.

And thank you for @patrickvonplaten and @stefan-it, as the review I got from the #13210, 
I added modeling_xlm_roberta_xl.py and convert_xlm_roberta_xl_original_pytorch_checkpoint_to_pytorch.py for conversion script.

I compared fairseq and transformers side by side,
and managed output same.

torch.Size([1, 11, 250880]) torch.Size([1, 11, 250880])
max_absolute_diff = 0.000186920166015625
Do both models output the same tensors? 🔥
Saving model to converted_xlmr_xl
Configuration saved in converted_xlmr_xl/config.json
Model weights saved in converted_xlmr_xl/pytorch_model.bin

Since fairseq roberta to transformers conversion was made a long time ago,
transformers architecture differs far from fairseq which originally started from,
and it makes quite confusion to write right code.
I synced transformers code to allow fairseq model structure.

- [ ] add test for XLM-R XL and XXL
- [ ] upload model for XLM-R XL and XXL to official repo 

Comments:
Thanks for the PR @Soonhwan-Kwon!

Could you also add a test file and some integration tests? :-)

@patrickvonplaten I started to work on test file, It seems test needs models uploaded on official repo. but how I can upload model files for xlm-roberta-xl or xlm-roberta-xxl to the official repo? 

Hey @Soonhwan-Kwon, 

Thanks a lot for working and this and sorry to reply so late!
Would it be ok to upload the checkpoints for now under your name on the hub and to make the tests pass and then in a last step, will move the checkpoints to the official organization?

Let me know if you need some help fixing the last steps :-)

> Hey @Soonhwan-Kwon,
> 
> Thanks a lot for working and this and sorry to reply so late! Would it be ok to upload the checkpoints for now under your name on the hub and to make the tests pass and then in a last step, will move the checkpoints to the official organization?
> 
> Let me know if you need some help fixing the last steps :-)

Thank you for the reply, I'm middle of uploading models, but it takes time for xxlarge(over 24GB) model.

@patrickvonplaten I have uploaded all models, but I have no idea how to fix last steps because I'm kind of newbie here. How can I fix last steps? Thank you in advance.

@Soonhwan-Kwon, could you maybe also add a configuration file (just copy the xlm-roberta one) and also add a full test suite for the model? :-) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> @Soonhwan-Kwon, could you maybe also add a configuration file (just copy the xlm-roberta one) and also add a full test suite for the model? :-)

sorry for the late response. I already added config.json, so is it the tokenizer.json you're talking about? And I added simple test for models(tests/test_modeling_xlm_roberta_xl.py) but where can I find the full test suite?

Hey @Soonhwan-Kwon, 

I meant more a new `configuration_xlm_roberta_xl.py` python file that is more or less a copy of `configuration_xlm_robert.py`:-) But I see that the configs are exactly similar so maybe we can leave as is. 

@sgugger @LysandreJik - This PR adds the checkpoints of https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/ to `transformers`. The model is essentially a "scaled-up" version of https://huggingface.co/docs/transformers/master/en/model_doc/xlmroberta#overview . Since the "scaled-up" version has a significantly different architecture (layer_norm is used very different amongst others) we decided to make a new `modeling_xlm_roberta_xl.py` model file. Now would it be ok for you to a) **not** have a corresponding `configuration_xlm_roberta_xl.py` and just use the `config_xlm_roberta.py` code or do you prefer b) adding a new `configuration_xlm_roberta_xl.py` file for consistency? I'm a bit indifferent here, but do slighly prefer b). What do you think? 

@Soonhwan-Kwon - there are some failing tests which I think can partly be solved by rebasing to current master. Otherwise, if ok for you I'm also happy to dive in the PR and help you finish the last parts of it. Let me know what you prefer :-)

Hey @Soonhwan-Kwon, thanks a lot for your PR!!

@patrickvonplaten, I prefer `b)`: a lot of the library is built on the assumption that you have one configuration file/object per modeling file/model objects. Since we've authorized auto models to map one configuration to multiple models this isn't as much of an issue as it could have been in the past, but I'm positive we'll find edge cases where it doesn't work as well as we expect it to simply because of the wrong assumption.

Also, since it falls into our "new architecture test", there should be a new folder regrouping this modeling file and configuration file instead of putting everything in the xlm-roberta folder.

> Hey @Soonhwan-Kwon,
> 
> I meant more a new `configuration_xlm_roberta_xl.py` python file that is more or less a copy of `configuration_xlm_robert.py`:-) But I see that the configs are exactly similar so maybe we can leave as is.
> 
> @sgugger @LysandreJik - This PR adds the checkpoints of https://ai.facebook.com/blog/-xlm-r-state-of-the-art-cross-lingual-understanding-through-self-supervision/ to `transformers`. The model is essentially a "scaled-up" version of https://huggingface.co/docs/transformers/master/en/model_doc/xlmroberta#overview . Since the "scaled-up" version has a significantly different architecture (layer_norm is used very different amongst others) we decided to make a new `modeling_xlm_roberta_xl.py` model file. Now would it be ok for you to a) **not** have a corresponding `configuration_xlm_roberta_xl.py` and just use the `config_xlm_roberta.py` code or do you prefer b) adding a new `configuration_xlm_roberta_xl.py` file for consistency? I'm a bit indifferent here, but do slighly prefer b). What do you think?
> 
> @Soonhwan-Kwon - there are some failing tests which I think can partly be solved by rebasing to current master. Otherwise, if ok for you I'm also happy to dive in the PR and help you finish the last parts of it. Let me know what you prefer :-)

@patrickvonplaten Sure, I will be glad if you help the last parts and feel free to dive in this PR.

@patrickvonplaten I added you as collaborator in my repo, perhaps you might need access.

Thanks @Soonhwan-Kwon - I'll try to tackle this tomorrow :-) 

@Soonhwan-Kwon - I've done some changes and left some final To-Dos in case you would like to tackle them :-) 

We should also add this model to the README.md with this paper: https://arxiv.org/pdf/2105.00572.pdf and give it a doc

Feel free to finish those last tasks if you want. Otherwise, I think I'm available to do this by mid- or end of the next week :-)

Thanks for working on that @Soonhwan-Kwon . I made some minor suggestions and will look at the tokenization part now (to check if there are any differences between XLM-R and XLM-R-XL/XXL :)

The tokenization part is working as expected. Here are some details:

* Underlying sentence piece models (XLM-R and XLM-R-XL) are identical (checked that via `torch.hub.load`, that downloads the model and stores them under `~/.cache/torch/pytorch_fairseq`). Checksums are the same.

* Tokenizer mapping (fairseq to spm model) is thankfully the same as for XLM-R, here I documented that mapping:

https://github.com/huggingface/transformers/blob/2e9af294940083915ccb2740a7c8d5b154194f15/src/transformers/models/xlm_roberta/tokenization_xlm_roberta.py#L163-L167

* But where does this vocab size "mismatch" come from? XLM-R has a vocab size of 250,002, whereas XLM-R-XL has 250,880.

Fairseq comes with an own dictionary file (`dict.txt`), for XLM-R it has 249,997 entries, and 250,875 entries for XLM-R-XL.

The dictionary file for XLM-R-XL is the same as XLM-R, but it contains `madeupword` tokens ranging from `madeupword0` to `madeupword877` at the end.

@stefan-it - it would also be great if you could do a final review :-)

Really cool! I'm currently running experiments on token classification with that new model :hugs: 

@patrickvonplaten @sgugger @stefan-it  Thank you for the merge, it was a great experience, and I came to respect committers of transformers. And below revert is just miss click, sorry.