Add X-MOD
Author: jvamvas
Pull request: yes

Add the X-MOD models released with the paper [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](http://dx.doi.org/10.18653/v1/2022.naacl-main.255).

## Implementation notes
- There are nine pre-trained models released in the fairseq repo: https://github.com/facebookresearch/fairseq/tree/main/examples/xmod. I will upload them under my own name and they can be moved to the [facebook](https://huggingface.co/facebook) organization after merging.
- The model code can be adapted from XLM-RoBERTa. Separate code is required due to the language adapters and the pre-norm.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- text models: @ArthurZucker and @younesbelkada

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This PR is now ready for review.

Uploaded models:
- https://huggingface.co/jvamvas/xmod-base
- https://huggingface.co/jvamvas/xmod-large-prenorm
- https://huggingface.co/jvamvas/xmod-base-13-125k
- https://huggingface.co/jvamvas/xmod-base-30-125k
- https://huggingface.co/jvamvas/xmod-base-30-195k
- https://huggingface.co/jvamvas/xmod-base-60-125k
- https://huggingface.co/jvamvas/xmod-base-60-265k
- https://huggingface.co/jvamvas/xmod-base-75-125k
- https://huggingface.co/jvamvas/xmod-base-75-269k

@younesbelkada Thank you for the swift code review, much appreciated!
I have now implemented your comments.

@sgugger Thanks for the review. Your suggestions have now been implemented

Can you also add the model to the `documentation_tests.txt` file to and run the doctests to be sure that they are valid?

@ArthurZucker Thanks for the code review. I have now implemented the changes you requested.

I agree that the models should be moved to the [facebook](https://huggingface.co/facebook) organization but do not have the permissions to do so.


About moving the weights, I think I am in the org, and can help with that / ask to add you to transfer them 😉 
Looks very good, almost there! 🚀 

Hi @ArthurZucker, thanks for pointing out that there are missing tests in this PR.
Unfortunately, I have not been able to figure out which tests are missing, exactly.

As of now, there are the following tests:
- `tests.models.xmod.test_modeling_xmod.XmodModelTest` – checks that there are no errors when calling the methods of `XmodFor...`, including `model.generate()`
- `tests.models.xmod.test_modeling_xmod.XmodModelIntegrationTest` – checks that the output of the pre-trained models [jvamvas/xmod-base](https://huggingface.co/jvamvas/xmod-base) and [jvamvas/xmod-large-prenorm](https://huggingface.co/jvamvas/xmod-large-prenorm) is identical to the corresponding Fairseq models.

Could you please clarify which tests need to be added still?

Hey! Thanks for bearing with me. 
- What is there but should not:  a pipeline test inside the `test_modeling` file
- The missing tests : 
Something like what we have in opt , which will be part of the tests.models.xmod.test_modeling_xmod.XmodModelIntegrationTest. You can also have a `class XmodGenerationTest(unittest.TestCase):`
A sample test is the following.
```python 
    def test_batch_generation(self):
        model_id = "facebook/opt-350m"

        tokenizer = GPT2Tokenizer.from_pretrained(model_id)
        model = OPTForCausalLM.from_pretrained(model_id)
        model.to(torch_device)

        tokenizer.padding_side = "left"

        # use different length sentences to test batching
        sentences = [
            "Hello, my dog is a little",
            "Today, I",
        ]

        inputs = tokenizer(sentences, return_tensors="pt", padding=True)
        input_ids = inputs["input_ids"].to(torch_device)

        outputs = model.generate(
            input_ids=input_ids,
            attention_mask=inputs["attention_mask"].to(torch_device),
        )

        inputs_non_padded = tokenizer(sentences[0], return_tensors="pt").input_ids.to(torch_device)
        output_non_padded = model.generate(input_ids=inputs_non_padded)

        num_paddings = inputs_non_padded.shape[-1] - inputs["attention_mask"][-1].long().sum().cpu().item()
        inputs_padded = tokenizer(sentences[1], return_tensors="pt").input_ids.to(torch_device)
        output_padded = model.generate(input_ids=inputs_padded, max_length=model.config.max_length - num_paddings)

        batch_out_sentence = tokenizer.batch_decode(outputs, skip_special_tokens=True)
        non_padded_sentence = tokenizer.decode(output_non_padded[0], skip_special_tokens=True)
        padded_sentence = tokenizer.decode(output_padded[0], skip_special_tokens=True)

        expected_output_sentence = [
            "Hello, my dog is a little bit of a dork.\nI'm a little bit",
            "Today, I was in the middle of a conversation with a friend about the",
        ]
        self.assertListEqual(expected_output_sentence, batch_out_sentence)
        self.assertListEqual(batch_out_sentence, [non_padded_sentence, padded_sentence])
```
Does that make sense? 😉 


The CI tests are broken but it is not your fault ! We are going to have to wait until the basic docker properly runs, but the added test looks good 😉 

hi @jvamvas !
For the code quality tests just need to rebase with `main` and run:
```
pip install --upgrade -e .["quality"]
```
Then run the usual `make style` or `make fixup`

@younesbelkada Sorry about the bad rebase. On the plus side, the tests are now passing again :tada: 

Yeah hahah. Do you think you can reset, then rebase instead of merge? 😉 


@ArthurZucker Done. The failing test is not related to this PR

Great work! Thanks for working on this model! 🥳