XLM-R XL/XXL
Author: sbmaruf
Pull request: no
Labels: name: New model

# 🌟 New model addition

## Model description

The larger version of XLMR.

[Source](https://github.com/pytorch/fairseq/tree/master/examples/xlmr)

Model | Description | #params | vocab size | Download
---|---|---|---|---
`xlmr.xl` | XLM-R (`layers=36, model_dim=2560`) | 3.5B | 250k | [xlm.xl.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xlmr/xlmr.xl.tar.gz)
`xlmr.xxl` | XLM-R (`layers=48, model_dim=4096`) | 10.7B | 250k | [xlm.xxl.tar.gz](https://dl.fbaipublicfiles.com/fairseq/models/xlmr/xlmr.xxl.tar.gz)



<!-- Important information -->

## Open source status

* [x] the model implementation is available: (give details) -> Already available in huggingface
* [x] the model weights are available: (give details) -> link + source provided.
* [ ] who are the authors: (mention them, if possible by @gh-username)


Comments:
I believe @stefan-it is on it :sunglasses: 

I successfully converted xlmr.xl to huggingface model.
```
torch.Size([1, 11, 250880]) torch.Size([1, 11, 250880])
max_absolute_diff = 4.482269287109375e-05
Do both models output the same tensors? 🔥
Saving model to converted_xlmr.xl2
Configuration saved in converted_xlmr.xl2/config.json
Model weights saved in converted_xlmr.xl2/pytorch_model.bin
```
is there anything I can do to help?

I'm middle of converting xxl model too.

While processing xxl it produces output with absolute error of 0.003273.
Is it possible because of the model size(10.7B)?
```
torch.Size([1, 11, 250880]) torch.Size([1, 11, 250880])
max_absolute_diff = 0.00327301025390625
Do both models output the same tensors? 💩
```

@Soonhwan-Kwon  Did you able to solve the issue? I don't have much experience. But I think the good error margin is `< 1e-6`. 

@sbmaruf I found out that model conversion in fairseq ver 0.10.2 produced wrong result on both side, and it made min absolute diff small. @stefan-it told that he made it work and it is a great news! https://github.com/huggingface/transformers/pull/12082

I've managed to get the same value and pushed PR in @stefan-it's repo.
```
our_output
tensor([[[ 4.9569e+01, -1.0970e+00,  3.6279e+01,  ...,  1.3821e+00,
           1.2402e+00,  1.0905e+01],
         [ 8.5117e+00, -9.9209e-02,  3.3087e+01,  ...,  1.4223e+00,
           1.5715e+00,  1.1260e+01],
         [ 9.4228e+00,  1.8814e-01,  2.4515e+01,  ...,  2.4245e+00,
           1.0935e+00,  1.1929e+01],
         ...,
         [ 8.8886e+00, -1.7367e-02,  2.5994e+01,  ...,  1.9401e+00,
           1.8700e+00,  1.2002e+01],
         [ 9.7415e+00, -2.6768e-01,  3.2220e+01,  ...,  1.9813e+00,
           1.3128e+00,  9.6978e+00],
         [ 1.6002e+01,  1.6512e+00,  5.7907e+01,  ...,  1.9653e+00,
           1.3225e+00,  1.8848e+01]]], grad_fn=<AddBackward0>)
their_output
tensor([[[ 4.9569e+01, -1.0970e+00,  3.6280e+01,  ...,  1.3821e+00,
           1.2402e+00,  1.0905e+01],
         [ 8.5117e+00, -9.9211e-02,  3.3087e+01,  ...,  1.4223e+00,
           1.5715e+00,  1.1260e+01],
         [ 9.4228e+00,  1.8814e-01,  2.4515e+01,  ...,  2.4245e+00,
           1.0935e+00,  1.1929e+01],
         ...,
         [ 8.8886e+00, -1.7370e-02,  2.5994e+01,  ...,  1.9401e+00,
           1.8700e+00,  1.2002e+01],
         [ 9.7415e+00, -2.6768e-01,  3.2220e+01,  ...,  1.9813e+00,
           1.3128e+00,  9.6978e+00],
         [ 1.6002e+01,  1.6512e+00,  5.7907e+01,  ...,  1.9653e+00,
           1.3225e+00,  1.8848e+01]]], grad_fn=<AddBackward0>)
```

Hi @Soonhwan-Kwon  Thanks for contributing the convertion code. Have you tested whether you could load the converted xlmr-xl or xlm-xxl  using huggingface? 

@ccclyu Yes I have tested the model and confirmed the better performance than xlmr large model in specific task. 

@Soonhwan-Kwon Glad to know that.  I have successfully converted the parameters using your PR https://github.com/stefan-it/transformers/pull/1 but it may have minor conflict with the current transformer codebase.  

By the way, how do you load the huge model (13GB parameters for xlm-xl) using huggingface since one single GPU could not load the whole model?  Did you use DeepSpeed for model parallels ? 

 

 

@ccclyu There are many options and deepspeed is the one option as you mentioned, and you can freeze layers to reduce gpu memory usage.

In progress in https://github.com/huggingface/transformers/pull/13210 by @Soonhwan-Kwon 

is there any news about this?