Add support for XLM-RoBERTa
Author: stefan-it
Pull request: yes

Hi,

this model adds support for the recently released XLM-RoBERTa model from the Facebook AI team.

XLM-RoBERTa is described in the ["Unsupervised Cross-lingual Representation Learning at Scale](https://arxiv.org/abs/1911.02116) paper from Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov.

The model itself is integrated into `fairseq` library and weights for the base and large XLM-R are available, see example [here]().

## Results

### NER

This PR also extends the `run_ner` script to support XLM-R. Results for NER (CoNLL datasets):

#### Base model

| Model      | English             | Dutch               | Spanish             | German              | Avg.
| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | -------------------
| Paper      |   -   (dev) / 91.95 |   -   (dev) / 91.21 |   -   (dev) / 88.46 |   -   (dev) / 83.65 |   -   (dev) / 88.82
| Reproduced | 95.31 (dev) / 91.20 | 91.66 (dev) / 91.37 | 85.23 (dev) / 88.15 | 87.11 (dev) / 84.02 | 89.83 (dev) / 88.69

#### Large model

| Model      | English             | Dutch               | Spanish             | German              | Avg.
| ---------- | ------------------- | ------------------- | ------------------- | ------------------- | -------------------
| Paper      |   -   (dev) / 92.74 |   -   (dev) / 93.25 |   -   (dev) / 89.04 |   -   (dev) / 85.53 |   -   (dev) / 90.14
| Reproduced | 96.84 (dev) / 92.80 | 94.02 (dev) / 94.41 | 88.94 (dev) / 89.30 | 88.60 (dev) / 86.04 | 92.10 (dev) / 90.64

Parameters used for reproducing the paper results: 20 training epochs with a learning rate of `5.0e-6` and a batch size of 16. Only one run is reported here.

## Tasks

* [x] Upload model to 🤗/ Transformers S3
* [x] Add support for base model (convert script needs to be adjusted)
* [x] Report results for NER (CoNLL datasets)
* [x] Add XLM-R to `Auto*` interfaces
* [x] Check tokenization methods

Comments:
@stefan-it Following the merge of #1959, you should not have to duplicate the weights conversion script anymore. It should work out of the box, `fairseq.XLMRModel` being a subclass of `fairseq.RobertaModel`.

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=h1) Report
> Merging [#2189](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2f1c745cded91b2f6cfed5b502ea5cbd7d6b9ac7?src=pr&el=desc) will **increase** coverage by `1.12%`.
> The diff coverage is `45.45%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2189/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2189      +/-   ##
==========================================
+ Coverage    80.2%   81.33%   +1.12%     
==========================================
  Files         125      125              
  Lines       18444    18458      +14     
==========================================
+ Hits        14793    15012     +219     
+ Misses       3651     3446     -205
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/configuration\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25fYmVydC5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3hsbV9yb2JlcnRhLnB5) | `100% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.75% <ø> (ø)` | :arrow_up: |
| [transformers/tokenization\_distilbert.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9kaXN0aWxiZXJ0LnB5) | `100% <ø> (ø)` | :arrow_up: |
| [transformers/configuration\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2NvbmZpZ3VyYXRpb25feGxtX3JvYmVydGEucHk=) | `100% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_tf\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2JlcnQucHk=) | `96.31% <ø> (ø)` | :arrow_up: |
| [transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9iZXJ0LnB5) | `96.38% <ø> (ø)` | :arrow_up: |
| [transformers/tokenization\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl94bG1fcm9iZXJ0YS5weQ==) | `36.92% <0%> (-0.58%)` | :arrow_down: |
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `71.42% <100%> (ø)` | :arrow_up: |
| [transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2F1dG8ucHk=) | `36.36% <28.57%> (-0.53%)` | :arrow_down: |
| ... and [9 more](https://codecov.io/gh/huggingface/transformers/pull/2189/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=footer). Last update [2f1c745...3376adc](https://codecov.io/gh/huggingface/transformers/pull/2189?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@julien-c Thanks for that hint! I tested the script (#1959) and it's working with XLM-R (training was sucessful). I'm going to test the base model now 😅

This is really awesome @stefan-it!
Merging now!

@stefan-it Hi, how to save the XLM-Roberta model? I tried `torch.save` for the model but reported `*** TypeError: can't pickle SwigPyObject objects`