RoBERTa: model conversion, inference, tests 🔥
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=h1) Report
> Merging [#964](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/7729ef738161a0a182b172fcb7c351f6d2b9c50d?src=pr&el=desc) will **increase** coverage by `0.43%`.
> The diff coverage is `84.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #964      +/-   ##
==========================================
+ Coverage   79.16%   79.59%   +0.43%     
==========================================
  Files          38       42       +4     
  Lines        6406     6845     +439     
==========================================
+ Hits         5071     5448     +377     
- Misses       1335     1397      +62
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_xlnet.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbmV0LnB5) | `88.99% <100%> (+0.87%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `95.28% <100%> (+0.13%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_xlm.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3hsbS5weQ==) | `83.73% <100%> (+1.68%)` | :arrow_up: |
| [...ytorch\_transformers/tests/tokenization\_xlm\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbV90ZXN0LnB5) | `97.72% <100%> (+0.5%)` | :arrow_up: |
| [...torch\_transformers/tests/tokenization\_bert\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX2JlcnRfdGVzdC5weQ==) | `98.66% <100%> (+0.15%)` | :arrow_up: |
| [...orch\_transformers/tests/tokenization\_xlnet\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3hsbmV0X3Rlc3QucHk=) | `97.91% <100%> (+0.41%)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `86.12% <66.66%> (-0.2%)` | :arrow_down: |
| [pytorch\_transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfcm9iZXJ0YS5weQ==) | `73.52% <73.52%> (ø)` | |
| [...ytorch\_transformers/tests/modeling\_roberta\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfcm9iZXJ0YV90ZXN0LnB5) | `78.81% <78.81%> (ø)` | |
| [...ch\_transformers/tests/tokenization\_roberta\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3JvYmVydGFfdGVzdC5weQ==) | `92.15% <92.15%> (ø)` | |
| ... and [7 more](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=footer). Last update [7729ef7...c4ef103](https://codecov.io/gh/huggingface/pytorch-transformers/pull/964?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I think RoBERTa is missing in `__init__.py`, so it can't be imported :(

Would be nice with a modified lm pretraining script to support RoBERTa (ie both removing the NSP task and adding dynamic masking). I might do it in next week.

@julien-c Does RoBERTa uses token_type_embeddings or token_type_ids as an input? It looks like it doesn't use because token type embeddings matrix has only one row with zeros inside. Am I right?

@avostryakov You're right.

@julien-c I modified 
MODEL_CLASSES = {...
'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer)} in run_glue.py and it started to train with a parameter "--model_type roberta". I think you can modify run_glue.py too to have an example of roberta usage.

@avostryakov Yes!! I was about to add this indeed.

Thanks for this! It would be helpful with entries in `modeling_auto` and `tokenization_auto` as well (just remember to check for `'roberta' in model_name` before `'bert' in model_name` ;) )