[ci] Run all tests on (self-hosted) GPU
Author: julien-c
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=h1) Report
> Merging [#3020](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e8ce63ff2163259276fc0a4a2f35b836fe9f4aa0?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3020/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3020      +/-   ##
==========================================
- Coverage   77.25%   77.21%   -0.04%     
==========================================
  Files          98       98              
  Lines       16040    16040              
==========================================
- Hits        12392    12386       -6     
- Misses       3648     3654       +6
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3020/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.71% <0%> (-0.86%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3020/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.38% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=footer). Last update [e8ce63f...2a48145](https://codecov.io/gh/huggingface/transformers/pull/3020?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


**Update:** 
The remaining failing tests:

- [ ] Test_doc_samples: multilingual.rst and modeling_flaubert.py (@LysandreJik)
- [ ] test_modeling_auto.py::AutoModelTest::test_model_for_pretraining_from_pretrained (@thomwolf  https://github.com/huggingface/transformers/commit/0e31e06a75b0022171056e51c8b5d53078ac5170)
- [x] test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_masked_lm (@julien-c https://github.com/huggingface/transformers/commit/9d0603148bc34255fad0cad73ce438ecd7306322)
- [ ] test_tokenization_roberta.py::RobertaTokenizationTest::test_sequence_builders (@LysandreJik  https://github.com/huggingface/transformers/commit/634a3172d869e2ff772b2e0813169641ca9e6cc5)
- [ ] test_tokenization_xlm_roberta.py:: XLMRobertaTokenizationIntegrationTest::test_tokenization_base_hard_symbols (@patrickvonplaten https://github.com/huggingface/transformers/commit/c913eb9c3894b4031dc059d22b42e38a5fcef989)


> * test_tokenization_xlm_roberta.py::XLMRobertaTokenizationIntegrationTest::test_tokenization_base_hard_symbols (@patrickvonplaten [c913eb9](https://github.com/huggingface/transformers/commit/c913eb9c3894b4031dc059d22b42e38a5fcef989))

I think this test currently fails because there is a problem with the xlm_roberta_tokenizer . In general so far there are no tests at all for the xlm_roberta_tokenizer . I can try to add those (will prob need a bit help from @mfuntowicz and @LysandreJik) 


Update on my failing unit (integration) test on RoBERTa: the max absolute diff between expected and actual output is `0.004` whereas it used to be under `1e-3` (both CPU and cuda) – should I dive in to why this changed, or should i just lazily bump up to tolerance?

(The same integration test without the maskedLM head still passes with `1e-5` abs diff)

Could this be due to the bias? https://github.com/huggingface/transformers/pull/2958

It could have been computed twice when creating the integration test.

I'm not sure how to exactly get the expected logits from the fairseq **LM** Models. It's very easy to get the last embeddings for verification of the no **LM** model via:

```
import torch
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
roberta.eval()  # disable dropout (or leave in train mode to finetune)
last_layer_embeddings = roberta.extract_features(input_ids)   # shape [1, seq_len, 1024]
```

The integration test that were implemented for Roberta LMHeadModels did they correspond to the original weights? 


@patrickvonplaten https://github.com/huggingface/transformers/blob/53ce3854a16ad2a715bc6ac8af3e30c18b5a1d11/tests/test_modeling_roberta.py#L324 is the test (I think)

Ok found a resolution strategy with @LysandreJik, will push fix soon.

Getting closer:
> ===== 6 failed, 939 passed, 32 skipped, 84 warnings in 1508.72s (0:25:08) ======