Fix nn.DataParallel compatibility in PyTorch 1.5
Author: julien-c
Pull request: yes

As reported in #3936 

PyTorch 1.5 changes the handling of nn.Parameters in `DataParallel` replicas. (https://github.com/pytorch/pytorch/pull/33907). The replicas now don't have parameters() anymore.

This PR updates our `self.device` and `self.dtype` helpers to mimic `nn.Module`'s `parameters()` helper but for attributes, i.e. recursively look for an attribute of type Tensor.

Reformer and TransfoXL seem to be doing fancy things based on the module's Parameters so I didn't attempt to fix them.

Finally I'm introducing a `multigpu` CI flag. CI does not currently run on multiple GPUs so remember to run it locally.

---

Also pinging @ngimel the author of the change in PyTorch, to check if I'm doing something stupid here.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=h1) Report
> Merging [#4300](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7cb203fae4e7964e9e99400b375d660ebce765ee&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `91.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4300/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4300      +/-   ##
==========================================
- Coverage   78.21%   78.21%   -0.01%     
==========================================
  Files         120      120              
  Lines       20038    20040       +2     
==========================================
+ Hits        15673    15674       +1     
- Misses       4365     4366       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `75.73% <75.00%> (ø)` | |
| [src/transformers/modeling\_albert.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19hbGJlcnQucHk=) | `77.20% <100.00%> (ø)` | |
| [src/transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iZXJ0LnB5) | `88.82% <100.00%> (ø)` | |
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `83.66% <100.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.98% <100.00%> (+0.02%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4300/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.44% <0.00%> (-0.42%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=footer). Last update [7cb203f...7eef4f5](https://codecov.io/gh/huggingface/transformers/pull/4300?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Update: Also asked about this on shared [PyTorch Slack channel](https://app.slack.com/client/T2077MDKQ/CN963QVJB)

I don't see anything obviously wrong, but I'm not very familiar with your codebase. You are looking for tensor attributes - are you sure that you don't have any other tensor attributes that don't correspond to former parameters? 
Finally, we also have _former_parameters in pytorch, introduced here https://github.com/pytorch/pytorch/pull/36523. You may find it useful, but we can't guarantee that it will remain as a stable API. 

I think you forgot GPT-2:
`transformers/modeling_gpt2.py", line 464, in forward
    attention_mask = attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
StopIteration
`

Yes, if I remember correctly I didn't try to remove all the calls to `next(self.parameters())` in this PR – do you want to open a PR to fix this?