test suite fails due to pytorch bug in torch.seed
Author: stas00
Pull request: no

# 🐛 Bug

## Information

This is on a dual-gpu machine.

Almost all tests/test_modeling_reformer.py sub-tests fail with: 

```
    def cb():
        for i in range(device_count()):
            default_generator = torch.cuda.default_generators[i]
>           default_generator.manual_seed(seed)
E           RuntimeError: Overflow when unpacking long
```

when run after any test_multigpu_data_parallel_forward sub-test, e.g.:

`python -m pytest -n 1 --dist=loadfile -v    tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward  tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs`

**The failure gets triggered here:**
```
transformers/modeling_reformer.py:1102: in _init_attention_seed
    self.attention_seed = int((torch.seed() % sys.maxsize))
```

I reduced the failing sequence of code to this:

```
# test.py
import torch
print(f"Torch version: {torch.__version__}")

x = torch.tensor(data=[[1,2],[3,4]], dtype=torch.long, device=None)
x = x.to('cuda:0')
seed = torch.seed()
```

```
$ python tests/test.py
Torch version: 1.5.1

Traceback (most recent call last):
  File "tests/test.py", line 10, in <module>
    seed = torch.seed()
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/random.py", line 45, in seed
    torch.cuda.manual_seed_all(seed)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 111, in manual_seed_all
    _lazy_call(cb)
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/__init__.py", line 99, in _lazy_call
    callable()
  File "/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py", line 109, in cb
    default_generator.manual_seed(seed)
RuntimeError: Overflow when unpacking long
```

It fails about 75% of time.

It happens after moving the tensor to gpu.

This seems to be related to this [pytorch bug](https://github.com/pytorch/pytorch/issues/33546), albeit somewhat different sequence of code. 

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

Full trace of failing sub-tests (one of them):

```
python -m pytest -n 1 --dist=loadfile -v tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs
====================================================================== test session starts =======================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /home/stas/anaconda3/envs/main/bin/python
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification/.hypothesis/examples')
rootdir: /mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
[gw0] linux Python 3.7.5 cwd: /mnt/nvme1/code/huggingface/transformers-FlaubertForTokenClassification
[gw0] Python 3.7.5 (default, Oct 25 2019, 15:51:11)  -- [GCC 7.3.0]
gw0 [2]
scheduling tests via LoadFileScheduling

tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward 
[gw0] [ 50%] PASSED tests/test_modeling_electra.py::ElectraModelTest::test_multigpu_data_parallel_forward 
tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs 
[gw0] [100%] FAILED tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs 

============================================================================ FAILURES ============================================================================
_______________________________________________________ ReformerLocalAttnModelTest.test_attention_outputs ________________________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <tests.test_modeling_reformer.ReformerLocalAttnModelTest testMethod=test_attention_outputs>

    def test_attention_outputs(self):
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
        seq_len = getattr(self.model_tester, "seq_length", None)
        decoder_seq_length = getattr(self.model_tester, "decoder_seq_length", seq_len)
        encoder_seq_length = getattr(self.model_tester, "encoder_seq_length", seq_len)
        decoder_key_length = getattr(self.model_tester, "key_length", decoder_seq_length)
        encoder_key_length = getattr(self.model_tester, "key_length", encoder_seq_length)
        chunk_length = getattr(self.model_tester, "chunk_length", None)
        if chunk_length is not None and hasattr(self.model_tester, "num_hashes"):
            encoder_seq_length = encoder_seq_length * self.model_tester.num_hashes
    
        for model_class in self.all_model_classes:
            inputs_dict["output_attentions"] = True
            inputs_dict["output_hidden_states"] = False
            model = model_class(config)
            model.to(torch_device)
            model.eval()
            with torch.no_grad():
>               outputs = model(**self._prepare_for_class(inputs_dict, model_class))

tests/test_modeling_common.py:149: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1623: in forward
    output_attentions=output_attentions,
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1371: in forward
    output_attentions,
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1267: in forward
    output_attentions=output_attentions,
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/nn/modules/module.py:550: in __call__
    result = self.forward(*input, **kwargs)
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1138: in forward
    self._init_attention_seed()
../transformers-XLMForTokenClassification/src/transformers/modeling_reformer.py:1102: in _init_attention_seed
    self.attention_seed = int((torch.seed() % sys.maxsize))
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/random.py:45: in seed
    torch.cuda.manual_seed_all(seed)
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py:111: in manual_seed_all
    _lazy_call(cb)
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/__init__.py:99: in _lazy_call
    callable()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def cb():
        for i in range(device_count()):
            default_generator = torch.cuda.default_generators[i]
>           default_generator.manual_seed(seed)
E           RuntimeError: Overflow when unpacking long

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/torch/cuda/random.py:109: RuntimeError
======================================================================== warnings summary ========================================================================
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(args, collections.Mapping), '{} args must be a dict with argument names as keys.'.format(name)

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import OrderedDict, Sequence, defaultdict

-- Docs: https://docs.pytest.org/en/latest/warnings.html
==================================================================== short test summary info =====================================================================
FAILED tests/test_modeling_reformer.py::ReformerLocalAttnModelTest::test_attention_outputs - RuntimeError: Overflow when unpacking long
============================================================ 1 failed, 1 passed, 4 warnings in 5.57s =======================================
```
```


Comments:
A fix has been just applied here:
https://github.com/pytorch/pytorch/commit/5edd9aa95a8a73e940185f8448e7db05394ce6fe
will re-test with the nightly

I confirmed that this pytorch bug has been fixed in nightly and the tests don't fail anymore.