Repository: pytorch/ignite

Issue:
DeterministicEngine rng state on cuda if loaded checkpoint on cuda
## 🐛 Bug description

Using `engine` as `DeterministicEngine`, loading on cuda can lead to the following issue:
```
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/engine.py", line 702, in run
    return self._internal_run()
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/engine.py", line 775, in _internal_run
    self._handle_exception(e)
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/engine.py", line 469, in _handle_exception
    raise e
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/engine.py", line 743, in _internal_run
    self._setup_engine()
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/deterministic.py", line 238, in _setup_engine
    _set_rng_states(rng_states)
  File "/home/machine/.clearml/venvs-builds/3.6/lib/python3.6/site-packages/ignite/engine/deterministic.py", line 100, in _set_rng_states
    torch.set_rng_state(rng_states[1])
  File "/home/machine/miniconda3/envs/py36/lib/python3.6/site-packages/torch/random.py", line 14, in set_rng_state
    default_generator.set_state(new_state)
TypeError: expected a torch.ByteTensor, but got torch.cuda.ByteTensor
```

Code:
```
to_load = {
            'state_dict': model,
            'optimizer': optimizer,
            'lr_scheduler': lr_scheduler,
            'train_phase': engine
        }
checkpoint = torch.load(checkpoint_fp, map_location="cuda:0")
Checkpoint.load_objects(to_load=to_load, checkpoint=checkpoint)
```

To fix the issue, we have to make sure that rng tensor is on cpu (otherwise put to cpu):
https://github.com/pytorch/ignite/blob/b49a82a8fb3339f1442752a0dbf2ecd6f15cb1fa/ignite/engine/deterministic.py#L238-L240

and add appropriate test.

### Workaround

- Load on cpu
```diff
- checkpoint = torch.load(checkpoint_fp, map_location="cuda:0")
+ checkpoint = torch.load(checkpoint_fp, map_location="cpu")
```


## Environment

- PyTorch Version (e.g., 1.4): 1.7.1
- Ignite Version (e.g., 0.3.0): 0.4.4
- OS (e.g., Linux):
- How you installed Ignite (`conda`, `pip`, source):
- Python version:
- Any other relevant information:


Thanks @H4dr1en for reporting.


Discussion:

Resolving patch:
diff --git a/ignite/engine/deterministic.py b/ignite/engine/deterministic.py
--- a/ignite/engine/deterministic.py
+++ b/ignite/engine/deterministic.py
@@ -98,6 +98,10 @@ def _get_rng_states() -> List[Any]:
 
 def _set_rng_states(rng_states: List[Any]) -> None:
     random.setstate(rng_states[0])
+
+    if "cpu" not in rng_states[1].device.type:
+        rng_states[1] = rng_states[1].cpu()
+
     torch.set_rng_state(rng_states[1])
     try:
         import numpy as np


Test patch:
diff --git a/tests/ignite/engine/test_deterministic.py b/tests/ignite/engine/test_deterministic.py
--- a/tests/ignite/engine/test_deterministic.py
+++ b/tests/ignite/engine/test_deterministic.py
@@ -15,6 +15,7 @@
 from ignite.engine.deterministic import (
     DeterministicEngine,
     ReproducibleBatchSampler,
+    _set_rng_states,
     keep_random_state,
     update_dataloader,
 )
@@ -885,3 +886,12 @@ def test_dataloader_no_dataset_kind():
     dataloader = OldDataLoader(dataloader)
 
     engine.run(dataloader)
+
+
+@pytest.mark.skipif(not torch.cuda.is_available(), reason="Skip if no GPU")
+def test__set_rng_states_cuda():
+    # Checks https://github.com/pytorch/ignite/issues/2076
+
+    rng_states = [random.getstate(), torch.get_rng_state().cuda(), np.random.get_state()]
+    _set_rng_states(rng_states)
+    assert rng_states[1].device.type == "cpu"