DistributedSampler — *** RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

shawnbzhang: <p>Performing distributed training, I have the following code like this:</p>
<pre><code class="lang-auto">training_sampler = DistributedSampler(training_set, num_replicas=2, rank=0)
training_generator = data.DataLoader(training_set, **params, sampler=training_sampler)
for x, y, z in training_generator:  # Error occurs here.
    ...
</code></pre>
<p>Overall, I get the following message:</p>
<pre><code class="lang-auto">-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/ubuntu/VC/ppg_training_extraction/ppg_training_scripts/train_ASR_trim_scp.py", line 336, in train
    for local_batch_src, local_batch_tgt, lengths in dataloaders[phase]:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 352, in __iter__
    return self._get_iterator()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 294, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 827, in __init__
    self._reset(loader, first_iter=True)
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 857, in _reset
    self._try_put_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 1091, in _try_put_index
    index = self._next_index()
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 427, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 227, in __iter__
    for idx in self.sampler:
  File "/home/ubuntu/anaconda3/lib/python3.7/site-packages/torch/utils/data/distributed.py", line 97, in __iter__
    indices = torch.randperm(len(self.dataset), generator=g).tolist()  # type: ignore
RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'
</code></pre>
<p>Now at that line, I ran the following instructions in pdb:</p>
<pre><code class="lang-auto">(Pdb) g = torch.Generator()
(Pdb) g.manual_seed(0)
&lt;torch._C.Generator object at 0x7ff7f8143110&gt;
(Pdb) indices = torch.randperm(4556, generator=g).tolist()
(Pdb) indices = torch.randperm(455604, generator=g).tolist()
*** RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'
</code></pre>
<p>Why am I getting the runtime error when the upperbound integer is high, but not when it’s low enough?</p>
<p>Note, I ran on a clean Python session and found</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; g = torch.Generator()
&gt;&gt;&gt; g.manual_seed(0)
&lt;torch._C.Generator object at 0x7f9d2dfb39f0&gt;
&gt;&gt;&gt; indices = torch.randperm(455604, generator=g).tolist()
</code></pre>
<p>that this worked fine. Is it some configuration in how I’m handling distributed training among multiple GPUs? Any sort of insights would be appreciated!</p>

shawnbzhang: <p>So I found out why this error was occurring. It was because earlier in my code, I had the following line:</p>
<pre><code class="lang-auto">torch.set_default_tensor_type('torch.cuda.FloatTensor')
</code></pre>

nikolasborrel: <p>If you want to use CUDA, the argument passed to the generator should be ‘cuda’</p>
<pre><code class="lang-auto">generator = torch.Generator('cuda')
</code></pre>
<p>The generator is executing on the CPU per default (see <a href="https://pytorch.org/docs/stable/generated/torch.Generator.html" class="inline-onebox" rel="noopener nofollow ugc">Generator — PyTorch 1.11.0 documentation</a>)</p>