Error when using distributedsampler

CgYoung: <p>What’s the meaning of <code>RuntimeError: Expected a 'N2at13CUDAGeneratorE' but found 'PN2at9GeneratorE'</code></p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "train_cosine_iterations_distributed.py", line 335, in &lt;module&gt;
    batch_iterator = iter(train_loader)
  File "/home/ycg/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)
  File "/home/ycg/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 584, in __init__
    self._put_indices()
  File "/home/ycg/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 646, in _put_indices
    indices = next(self.sample_iter, None)
  File "/home/ycg/anaconda3/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 160, in __iter__
    for idx in self.sampler:
  File "/home/ycg/anaconda3/lib/python3.6/site-packages/torch/utils/data/distributed.py", line 45, in __iter__
    indices = torch.randperm(len(self.dataset), generator=g).tolist()
RuntimeError: Expected a 'N2at13CUDAGeneratorE' but found 'PN2at9GeneratorE'
</code></pre>

CgYoung: <p>Is there something wrong with torch.Generator()?</p>
<pre><code class="lang-auto">def __iter__(self):
    # deterministically shuffle based on epoch
    g = torch.Generator()
    g.manual_seed(self.epoch)
    indices = torch.randperm(len(self.dataset), generator=g).tolist()

    # add extra samples to make it evenly divisible
    indices += indices[:(self.total_size - len(indices))]
    assert len(indices) == self.total_size

    # subsample
    indices = indices[self.rank:self.total_size:self.num_replicas]
    assert len(indices) == self.num_samples

    return iter(indices)
</code></pre>

smth: <p>I tried to reproduce your issue with:</p>
<pre><code class="lang-auto">import torch

g = torch.Generator()
g.manual_seed(1)
indices = torch.randperm(100, generator=g).tolist()
</code></pre>
<p>It ran without error (I am on PyTorch 1.0.0).<br>
Can you try to give me a reproduction so that I can help figure out what’s going on?</p>

CgYoung: <p><code>len(self.dataset) == 118,287</code>, I run the code to train on MS COCO dataset</p>

bdusell: <p>I’m running into this error too, but only when using CUDA. Here’s a minimal script to reproduce:</p>
<pre><code class="lang-python">import torch
device = torch.device('cuda')
generator = torch.manual_seed(123)
x = torch.zeros(10, device=device)
x.uniform_(-1.0, 1.0, generator=generator)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">$ python bug.py 
Traceback (most recent call last):
  File "bug.py", line 5, in &lt;module&gt;
    x.uniform_(-1.0, 1.0, generator=generator)
RuntimeError: Expected a 'N2at13CUDAGeneratorE' but found 'PN2at9GeneratorE'
</code></pre>
<p>Same thing happens if I use <code>torch.Generator</code>. By the way, is <code>torch.Generator</code> documented anywhere?</p>

sftb644517236: <p>Bro, Have you solve this problem? I meet this error when I use   torch.nn.parallel.DistributedDataParallel.<br>
the problem is from sampler = DistributedSampler(dataset)  and<br>
data.DataLoader(dataset,…, sampler=sampler)</p>
<p>Here is log.</p>
<p>Traceback (most recent call last):<br>
File “train.py”, line 152, in <br>
sampler=sampler))<br>
File “/home/xdjf/.conda/envs/py36torch041/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 501, in <strong>iter</strong><br>
return _DataLoaderIter(self)<br>
File “/home/xdjf/.conda/envs/py36torch041/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 297, in <strong>init</strong><br>
self._put_indices()<br>
File “/home/xdjf/.conda/envs/py36torch041/lib/python3.6/site-packages/torch/utils/data/dataloader.py”, line 345, in _put_indices<br>
indices = next(self.sample_iter, None)<br>
File “/home/xdjf/.conda/envs/py36torch041/lib/python3.6/site-packages/torch/utils/data/sampler.py”, line 138, in <strong>iter</strong><br>
for idx in self.sampler:<br>
File “/home/xdjf/.conda/envs/py36torch041/lib/python3.6/site-packages/torch/utils/data/distributed.py”, line 42, in <strong>iter</strong><br>
indices = list(torch.randperm(len(self.dataset), generator=g))<br>
RuntimeError: Expected a ‘N2at13CUDAGeneratorE’ but found ‘PN2at9GeneratorE’</p>