RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

username: Jorge_Garcia
cooked: <p>Hello,</p>
<p>I’m getting “RuntimeError: Expected a ‘cuda’ device type for generator but found ‘cpu’” error when I try to iterate over my dataloader created as follows:</p>
<pre><code class="lang-auto">transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize(0.5, 0.5),
                           ])
dataset = dset.MNIST(root=dataroot, train=True, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)
</code></pre>
<p>The problem solves if I turn shuffle to False but I would like to mantain it to True. Solutions I’ve found imply to change some pytorch code but I would like to avoid it.</p>
<p>Thanks!</p>

username: srishti-git1110
cooked: <p>Hi,<br>
Can you please see if this works -</p>
<pre><code class="lang-auto">torch.utils.data.DataLoader(
    ...,
    generator=torch.Generator(device='cuda'),
)
</code></pre>

username: Jorge_Garcia
cooked: <p>It works!</p>
<p>Thank you very much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>I don’t understand what the issue in the posted code snippet was as it’s working for me locally and it’s not even using the GPU, so which part of the code raised the error?</p>
<p><a class="mention" href="/u/srishti-git1110">@srishti-git1110</a> have you seen this error before being raised in a <code>DataLoader</code> using the CPU only?</p>

username: Jorge_Garcia
cooked: <p>I’m sorry I didn’t explain myself properly. The problem only arised when using GPU as device and worked fine when working on CPU.</p>

username: srishti-git1110
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
No, I’ve never run into such an error while using CPU.<br>
Consequently, yes, the posted code doesn’t seem to produce any error.</p>
<p>I just assumed the OP is facing the error when using GPU, hence posted that as the solution. Sorry for not being clear - should’ve mentioned it there.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="srishti-git1110" data-post="6" data-topic="161463">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/srishti-git1110/48/52515_2.png" class="avatar"> srishti-git1110:</div>
<blockquote>
<p>Sorry for not being clear - should’ve mentioned it there.</p>
</blockquote>
</aside>
<p>Not at all. My post wasn’t any criticism as you’ve guessed it perfectly right and <a class="mention" href="/u/jorge_garcia">@Jorge_Garcia</a> clarified that indeed the GPU was used.</p>
<p>I was just concerned if this might be a known issue of raising CUDA errors when a CPU-only <code>DataLoader</code> is used, but it turns out the code was missing some parts. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

username: Dmitriy_Lamzin
cooked: <p>Hello here. I faced the same issue. But looks like the suggestion above will not work:</p>
<pre><code class="lang-auto">/local_disk0/.ephemeral_nfs/envs/pythonEnv-4aa41058-b8da-44ab-8b2d-453f51f9b1ec/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    575             shared_rng = torch.Generator()
    576             shared_rng.manual_seed(self._shared_seed)
--&gt; 577             self._dataset = torch.utils.data.graph_settings.apply_random_seed(self._dataset, shared_rng)
    578         self._dataset_kind = loader._dataset_kind
    579         self._IterableDataset_len_called = loader._IterableDataset_len_called

/local_disk0/.ephemeral_nfs/envs/pythonEnv-4aa41058-b8da-44ab-8b2d-453f51f9b1ec/lib/python3.8/site-packages/torch/utils/data/graph_settings.py in apply_random_seed(datapipe, rng)
    151 
    152     for pipe in random_datapipes:
--&gt; 153         random_seed = int(torch.empty((), dtype=torch.int64).random_(generator=rng).item())
    154         pipe.set_seed(random_seed)
    155 

/local_disk0/.ephemeral_nfs/envs/pythonEnv-4aa41058-b8da-44ab-8b2d-453f51f9b1ec/lib/python3.8/site-packages/torch/utils/_device.py in __torch_function__(self, func, types, args, kwargs)
     60         if func in _device_constructors() and kwargs.get('device') is None:
     61             kwargs['device'] = self.device
---&gt; 62         return func(*args, **kwargs)
     63 
     64 # NB: This is directly called from C++ in torch/csrc/Device.cpp

RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'
</code></pre>
<p>Basically here you can see that the generator is initilised inside function right before generating stuff</p>
<pre><code class="lang-auto">453f51f9b1ec/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    575             shared_rng = torch.Generator()
    576             shared_rng.manual_seed(self._shared_seed)
--&gt; 577             self._dataset = torch.utils.data.graph_settings.apply_random_seed(self._dataset, shared_rng)
</code></pre>
<p>In my case it looks more like a bug in the back-compatibility of data pipes from torchdata with DataLoader</p>

username: jamestang0219
cooked: <p>same problem, it seems the generator will always be initialized using cpu</p>

username: Nikita_Artemenko
cooked: <p>Alright.<br>
If you define loader like that:</p>
<pre><code class="lang-auto">    dataloader = DataLoader(
        image_dataset,
        batch_size=batch_size,
        num_workers=num_workers,
        pin_memory=True,
        shuffle=True,
        generator=torch.Generator(device='cuda:0'),
    )
</code></pre>
<p>And try to use <code>next(iter(dataloader))</code><br>
You get Error:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py in __iter__(self)
    163         else:
    164             for _ in range(self.num_samples // n):
--&gt; 165                 yield from map(int, torch.randperm(n, generator=generator).numpy())
    166             yield from map(int, torch.randperm(n, generator=generator)[:self.num_samples % n].numpy())
    167 

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>Seems like instead of numpy there should be .cpu() method in sampler.py</p>
<p>Otherwise, if we don’t specify generator or use ‘cpu’ device it produces error (which is the topic of discussion).</p>
<p>So, what should I do to solve the problem?</p>

username: fdila
cooked: <pre><code class="lang-auto">/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py in __iter__(self)
    163         else:
    164             for _ in range(self.num_samples // n):
--&gt; 165                 yield from map(int, torch.randperm(n, generator=generator).numpy())
    166             yield from map(int, torch.randperm(n, generator=generator)[:self.num_samples % n].numpy())
    167 

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>I’m having the same error, did you manage to find a solution?</p>

username: sc0v0ne
cooked: <p>Thank you very much!!!</p>

username: Geremia
cooked: <p>Thanks, that worked.<br>
I wonder why <code>torch.set_default_device(torch.device('cuda'))</code> didn’t tell <code>torch.utils.data.DataLoader</code> to use <code>cuda</code>. I suppose it only works for setting what tensors should use.<br>
Is there an analogous command to <code>set_default_device()</code> but for <code>DataLoader</code>s?</p>

username: raghavm1
cooked: <p>I’m getting the same error but this doesn’t seem to work for me <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Not sure why. I’m using PyTorch 2.3.1. Could this be related to the version of the library?</p>

username: DeerFreak
cooked: <p>I think you might need to update cuda. This fixed it for me</p>

username: raghavm1
cooked: <p>Just solved this issue by removing the line -</p>
<p><code>torch.set_default_device('cuda')</code></p>

username: ptrblck
cooked: <p>The used CUDA version (shipped as part of the PyTorch binaries or a local CUDA toolkit if you’ve built PyTorch from source) won’t change the device in your PyTorch code.</p>

username: xuanzera
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="17" data-topic="161463">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>rt of the PyTorch binaries or a local CUDA toolkit if you’ve built PyTorch from source) won’t change t</p>
</blockquote>
</aside>
<p>I faced the same problem. Delete this</p>
<pre><code class="lang-auto">torch.set_default_device('cuda') 
</code></pre>
<p>(I found this in main.py)<br>
I think you can find this somewhere!!!</p>
<pre><code class="lang-auto">    generator = torch.Generator(device="cuda").manual_seed(0)
    train_set, val_set = random_split(dataset, [n_train, n_val], generator=generator)
    loader_args = dict(
        batch_size = 1,
        num_workers = os.cpu_count(),
        pin_memory = True if DEVICE.type == 'cuda' else False,
        generator = generator
    )
    train_loader = DataLoader(train_set, shuffle=True, **loader_args)
    val_loader = DataLoader(val_set, shuffle=True, **loader_args)
</code></pre>
<p>Error:TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>
<pre><code class="lang-auto">    generator = torch.Generator(device="cpu").manual_seed(0)
    train_set, val_set = random_split(dataset, [n_train, n_val], generator=generator)
    loader_args = dict(
        batch_size = 1,
        num_workers = os.cpu_count(),
        pin_memory = True if DEVICE.type == 'cuda' else False,
        generator = generator
    )
    train_loader = DataLoader(train_set, shuffle=True, **loader_args)
    val_loader = DataLoader(val_set, shuffle=True, **loader_args)
</code></pre>
<p>TypeError: can’t convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.</p>