Device error with dataloaders

username: kwijibohan
cooked: <p>Torch version <code>2.1.0.dev20230702+cu121</code></p>
<pre><code class="lang-python">import torch
from torch.utils import data

torch.set_default_device('cuda')

class NullDataset(data.Dataset):
    def __len__(self) -&gt; int:
        return 100

dataloader = data.DataLoader(NullDataset(), batch_size=64, shuffle=True, generator=torch.Generator(device='cuda'))

for data in dataloader:
    print(data)
</code></pre>
<p>gives errors at <code>for data in dataloader:</code> with</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "blabla\venv\Lib\site-packages\torch\utils\data\dataloader.py", line 633, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "blabla\venv\Lib\site-packages\torch\utils\data\dataloader.py", line 676, in _next_data
    index = self._next_index()  # may raise StopIteration
            ^^^^^^^^^^^^^^^^^^
  File "blabla\venv\Lib\site-packages\torch\utils\data\dataloader.py", line 623, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "blabla\venv\Lib\site-packages\torch\utils\data\sampler.py", line 289, in __iter__
    for idx in self.sampler:
  File "blabla\venv\Lib\site-packages\torch\utils\data\sampler.py", line 167, in __iter__
    yield from map(int, torch.randperm(n, generator=generator).numpy())
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "blabla\venv\Lib\site-packages\torch\utils\_device.py", line 76, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>I deliberately omitted the definition of <code>__getitem__</code> for <code>NullDataSet</code> to show that it isn’t the cause of the error. How should I fix this? This error seems to be introduced by commit <code>Do not materialize entire randperm in RandomSampler (#103339)</code>, calling <code>.numpy()</code> on a cuda tensor. Previously there was <code>torch.randperm(n, generator=generator).tolist()</code> instead of <code>map(int, torch.randperm(n, generator=generator).numpy())</code></p>

username: Krayaty
cooked: <p>Hi Joshua,</p>
<p>I got the same Error with my setup last week. I’m using conda and I had the latest CUDA driver installed for my RTX 3090. I had Pytorch in some Version of 2.1 installed. I can’t recall the exact Versions for CUDA driver and Pytorch. But I’m absolutely sure that I didn’t have the latest Pytorch Version.<br>
I tracked the problem down to the same thing you suggest. At first I didn’t know how to fix it with coding so I just updated Pytorch to the latest stable version  2.1.2 and voila it was running again! I don’t know why and I can not reproduce it.<br>
Later on I got the problem again when running my project on a slurm cluster with the latest stable Image from <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch" class="inline-onebox" rel="noopener nofollow ugc">PyTorch | NVIDIA NGC</a>. So I went on searching for a proper solution. In the end I just used a custom sampler for my dataloader. This fixed the problem for me.</p>