Sending dataset to cuda breaks the dataloader iterator - TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu()

fredbutbetter: <p>I am trying to speed up my pytorch training by following the advice from here:</p>
<aside class="quote quote-modified" data-post="12" data-topic="25343">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/samarth-robo/48/3487_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/cpu-faster-than-gpu/25343/12">Cpu faster than gpu?</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/mattinjersey">@mattinjersey</a>, it seems to me that the difference between your code and <a class="mention" href="/u/ptrblck">@ptrblck</a>’s code is that the latter only measures the time for computation on the GPU, it does not account for the data transfer time (data transfer happens only once in <a class="mention" href="/u/ptrblck">@ptrblck</a>’s code). 
On the other hand, you transfer data to the GPU at every iteration, and hence you are observing the additional time required for that. 
Is it possible, as <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested, to write your ComputeResults() function as a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="noopener nofollow ugc">dataset</a>, so that a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="noopener nofollow ugc">d…</a>
  </blockquote>
</aside>

<p>So now, I am sending my trainingdata.data and .targets to cuda before starting training. What I am confused about it how to then use the Dataloader made off of the trainingdata in my train function, as when I try the way I had before (when I was sending individual batches to cuda), I get this error on the iterator of the dataloader:</p>
<pre><code class="lang-auto">TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>It is erroring on the following line (loader is the dataloader here):</p>
<pre><code class="lang-auto">for data, target in loader:
</code></pre>
<p>What am I doing wrong? How can I still use the dataloader while also sending the full dataset over to the gpu?</p>
<p>Python version is 3.12, pytorch is 2.3.0+cu121</p>
<p>Also, the error is the same if I dont include the pin_memory var to the dataloader</p>
<p>Full error trace:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2024.1.2\plugins\python-ce\helpers\pydev\pydevd.py", line 1537, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2024.1.2\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\me\PycharmProjects\NueralNetTests\TorchTests.py", line 181, in &lt;module&gt;
    main()
  File "C:\Users\me\PycharmProjects\NueralNetTests\TorchTests.py", line 169, in main
    train(epoch, model, loaders, device, optimizer, lossFN)
  File "C:\Users\me\PycharmProjects\NueralNetTests\TorchTests.py", line 41, in train
    for data, target in loaders['train']:
  File "C:\Users\me\PycharmProjects\NueralNetTests\venv\Lib\site-packages\torch\utils\data\dataloader.py", line 631, in __next__
    data = self._next_data()
           ^^^^^^^^^^^^^^^^^
  File "C:\Users\me\PycharmProjects\NueralNetTests\venv\Lib\site-packages\torch\utils\data\dataloader.py", line 675, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\PycharmProjects\NueralNetTests\venv\Lib\site-packages\torch\utils\data\_utils\fetch.py", line 51, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
            ~~~~~~~~~~~~^^^^^
  File "C:\Users\me\PycharmProjects\NueralNetTests\venv\Lib\site-packages\torchvision\datasets\mnist.py", line 143, in __getitem__
    img = Image.fromarray(img.numpy(), mode="L")
                          ^^^^^^^^^^^
TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
</code></pre>
<p>and The code:</p>
<pre><code class="lang-auto">    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')


    _transform = Compose([
        lambda img: rotate(img, -90),
        lambda img: hflip(img),
        ToTensor()
        ])

    trainData = datasets.EMNIST(
        root='data',
        train=True,
        transform=_transform,
        download=True,
        split='letters'
    )
    testData = datasets.EMNIST(
        root='data',
        train=False,
        transform=_transform,
        download=True,
        split='letters'
    )

    trainLoader = DataLoader(trainData,
                             batch_size=100,
                             shuffle=True,
                             pin_memory=True
                             )

    testLoader = DataLoader(testData,
                            batch_size=100,
                            shuffle=True,
                            pin_memory=True
                            )

    trainData.data = trainData.data.to(device)
    trainData.targets = trainData.targets.to(device)
    testData.data = testData.data.to(device)
    testData.targets = testData.targets.to(device)

    model = CNN().to(device)
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    lossFN = nn.CrossEntropyLoss()

    for epoch in range(1, 2):
        train(model, trainLoader, device, optimizer, lossFN)
        test(model, testLoader, device, lossFN)

def train(model, loader, device, optimizer, lossFN):
    model.train()
    for data, target in loader:
        optimizer.zero_grad()
        output = model(data)
        loss = lossFN(output, target)
        loss.backward()
        optimizer.step()

class CNN(nn.Module):

    def __init__(self):
        super(CNN, self).__init__()

        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2Drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 27)

    def forward(self, x):
        x = F.leaky_relu(F.max_pool2d(self.conv1(x), 2))
        x = F.leaky_relu(F.max_pool2d(self.conv2Drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.leaky_relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)

        return F.softmax(x)


</code></pre>

fredbutbetter: <p>I forgot to add - I tried adding ‘generator=torch.Generator(device=‘cuda’)’ to the dataloader parameters, but the error stays the same</p>

ptrblck: <p>The <code>__getitem__</code> method expects to use <code>CPUTensor</code>s and fails in the <code>img.numpy()</code> call <a href="https://github.com/pytorch/vision/blob/ab0b9a436bd64c4d0309f1b700868c2fe73c0f3e/torchvision/datasets/mnist.py#L143">here</a>. You could override the <code>__getitem__</code> with your custom logic and transform the <code>CUDATensor</code>s directly (i.e. remove the <code>Image.fromarray(img.numpy())</code> call.</p>