Generator does not load on GPU despite explicitly stated

AJ_00: <p>Dear everyone,</p>
<p>I am using PyTorch for training my network, and I am doing it over multiple nodes (distributed).</p>
<p>I am using the Rendezvous backend for the distributed computing, and when I run my code, I am faced with an error that is as it written below:</p>
<pre><code class="lang-auto">RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'
</code></pre>
<p>Although I have created an instance of DataLoader as below:</p>
<pre><code class="lang-auto">    model = DistributedDataParallel(model, device_ids=[local_rank])

    train_sampler = DistributedSampler(dataset)

    train_loader = DataLoader(dataset=dataset, batch_size=batch_size,
                              shuffle=False, num_workers=0, pin_memory=True,
                              sampler=train_sampler, generator=torch.Generator(device='cuda'))
</code></pre>
<p>It still gives me the same error not matter how many times I try to find whether I have problems saving the source files or just a typo.</p>
<p>Please come to my help if anything is in your mind.</p>
<p>Best,<br>
AJ</p>

ptrblck: <p>Is the <code>DataLoader</code> raising the error? If so, could you post the dataset code?</p>

AJ_00: <p>The error points to DataLoader when I start to iterate through its elements.<br>
This is the code for the Dataset.<br>
I am sorry if it is too long or badly written.<br>
Please let me know if you have any feedback.</p>
<pre><code class="lang-auto">import h5py
from torch.utils.data import Dataset
import numpy as np

class NYUDataset(Dataset):
    def __init__(self, files_address, image_transform=None, depth_transform=None, imu_transform=None):
        self.files_address = files_address
        self.image_transform = image_transform
        self.depth_transform = depth_transform
        self.imu_transform = imu_transform

        self.dsv1 = h5py.File(self.files_address[0])
        self.dsv2 = h5py.File(self.files_address[1])

        self.no_dsv1 = self.dsv1['depths'].shape[0]
        self.no_dsv2 = self.dsv2['depths'].shape[0]

    def __len__(self):
        return (self.no_dsv1 + self.no_dsv2) - 2

    def __getitem__(self, idx):
        if idx &lt; (self.no_dsv1 - 1):
            image1 = np.transpose(self.dsv1['images'][idx], (2, 1, 0))
            image2 = np.transpose(self.dsv1['images'][idx+1], (2, 1, 0))
            imu_t1 = self.dsv1['accelData'][:, idx]
            imu_t2 = self.dsv1['accelData'][:, idx+1]
            imu = imu_t2 - imu_t1            
            depth = np.transpose(self.dsv1['depths'][idx+1], (1, 0))

            if self.image_transform:
                image1 = self.image_transform(image1)
                image2 = self.image_transform(image2)
            if self.imu_transform:
                imu = self.imu_transform(imu)
            if self.depth_transform:
                depth = self.depth_transform(depth)
            return (imu, image1, image2, depth, 1)
        elif idx &lt; (self.no_dsv2 + self.no_dsv1):
            correct_idx = (idx - self.no_dsv1)
            image1 = np.transpose(self.dsv2['images'][correct_idx], (2, 1, 0))
            image2 = np.transpose(self.dsv2['images'][correct_idx+1], (2, 1, 0))
            imu_t1 = self.dsv2['accelData'][:, correct_idx]
            imu_t2 = self.dsv2['accelData'][:, correct_idx+1]
            imu = imu_t2 - imu_t1
            depth = np.transpose(self.dsv2['depths'][correct_idx+1], (1, 0))

            if self.image_transform:
                image1 = self.image_transform(image1)
                image2 = self.image_transform(image2)
            if self.imu_transform:
                imu = self.imu_transform(imu)
            if self.depth_transform:
                depth = self.depth_transform(depth)
            return (imu, image1, image2, depth, 2)

if __name__ == '__main__':
    test = NYUDataset(['data/nyu_depth_v1_labeled.mat', 'data/nyu_depth_v2_labeled.mat'])
    import matplotlib.pyplot as plt
    import time
    for imu, image1, image2, depth, ds in test:
        plt.subplot(1, 3, 1)
        plt.imshow(np.transpose(image1, (2, 1, 0)))
        plt.subplot(1, 3, 2)
        plt.imshow(np.transpose(image2, (2, 1, 0)))
        plt.subplot(1, 3, 3)
        plt.imshow(np.transpose(depth, (1, 0)))

        plt.savefig(f'figures/{time.time()}.png')
        print(imu)
</code></pre>
<p>Best,<br>
AJ</p>

AJ_00: <p>I’m sorry to bother, but could you please help me with this issue?<br>
Do you need more information that I can provide?</p>

ptrblck: <p>I don’t have access to the dataset so wasn’t able to reproduce and debug the issue, unfortunately. I’m also not seeing anything obviously wrong in your code.</p>
<p>Could you check if your code used e.g. <code>torch.set_default_device(torch.device('cuda'))</code> and if so could you remove it?</p>