Dataloader hangs when calling next() using custom data

Lampedoesa: <p>Im trying to use custom dataset with the CocoDetection format, the cocoapi gives a succes on indexing and code passes but hangs when calling next()</p>
<pre><code class="lang-auto">train_dataset = datasets.CocoDetection(args.image_path, args.data_path, transform=coco_transformer())

querry_dataloader = data.DataLoader(train_dataset, sampler=sampler, batch_size=args.batch_size, drop_last=True, num_workers=0)

labeled_data = self.read_data(querry_dataloader)

labeled_imgs, labels = next(labeled_data) #Hangs here
</code></pre>
<p>with read_data() being as follows</p>
<pre><code class="lang-auto">def read_data(self, dataloader, labels=True):
        if labels:
            while True:
                for img, label, _ in dataloader:
                    yield img, label
        else:
            while True:
                for img, _, _ in dataloader:
                    yield img
</code></pre>
<p>The source code I’m trying to implement can be found here <a href="https://github.com/sinhasam/vaal" rel="nofollow noopener">https://github.com/sinhasam/vaal</a><br>
I’m also doing this in the Nvidia NGC pytorch container version 19.10 but the issue persists in the latest version.</p>
<p>Thanks in advance!</p>

ptrblck: <p>Are you able to get a single data sample using the <code>Dataset</code> without a <code>DataLoader</code>?</p>
<pre><code class="lang-python">data, label, _ = train_dataset[0]
</code></pre>

Lampedoesa: <p>Yes, however, I do get an error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "../vaal/main.py", line 143, in &lt;module&gt;
    main(args)
  File "../vaal/main.py", line 73, in main
    ndata, nlabel, _ = train_dataset[0]
ValueError: not enough values to unpack (expected 3, got 2)
</code></pre>
<p>When I use a custom dataset, however, as they do in the source code I get past this (getting the image and the label), but it freezes again at  the same point.</p>
<p>Custom dataset:</p>
<pre><code class="lang-auto">class KFuji(Dataset):
    def __init__(self, image_path, json_path):
        self.kfuji = datasets.CocoDetection(root=image_path, annFile=json_path, transform=coco_transformer())

    def __getitem__(self, index):
        if isinstance(index, numpy.float64):
            index = index.astype(numpy.int64)
        data, target = self.kfuji[index]

        return data, target, index

    def __len__(self):
        return len(self.kfuji)
</code></pre>

ptrblck: <p>Could you try to iterate the dataset once for all samples just to make sure there is not a hidden bug somewhere, which lets the <code>DataLoader</code> hang?</p>

Lampedoesa: <p>It iterates through the data just fine, using:</p>
<pre><code class="lang-auto">for i in range(len(train_dataset)):
            data, label, _ = train_dataset[i]
</code></pre>

Lampedoesa: <p>Turns out my batch size was too large for the initial subset of the data used, which caused the dataloader to hang.</p>

thistlillo: <p>I have the same problem (July 2022). Dataloader with multiple workers hangs after 1st batch. Every one in a while it runs fine. I iterated over the entire dataset without problems.</p>

morenolq: <p>I got the same problem when setting <code>num_workers&gt;0</code>. I also tried putting a timeout and skipping the batch eventually, but after the first time the timeout is reached, all next <code>batch=next(dataloader_iter)</code> instructions fail with the same timeout error.</p>
<p>I surely tried to go through the whole dataset once without problems.</p>