ConcatDataset does not work with num workers>0

Max_Torch: <p>Hey,</p>
<p>I am trying to do a activity recognition. For that i have a lot of small videos to train my network. Each video has a different size in length. So i create for every video a new dataset and put them together with the ConcatDataset function, to avoid “jumps” when i would just load it as one big video or dataset. Now when i run my network with num_workers = 0 it works perfectly fine, but when i put num_workers to 1 or higher i get the following error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/1/c1173c7d2ce97a7db9df76b72be726ac6787693e.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1173c7d2ce97a7db9df76b72be726ac6787693e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1173c7d2ce97a7db9df76b72be726ac6787693e_2_690x361.png" alt="image" data-base62-sha1="ry9VZk1mtuuubQZSkVCBcPPvlPU" width="690" height="361" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1173c7d2ce97a7db9df76b72be726ac6787693e_2_690x361.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1173c7d2ce97a7db9df76b72be726ac6787693e_2_1035x541.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/1/c1173c7d2ce97a7db9df76b72be726ac6787693e_2_1380x722.png 2x" data-dominant-color="303034"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2646×1386 364 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The problem occurs in the dataloader while iterating. When i change the batch_size to a higher or lower value, the code will run for 1 or 2 epochs until the same error occurs.<br>
I checked all my batches for the input data type and they have exactly the same type. For input a float and output a int32. I am running the code on windows and on a cuda device.</p>
<p>My custom dataset looks like this:</p>
<p>class HandPoseDatasetNumpy(Dataset):<br>
def <strong>init</strong>(self, data, distances=Config.only_dist):<br>
self.data = data<br>
self.distances = distances</p>
<pre><code>def __len__(self):
    return len(self.data[0])

def __getitem__(self, idx):
    x = self.data[0]
    y = self.data[1]
    seq_x = x[idx:idx + Config.sequence_length:Config.sequence_steps]
    seq_y = y[idx:idx + Config.sequence_length:Config.sequence_steps]

    # ergänzt weitere zahlen für die sequenz damit diese nicht in das neue video springt
    to_pad = math.ceil(Config.sequence_length/Config.sequence_steps) - seq_x.shape[0]
    x_pad = np.pad(seq_x, ((0, to_pad), (0, 0)), mode='mean')
    y_pad = np.pad(seq_y, ((0, to_pad), (0, 0)), mode='mean')

    x = x_pad.reshape(math.ceil(Config.sequence_length/Config.sequence_steps), 21, 3)

    return x, y_pad
</code></pre>
<p>If you have any ideas or need more information, just tell me`</p>

ptrblck: <p>The error message and description sounds a bit weird so I’m unsure if the <code>DataLoader</code> is really failing in the mentioned line of code or just re-raises a previous error.<br>
Could you add some debug <code>print</code> statements into the <code>collate.py</code> file right before the <code>torch.stack</code> operation is called and check what <code>batch</code> contains?</p>

J_Johnson: <p>Are you using Linux or Windows? I recall having issues in Windows with num_workers&gt;0 with the vanilla dataloader.</p>

Max_Torch: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> I am using windows, do you remember how you solved it?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> batches look fine to me, containing my desired batches</p>
<p>the error message gets created in the worker.py when trying fetcher.fetch(index)</p>
<pre><code>            try:
                data = fetcher.fetch(index)
            except Exception as e:
                if isinstance(e, StopIteration) and dataset_kind == _DatasetKind.Iterable:
                    data = _IterableDatasetStopIteration(worker_id)
                    # Set `iteration_end`
                    #   (1) to save future `next(...)` calls, and
                    #   (2) to avoid sending multiple `_IterableDatasetStopIteration`s.
                    iteration_end = True
                else:
                    # It is important that we don't store exc_info in a variable.
                    # `ExceptionWrapper` does the correct thing.
                    # See NOTE [ Python Traceback Reference Cycle Problem ]
                    data = ExceptionWrapper(
                        where="in DataLoader worker process {}".format(worker_id))
</code></pre>
<p>The problem is, when I am in debug mode at the iteration where the error occurs and create a test_data via</p>
<pre><code>            data = [fetcher.dataset[idx] for idx in index] 
</code></pre>
<p>it works. But however if I run the</p>
<pre><code>            data = fetcher.fetch(index)
</code></pre>
<p>I get the same error as bevore.</p>

Max_Torch: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="173721">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you add some debug <code>print</code> statements into the <code>collate.py</code> file right before the <code>torch.stack</code> operation is called and check what <code>batch</code> contains?</p>
</blockquote>
</aside>
<p>I have now written my own collate_fn like:</p>
<pre><code>        def my_own_collate_fn(batch):
            data = np.array([item[0] for item in batch])
            target = np.array([item[1] for item in batch])
            target = torch.from_numpy(target).long()
            data = torch.from_numpy(data).float()
            return [data, target]
</code></pre>
<p>Now it works without an error, but I am not to sure if the new function uses all num_workers</p>

J_Johnson: <ol>
<li>You can use a Linux virtual machine.</li>
<li>You can upload the entire dataset to RAM or a spare GPU as a torch tensor object, then create a custom dataloader.</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="Max_Torch" data-post="5" data-topic="173721">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/max_torch/48/57703_2.png" class="avatar"> Max_Torch:</div>
<blockquote>
<p>Now it works without an error, but I am not to sure if the new function uses all num_workers</p>
</blockquote>
</aside>
<p>The <code>collate_fn</code> is applied after the workers have loaded all samples of the batch, so it should not decrease or block any workers.<br>
It’s still strange to see that your custom <code>collate_fn</code> fixed it. Were you able to inspect the <code>batch</code> created the failure?</p>