utils.data.dataLoader num_workers for custom dataLoader class

Ajinkya_Ambatwar: <p>Hello, I have a custom dataLoader class that I created. It has some extensive functionality for which I have to use python multiprocessing pool. Now I wonder, if I will able to apply <code>num_workers</code> &gt; 0 for such a dataLoader.<br>
Eg.</p>
<pre><code class="lang-auto">#Custom dataLoader class 
#which uses multiprocessing pool with 8 threads
trainData = dataLoader.dataset(
        train=True,
        batchSize=32
    )
# The dataLoader class itself return a 32 batchsize object
#Now creating torch dataLoader wrapper on in
dl = DataLoader(trainData, batch_size=1, num_workers={some_num&gt;0})
</code></pre>
<p>Now my question is, if I put <code>num_workers</code> to some non zero value, it doesn’t return an error. But when I try to get a sample out of that</p>
<pre><code class="lang-auto">sample = next(iter(dl))
</code></pre>
<p>it takes forever to return anything, probably it didn’t work at all.<br>
Is this expected?</p>
<p>My main requirement is, I need to use <code>prefetch_factor</code> in torch DataLoader, which works only when <code>num_workers</code> is not 0</p>
<p>The reason behind putting <code>batchSize=32</code> for custom class is that the sample generated from the dataLoader is a list of arrays. The individual arrays are not bound to have exact same dimension across the dataSet. Hence I have to vertically stack the arrays from 32 samples and return as one huge array.</p>
<p>The torch DataLoader wrapper does not support variable length data per sample if we use <code>num_workers</code></p>
<p>Is it possible to put <code>num_workers</code> to a such a custom dataLoader class?</p>

ptrblck: <aside class="quote no-group" data-username="Ajinkya_Ambatwar" data-post="1" data-topic="121250">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajinkya_ambatwar/48/11828_2.png" class="avatar"> Ajinkya_Ambatwar:</div>
<blockquote>
<p>The torch DataLoader wrapper does not support variable length data per sample if we use <code>num_workers</code></p>
</blockquote>
</aside>
<p>I don’t think this behavior depends on the number of workers, but the default <code>collate_fn</code> would fail to create a single batch using inputs with different shapes.<br>
If you want to apply a custom creation of the batch you could write a custom <code>collate_fn</code> and pass it to the <code>DataLoader</code>.</p>
<aside class="quote no-group" data-username="Ajinkya_Ambatwar" data-post="1" data-topic="121250">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ajinkya_ambatwar/48/11828_2.png" class="avatar"> Ajinkya_Ambatwar:</div>
<blockquote>
<p>it takes forever to return anything, probably it didn’t work at all.<br>
Is this expected?</p>
</blockquote>
</aside>
<p>I don’t think it’s expected to take “forever” so I would guess that the multiprocessing used in the <code>DataLoader</code> and your custom multiprocessing approach might block each other / hang etc.</p>

Ajinkya_Ambatwar: <p>I shall take a look at creating the custom <code>collate_fn</code><br>
Regarding taking forever, I also meant that one process is blocking another.<br>
If I don’t use num workers, my dataLoader with 8 threads take around 1.5 second to load the batch. But when I introduced num_workers=1(or any +ve number), even after 5-10 seconds nothing came up.</p>