Num_worker problem

Sky-iii: <p>When I use num_workers &gt;0, my threads get extremely slow randomly at the end of each iteration.</p>
<p>Here’s part of my code:</p>
<pre><code class="lang-auto">
    for epoch in range(1, num_epochs): 
            for i, (x_train, y_train) in enumerate(train_batch):  # y_train stands for label

                trainPhase_start_time = time.time()
                print('trainPhaseEnd-NextTrainPhaseStart:', i, '           ', trainPhase_start_time -trainPhase_end_time)
                
                x_train = x_train.type(torch.FloatTensor)
                y_train = y_train.type(torch.FloatTensor)
                x_train = x_train.to(device)  
                y_train = y_train.to(device)

                predict_train = net(x_train)  
                loss = criterion(predict_train, y_train)

                optimizer.zero_grad()  
                loss.backward()  # backward
                optimizer.step()  

                trainPhase_end_time = time.time()
                print('trainPhaseStart-trainPhaseEnd:(single batch)', trainPhase_end_time - trainPhase_start_time)

</code></pre>
<p>Every 16(num_worker) iterations, data loading may get stuck:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cb065496a3636ccd2f745b160d95ca1a889dfd83.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/cb065496a3636ccd2f745b160d95ca1a889dfd83" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb065496a3636ccd2f745b160d95ca1a889dfd83_2_690x230.jpeg" alt="image" data-base62-sha1="sY2ul8Pt5FklU5L3IT9fbAdDPCH" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb065496a3636ccd2f745b160d95ca1a889dfd83_2_690x230.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb065496a3636ccd2f745b160d95ca1a889dfd83_2_1035x345.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cb065496a3636ccd2f745b160d95ca1a889dfd83_2_1380x460.jpeg 2x" data-dominant-color="413F3F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1772×592 273 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And after like 100 iterations, it may happen irregularity.</p>

Sky-iii: <p>Normally, training and data loading take very short time:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577_2_690x373.jpeg" alt="image" data-base62-sha1="6tn0CdjpGCIjSAheavGVn7cwDfp" width="690" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577_2_690x373.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577_2_1035x559.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d5efe78c45f18ed4e0a21c5e312e7e3ab355577_2_1380x746.jpeg 2x" data-dominant-color="424242"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1843×998 450 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sky-iii: <p>And this situation doesn’t happen every 16 iterations:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fba170efbdb6c50754740a86d3d676b04ad8500.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5fba170efbdb6c50754740a86d3d676b04ad8500" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fba170efbdb6c50754740a86d3d676b04ad8500_2_690x202.jpeg" alt="image" data-base62-sha1="dEQ3sCNCcJJSer2agOXdEpLwWAg" width="690" height="202" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fba170efbdb6c50754740a86d3d676b04ad8500_2_690x202.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fba170efbdb6c50754740a86d3d676b04ad8500_2_1035x303.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fba170efbdb6c50754740a86d3d676b04ad8500_2_1380x404.jpeg 2x" data-dominant-color="403E3E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1791×526 240 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>