Why the more num_work the time for inference is amazingly longer?

username: jerly-hjzhou
cooked: <p>I think the more num_works, the time cost for inference should be less. But the result from my code is in opposite. I use trained resnet-18 to inference the test set. When the num_work=0, the time cost is around 12s, but when num_work =6, the time is around 25s.<br>
Below is my code.</p>
<pre><code class="lang-auto">import torch
from torchvision import datasets, transforms
import time
import torch.nn.functional as F

def testReability():
    lossSum = 0.0
    lossNumebr = 0
    for i in range(1):
        model = torch.load('resnet_18-cifar_10.pth')
        model.to(device)
        model.eval()
        for data, target in test_loader:
            data = data.to(device)
            target = target.to(device)
            output = model(data)
            lossSum = lossSum + F.cross_entropy(output, target, reduction='sum').item()
        lossNumebr = lossNumebr + len(test_loader) * batchSize
        curLoss = lossSum / lossNumebr
        print("the {} iteration, loss value is {:.8f}".format(i+1, curLoss))

if __name__ == '__main__':
    batchSize=10
    device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
    print("device is ", device)
    data_transforms = transforms.Compose([
            transforms.ToTensor(),
             transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
        ])
    image_datasets = datasets.CIFAR10(root='../data', train = False,
                                              download=True, transform=data_transforms)
    test_loader = torch.utils.data.DataLoader(image_datasets, batch_size=batchSize,
                                              shuffle=True, num_workers=0)
    since = time.time()
    testReability()
    time_elapsed = time.time() - since
    print('Test complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
</code></pre>
<p>There are my results.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd2e178345577b65964ac559130de68f4c2d0204.png" data-download-href="https://discuss.pytorch.org/uploads/default/cd2e178345577b65964ac559130de68f4c2d0204" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd2e178345577b65964ac559130de68f4c2d0204_2_690x484.png" alt="image" data-base62-sha1="th6DempPlodx1uVjQjobuyaXWLO" width="690" height="484" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/d/cd2e178345577b65964ac559130de68f4c2d0204_2_690x484.png, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd2e178345577b65964ac559130de68f4c2d0204.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/d/cd2e178345577b65964ac559130de68f4c2d0204.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">896×629 59.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff_2_690x472.png" alt="image" data-base62-sha1="xJz5MnTeCENFqKqda7xrQD8hrgr" width="690" height="472" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff_2_690x472.png, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec6e57fa42a408cd80f5c70aab83f86e0765b5ff.png 2x" data-dominant-color="F5F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">902×618 59.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Each worker could create an (initial) overhead by copying the dataset to each process and could thus slow down the first iteration. Also, depending on your setup, you would have to find a sweet spot of for the number of workers, as increasing them won’t necessarily always result in a speedup as described e.g. <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">here</a>.</p>
<p>To narrow down the slowdown you could profile your code using e.g. <code>torch.profiler</code>.</p>