Why batch loading time varies for different batches (Image Loading)?

username: ammary-mo
cooked: <p>I have the same problem reported in the post <a href="https://discuss.pytorch.org/t/different-batches-take-different-times-to-load/19881" class="inline-onebox">Different batches take different times to load</a>. I am using 3 workers Dataloading workers for loading images from a local folder. And as you can see below, it seems to me that one worker is constantly having a slower time than the other two.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e94f001e66ac0b9360b4bad9fc58791e3d187c48.png" alt="image" data-base62-sha1="xhWvw0LIEohUd00ulHLoAi1SlAs" width="561" height="463"></p>
<p>The accepted answer suggests using SSD, which is true in my case. Also suggests increasing the number of workers. When I increased the num_workers=8, I observed the same pattern, where every 8 batches, one of them is constantly taking more time to load.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/4/c4ab6395de9dc17e7cf4732a1b7938a7c16a1ba8.png" alt="image" data-base62-sha1="s3OMzCcReVY6PoPjo6Tfi9QIWNy" width="565" height="419"></p>
<p>Below is my code for loading. Is there anything missing or optimizations that I need to make to fix this?</p>
<pre><code class="lang-auto">class MyDataset(data.Dataset):

    def __init__(self, datasets, transform=None):
        self.datasets = datasets
        self.transform = transform

    def __len__(self):
        return len(self.datasets)

    def __getitem__(self, index):
        image = Image.open(os.path.join(self.datasets[index][0]))
        if self.transform:
            image = self.transform(image)
        return image, torch.tensor(self.datasets[index][1], dtype=torch.long)

    # create Dataset object
    training_dataset = MyDataset(training_set, transformer)

    training_dataloader = torch.utils.data.DataLoader(
        training_dataset,
        batch_size=batch_size,
        num_workers=8,
        shuffle=True,
        pin_memory=True
    )
</code></pre>

username: ptrblck
cooked: <p>Your dataloading might be too slow and a potential explanation of the peak in the loading time is given <a href="https://discuss.pytorch.org/t/strange-behavior-in-data-loader-with-workers/83769/2">here</a>.</p>

username: ammary-mo
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>But are you saying that my data loading is slow compared with training loop time, or you mean that I have an issue with MyDataset class?</p>
<p>What I understand from the discussion, that in my case, there is no way to avoid this other than increasing the number of workers to the limit that makes their loading time hidden compared with the training loop time. Please correct my understanding.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ammary-mo" data-post="3" data-topic="117107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ammary-mo/48/36146_2.png" class="avatar"> ammary-mo:</div>
<blockquote>
<p>But are you saying that my data loading is slow compared with training loop time</p>
</blockquote>
</aside>
<p>Yes, it seems that the data loading time is large compared to the actual model training time, which would explain the peaks in the data loading time.</p>
<aside class="quote no-group" data-username="ammary-mo" data-post="3" data-topic="117107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ammary-mo/48/36146_2.png" class="avatar"> ammary-mo:</div>
<blockquote>
<p>or you mean that I have an issue with MyDataset class?</p>
</blockquote>
</aside>
<p>There might be a potential improvement, if replacing:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor(self.datasets[index][1], dtype=torch.long)
</code></pre>
<p>which will trigger a copy with:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.from_numpy(self.datasets[index][1]astype(np.int64))
</code></pre>
<p>which would reuse the underlying data (assuming <code>self.datasets</code> returns a numpy array).</p>
<aside class="quote no-group" data-username="ammary-mo" data-post="3" data-topic="117107">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ammary-mo/48/36146_2.png" class="avatar"> ammary-mo:</div>
<blockquote>
<p>What I understand from the discussion, that in my case, there is no way to avoid this other than increasing the number of workers to the limit that makes their loading time hidden compared with the training loop time. Please correct my understanding.</p>
</blockquote>
</aside>
<p>Yes, that’s correct and you would have to either speed up the data loading (or increase the workload for the model training).<br>
Note that, the smaller the GPU workload is, the more likely you’ll hit a data loading bottleneck.<br>
In the extreme case that your model immediately trains an iteration (just remove the model training), you would have to make sure that the data loading is fast enough to load and process the next batch while Python “executes the loop” (which would be fast as there is no real workload).</p>

username: ammary-mo
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the detailed feedback.</p>
<p>You are right, increasing the num_workers would enhance the performance. But not if I am using a distributed training, I noticed that in the case of distributed training (DistributedDataParallel) increasing num_workers is hindering performance. I reported a similar problem in this <a href="https://discuss.pytorch.org/t/why-my-distributeddataparallel-is-slower-than-dataparallel-if-my-dataset-is-not-loaded-fully-in-memory/115677/8">post</a>.</p>