Strange behavior in data loader with workers

nirandaperera: <p>Hi,</p>
<p>I am testing Microsoft’s pipedream approach for distributed learning (<a href="https://github.com/msr-fiddle/pipedream" rel="nofollow noopener">https://github.com/msr-fiddle/pipedream</a>)<br>
I am using Alexnet with a subset of ImageNet data and testing the approach using 4 K80 GPUs.</p>
<p>I am observing a strange behavior in the data loader timing in the first GPU in the pipeline. Pipedream code uses a dataloader as follows.</p>
<pre><code class="lang-python">train_dataset = datasets.ImageFolder(
            traindir,
            transforms.Compose([
                transforms.RandomResizedCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))
...
train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
        num_workers=args.workers, pin_memory=True, sampler=train_sampler, drop_last=True)
</code></pre>
<p>Issue is, I am passing <code>args.workers=10</code>, and every multiple of <code>args.workers</code> batch (ex: 10, 20, …) takes significantly higher amount of time in the <code>next(iter(train_loader))</code> call.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>batch</th>
<th>time on next(…)</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>6.153</td>
</tr>
<tr>
<td>1</td>
<td>0.001</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>10</td>
<td>5.408</td>
</tr>
<tr>
<td>11</td>
<td>0.001</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>20</td>
<td>4.916</td>
</tr>
<tr>
<td>21</td>
<td>0.001</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
</tbody>
</table>
</div><p>I have checked for multiple batch sizes (64, …, 256, 512) and by changing workers 4 to 10.</p>
<p>Can someone shed some light on this?<br>
Best</p>

ptrblck: <p>Each worker will load a batch in the background. Once its batch is returned, the worker will start loading the next batch.<br>
Since you are seeing a slowdown after <code>num_workers</code> batches, this points towards a data loading bottleneck (or a tiny model).</p>
<p>I.e. the first iteration (batch 0) will take some time to load the first complete batch. Once it’s loaded, it’ll be returned and after the forward/backward pass is done, the next iteration will start.<br>
The next worker(s) seem to have already loaded the data and will return it immediately (batch 1 to 9).<br>
In batch 10, the first worker is supposed to yield the next batch, but since the previous 9 iterations were really fast, you will see the slowdown again until all samples for the batch are loaded again.</p>

nirandaperera: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>. Thanks for you feedback.<br>
I was expecting the same, and to test that, I ran the sequential model with 10 data loading workers. It uses the same dataloader in the same way. But there, I did not see  a similar behavior.</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>batch</th>
<th>time on next(…)</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>5.76223</td>
</tr>
<tr>
<td>1</td>
<td>0.71920</td>
</tr>
<tr>
<td>2</td>
<td>0.00025</td>
</tr>
<tr>
<td>3</td>
<td>0.00025</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>10</td>
<td>0.00083</td>
</tr>
<tr>
<td>11</td>
<td>0.00077</td>
</tr>
<tr>
<td>12</td>
<td>0.00030</td>
</tr>
<tr>
<td>13</td>
<td>0.00027</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>20</td>
<td>0.00056</td>
</tr>
<tr>
<td>21</td>
<td>0.00029</td>
</tr>
<tr>
<td>22</td>
<td>0.00032</td>
</tr>
<tr>
<td>23</td>
<td>0.00031</td>
</tr>
<tr>
<td>…</td>
<td>…</td>
</tr>
<tr>
<td>30</td>
<td>0.00035</td>
</tr>
<tr>
<td>31</td>
<td>0.00030</td>
</tr>
</tbody>
</table>
</div><p>Coming back to your point, shouldn’t the data loading threads load batches ahead of time? Is this an expected behavior for the multi-process dataloader (slow-down in every <code>num_workers</code> batch)?</p>
<p>PS: Clarification on the usage here. In Pipedream, rank 0 worker would be the only one loading data from the dataset, and theoretically, when it comes to data loading, there’s no difference between rank0 worker of pipedream and sequential execution. Main difference would be pipedream rank0 worker would be doing <code>1/world_size</code> amount work.</p>

ptrblck: <aside class="quote no-group" data-username="nirandaperera" data-post="3" data-topic="83769">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nirandaperera/48/20977_2.png" class="avatar"> nirandaperera:</div>
<blockquote>
<p>shouldn’t the data loading threads load batches ahead of time? Is this an expected behavior for the multi-process dataloader (slow-down in every <code>num_workers</code> batch)?</p>
</blockquote>
</aside>
<p>Each worker is starting to load the next batch once the current one was returned.<br>
However, if the actual workload is small, you would basically just use the data loading loop as:</p>
<pre data-code-wrap="python"><code class="lang-python">for data in loader:
    pass
</code></pre>
<p>This dummy loop would have a startup time, yield the next batches pretty fast and slow down again, as each worker has to load the next batch.</p>
<p>Your current output looks interesting, as I thought you have already used 10 workers (which would explain the slow down after 10 iterations).<br>
What’s the difference between the new output and the previous one?</p>

nirandaperera: <p>Second scenario would load data and run the full model. First scenario would only work on a partition of the model. Hence the calculation time is comparatively short.</p>
<p>I guess that would correspond to your explanation. I will check this again with the timeline. Thanks for the explanation in the meantime.</p>