Runtime slowdown related to number of datasets

talfur: <p>Hi,</p>
<p>I have a training scheme where the training data is spread across many sources.<br>
I’ve defined a dataset per source, ending up with many (30-60) datasets (torch.utils.data.ConcatDataset).</p>
<p>Noticed a slowdown behavior watching GPU load during training. Once in a while the GPU load drops to 0 (not periodic).<br>
Tried to look at transforms time Vs. number of workers, but all seems to be working well (when spread across all workers, get_batch time is smaller then net calculations).</p>
<p>All seems to run smooth (GPU well loaded), until number of datasets goes up above a certain number (e.g 30).<br>
Then I experience the non-periodic slowdown.</p>
<p>Thought about memory … but seems like I have enough free memory during the run.</p>
<p>Was hoping one of you could have an idea where should I look next.</p>
<p>Many thanks,</p>
<p>Tal</p>

ptrblck: <p>Are you lazily loading the data in all “small” datasets or are you preloading the complete data?<br>
In the former case, could you check, if a specific <code>Dataset</code>, which is added to <code>ConcatDataset</code>, creates the slowdown?</p>

talfur: <p>Thank you for the replay.<br>
Data loading:</p>
<ul>
<li>Each dataset has access to preloaded lidar point cloud, saved in frames. during run time - the relevant frame is returned. All data from all dataloader fits in memory (I thought it was a memory loading issue… but no such luck)</li>
</ul>
<p>Dependency on specific dataset:</p>
<ul>
<li>I’ve tried checking for a specific dataset, but found no correlation.<br>
Also tried duplicating a single dataset for N (e.g 30) times, and was able to reproduce the effect.<br>
Now trying to see if larger N for single dataset duplication would lead to longer delay.</li>
</ul>
<p>A bit lost with the next testing direction though <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>Are you sure the complete data fits into memory or could your machine be using the <code>swap</code>?</p>

talfur: <p>Maybe I’m missing something with the way I deduce my memory assumption.<br>
I’ve used free -m, and a typical result looks like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17599c2380bf71a1db5a0a87821a8b9011167dda.png" alt="image" data-base62-sha1="3kyY9npBeWK7rkBiC8fzwFIiMzg" width="690" height="102"></p>
<p>I assumed this means I have pleanty of memory left…<br>
Can you explain (or point me to one) your point with using swap and not memory?</p>

ptrblck: <p>It looks like you have plenty of memory free, but also the swap seems to be used at least a bit.<br>
You could have a look at <a href="https://help.ubuntu.com/community/SwapFaq">this website</a> to change the <code>swappiness</code>.</p>

talfur: <p>Thank you!<br>
I’ll do that and update.</p>

talfur: <p>One more question.</p>
<p>It seems I have been wrong to say there is no pattern to the slowdown.<br>
Giving it another look, the slowdown seems to correlate with the number of workers.<br>
Working with 20 processes (in addition to main process) would give slow down every 20 iterations (not exactly but close to, with some variance).<br>
Moving to 10 processes will cause shorter intervals between slow downs.</p>
<p>Am I missing some other route cause for my slow downs? related to initialization of workers?</p>

ptrblck: <p>This pattern would point towards a data loading bottleneck, i.e. the workers are not fast enough in preparing the next batch relative to the model training.<br>
This could be the case, if your model workload is small or if your data loading is the bottleneck as explained e.g. <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">here</a>.</p>