Why the training slow down with time if training continuously? And Gpu utilization begins to jitter dramatically?

username: Musoy_King
cooked: <p>The model is just a resnet50, finetuned on a 1.6m images dataset.<br>
The first 10k steps just took about 106m, which the 10k steps from 30k to 40k took about 171m.<br>
Before around 15k steps, the gpu util is quite stable and nearly reach 100%. Afterwards, the gpu util begins to jitter rapidly and seems periodically from 0~100%, but pcie bandwidth util is quite low(occasionally reach 60%) and the loading subprocesses seem to work fine: the cpu utilization is low and the disk reading speed is low (checked with iotop) which I guess is a sign that every loading subprocess has already loaded sufficient data. The gpu temperature is not high either(69C top) and gpu on the highest performace level.<br>
Why does this happen? Is there any solutions?</p>
<p>ps. the image files reside in a ssd. the batch size is 64. The nivida-smi daemon mode is on. Once I stopped after the first epoch, and then loaded the trained model and resumed training, the speeds of the two epoch were quite same, as opposed to training continuously.</p>

username: Musoy_King
cooked: <p>Using <code>torch.cuda.empty_cache()</code> once in a while solved this problem.</p>

username: SimonW
cooked: <p>That’s quite interesting. You shouldn’t need to do that to solve the issue. Are you using 0.3 or some commit after 0.3?</p>

username: Musoy_King
cooked: <p><a class="mention" href="/u/simonw">@SimonW</a><br>
0.3.0.post4<br>
And here’s a duration plot every 100 batches.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/72f5cd13206be3d74b142765fb7b30da108771ba.png" data-download-href="https://discuss.pytorch.org/uploads/default/72f5cd13206be3d74b142765fb7b30da108771ba" title="263Z"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/72f5cd13206be3d74b142765fb7b30da108771ba_2_690x308.png" alt="263Z" data-base62-sha1="goZ3AXo8z1MmzxLiDH5jVs6R0w2" width="690" height="308" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/72f5cd13206be3d74b142765fb7b30da108771ba_2_690x308.png, https://discuss.pytorch.org/uploads/default/original/2X/7/72f5cd13206be3d74b142765fb7b30da108771ba.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/72f5cd13206be3d74b142765fb7b30da108771ba.png 2x" data-dominant-color="F8FBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">263Z</span><span class="informations">824×368 29.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Sorry about the step overlap. The linearly growing part corresponds to code without <code>torch.cuda.empty_cache()</code>, and the lower stable part corresponds to code with this magic command.</p>
<p>And as I stated before, even without this command, the gpu memory utilization is quite stable during the whole training process. So I’m quite confused too.</p>

username: SimonW
cooked: <p>This might suggest something weird in our code. Do you have a script that I can try to reproduce the issue? Thanks!</p>

username: samra-irshad
cooked: <p>I am using pytorch version 1.1.0 and facing the same issue where the training time per step increases within the epoch. I tried removing all the lists/other data objects that could result in storing variables from previous iterations but that did not help. Any suggestions? Also, I am using multiple GPUs to train my model and using <code>nn.DataParallel</code> package</p>

username: ptrblck
cooked: <p>Is this issue solved <a href="https://discuss.pytorch.org/t/slow-training-with-nn-dataparallel/91377/3">here</a> or is this unrelated?</p>

username: samra-irshad
cooked: <p>This is resolved in the reference you provided, thanks.</p>