Iterations per second deacreasing over time

username: axandrei
cooked: <p>Hello, I am running a UNET in order to train a melspectrogram denoiser I am using 4 2080 Tis to train it. everything works fine for the first ~800 iterations of the first epoch (out of 1200) having approx 7-8 iterations per second. But after that 800iterations something happens (and I can’t tell what) but somehow I only get to 1-2 iterations per second. I am using BATCH_SIZE = 32 pin_memory=True DataLoader num_workers = 4 non_blocking = True</p>
<p>this is what I see when I do some profiling I can observe that it takes some time for the CPU to processes the next batches??? But I can’t understand why at first everything works fine having a high number of iterations per second</p>
<p>There is a gap of 1.5 seconds between batch 755 and batch 756</p>
<p>This behavior resets after each epoch<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c317ab6e50af481de69a3dd4e2e51df0cdb0d128.png" data-download-href="https://discuss.pytorch.org/uploads/default/c317ab6e50af481de69a3dd4e2e51df0cdb0d128" title="ss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c317ab6e50af481de69a3dd4e2e51df0cdb0d128_2_690x52.png" alt="ss" data-base62-sha1="rPROLd278vCx0qsZCKu5RV4RHhC" width="690" height="52" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c317ab6e50af481de69a3dd4e2e51df0cdb0d128_2_690x52.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c317ab6e50af481de69a3dd4e2e51df0cdb0d128_2_1035x78.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/3/c317ab6e50af481de69a3dd4e2e51df0cdb0d128_2_1380x104.png 2x" data-dominant-color="E8E7E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ss</span><span class="informations">1612×122 10.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/axandrei">@axandrei</a><br>
There is a nice older post that has some good tips</p>
<aside class="quote" data-post="24" data-topic="4460">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/training-gets-slow-down-by-each-batch-slowly/4460/24">Training gets slow down by each batch slowly</a> <a class="badge-category__wrapper " href="/c/reinforcement-learning/6"><span data-category-id="6" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A section to discuss RL implementations, research, problems"><span class="badge-category__name">reinforcement-learning</span></span></a>
  </div>
  <blockquote>
    Hi, 
Given what you describe, I guess the issue is with the way to initialize your Parameters when there is no checkpoint to load.
  </blockquote>
</aside>