Drastically slow speed at first epoch

hyojin: <p>Hi,</p>
<p>I have been experiencing this problem for a while, and it only happens with somewhat large datasets &gt; 1M.<br>
The training speed at the first epoch is about 5~6 times slower than the next epochs, and it is not just the first few iterations (cold start). The speed gradually gets faster, but on average, it is about 5~6 times slower.</p>
<p>Do any of you have an idea of what could be happening?<br>
I am using a data loader based on image list files:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/vision/issues/81" target="_blank" rel="nofollow noopener">github.com/pytorch/vision</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/xiahouzuoxin" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/4554770?v=2&amp;s=96" class="thumbnail onebox-avatar" width="96" height="96">
</a>

<h4><a href="https://github.com/pytorch/vision/issues/81" target="_blank" rel="nofollow noopener">Issue: load image dataset from list files</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/xiahouzuoxin" target="_blank" rel="nofollow noopener">xiahouzuoxin</a>
	on <a href="https://github.com/pytorch/vision/issues/81" target="_blank" rel="nofollow noopener">2017-03-01</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/fmassa" target="_blank" rel="nofollow noopener">fmassa</a>
	on <a href="https://github.com/pytorch/vision/issues/81" target="_blank" rel="nofollow noopener">2017-11-12</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">I suggest to add a IO for read images from a list like this to support custom image data input,
img1 label1
img2...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">awaiting response</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Thanks a lot in advance.</p>

SimonW: <p>It could be disk IO problem. Could you monitor the time to load each data?</p>

hyojin: <p>I think so, too. I just don’t know how to fix this…</p>
<p>Another evidence is that when I start running another instance of training (that uses the same dataset) on the other GPU, the one that was previously running gets affected and slows down.</p>
<p>It alternates between 10-20 seconds and 1-4 seconds per loading each batch during the first epoch. In the next epoch, it maintains 0.3 seconds per batch, which seems normal.</p>

SimonW: <p>a couple of ideas:</p>
<ol>
<li>if data is on hdd, maybe defragment?</li>
<li>move the dataset to a drive that no other process uses</li>
<li>increase num_workers</li>
</ol>

hyojin: <p>Thanks for the help! I will try those.</p>

hyojin: <p>Increasing number of workers did not help. I will stick with your second solution. By the way, could you give me an insight on why this would not occur in the next epochs, but just during the first epoch?</p>

SimonW: <p>Sorry, I’m not certain what exactly happens either.</p>

jdhao: <p>I am experiencing the same issue. When I switch to validation and switch back, the beginning few hundred iterations is always very slow, about 4~5 times slower than normal. Some example statistics from the training process:</p>
<pre><code class="lang-auto">2018-10-17 20:12:12 Validation: max_iter: 1725, loss:1.191111, Validation cer: 0.065339, accuracyCTC: 0.625263
2018-10-17 20:16:01 Train: [26/150][430500/2598450] Loss: 0.408760 Loss_CTC: 0.408760
time elapsed 1747
2018-10-17 20:19:37 Train: [26/150][431000/2598450] Loss: 0.353606 Loss_CTC: 0.353606
time elapsed 216
2018-10-17 20:23:46 Train: [26/150][431500/2598450] Loss: 0.396339 Loss_CTC: 0.396339
time elapsed 249
2018-10-17 20:30:23 Train: [26/150][432000/2598450] Loss: 0.379618 Loss_CTC: 0.379618
time elapsed 396
2018-10-17 20:35:08 Train: [27/150][432500/2598450] Loss: 0.395716 Loss_CTC: 0.395716
time elapsed 285
2018-10-17 20:40:20 Train: [27/150][433000/2598450] Loss: 0.430345 Loss_CTC: 0.430345
time elapsed 311
2018-10-17 20:45:49 Train: [27/150][433500/2598450] Loss: 0.451907 Loss_CTC: 0.451907
time elapsed 328
2018-10-17 20:49:50 Train: [27/150][434000/2598450] Loss: 0.393155 Loss_CTC: 0.393155
time elapsed 240
2018-10-17 20:54:10 Train: [27/150][434500/2598450] Loss: 0.375856 Loss_CTC: 0.375856
time elapsed 260
2018-10-17 20:59:41 Train: [27/150][435000/2598450] Loss: 0.374424 Loss_CTC: 0.374424
time elapsed 330
2018-10-17 21:24:42 Validation: max_iter: 1725, loss:1.233052, Validation cer: 0.065245, accuracyCTC: 0.623922
2018-10-17 21:29:18 Train: [27/150][435500/2598450] Loss: 0.400490 Loss_CTC: 0.400490
time elapsed 1777
2018-10-17 21:34:05 Train: [27/150][436000/2598450] Loss: 0.559940 Loss_CTC: 0.559940
time elapsed 286
2018-10-17 21:38:16 Train: [27/150][436500/2598450] Loss: 0.469486 Loss_CTC: 0.469486
time elapsed 250
2018-10-17 21:42:23 Train: [27/150][437000/2598450] Loss: 0.460522 Loss_CTC: 0.460522
time elapsed 246
2018-10-17 21:46:36 Train: [27/150][437500/2598450] Loss: 0.417255 Loss_CTC: 0.417255
time elapsed 252
2018-10-17 21:50:29 Train: [27/150][438000/2598450] Loss: 0.446802 Loss_CTC: 0.446802
time elapsed 233
2018-10-17 21:54:54 Train: [27/150][438500/2598450] Loss: 0.405387 Loss_CTC: 0.405387
time elapsed 264
2018-10-17 22:00:10 Train: [27/150][439000/2598450] Loss: 0.388070 Loss_CTC: 0.388070
time elapsed 316
2018-10-17 22:04:14 Train: [27/150][439500/2598450] Loss: 0.399813 Loss_CTC: 0.399813
time elapsed 243
2018-10-17 22:08:08 Train: [27/150][440000/2598450] Loss: 0.386462 Loss_CTC: 0.386462
time elapsed 233
</code></pre>

Yuhang: <p>Hi, I solved this problem through upload the data to the temporary folder in colab instead read the data from the google drive which is extremely slow in the first epoch. Below is the difference between read from temporary folder(/content/)and google drive.<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed9a0c69897664d80f7600063be8ed9c6709d9a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/ed9a0c69897664d80f7600063be8ed9c6709d9a0" title="WeChat2832ca0ee811b9bffc14a3eca28c343e"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed9a0c69897664d80f7600063be8ed9c6709d9a0_2_509x500.png" alt="WeChat2832ca0ee811b9bffc14a3eca28c343e" data-base62-sha1="xTVcSZsUPUCKkuuH2adO8yGpPC8" width="509" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/d/ed9a0c69897664d80f7600063be8ed9c6709d9a0_2_509x500.png, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed9a0c69897664d80f7600063be8ed9c6709d9a0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/d/ed9a0c69897664d80f7600063be8ed9c6709d9a0.png 2x" data-dominant-color="E3DEDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WeChat2832ca0ee811b9bffc14a3eca28c343e</span><span class="informations">618×606 236 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>