Strange very slow training on the server

username: karlTUM
cooked: <p>Hi all,</p>
<p>I come across a strange problem when I train my model on the server. At the first several iterations, the running time is OK. All the CPUs are working for IO and GPU is working. But after several iterations, suddenly the CPUs do not work for IO. I do not know what happened.</p>
<p>Could someone help me? Thank you very much!</p>
<p>During the first several iterations:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/55ea6b7fcf5a74fa371abf67e71b72d2cae81666.png" data-download-href="https://discuss.pytorch.org/uploads/default/55ea6b7fcf5a74fa371abf67e71b72d2cae81666" title="Selection_002"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/55ea6b7fcf5a74fa371abf67e71b72d2cae81666_2_690x51.png" alt="Selection_002" data-base62-sha1="cg2PikfQ9l5otEN6dUcAVbL0BBc" width="690" height="51" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/55ea6b7fcf5a74fa371abf67e71b72d2cae81666_2_690x51.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/55ea6b7fcf5a74fa371abf67e71b72d2cae81666_2_1035x76.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/55ea6b7fcf5a74fa371abf67e71b72d2cae81666.png 2x" data-dominant-color="2E3436"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Selection_002</span><span class="informations">1228×92 8.48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/a/adab8f15c9051aa71bfbb87dc650238053789b6b.png" data-download-href="https://discuss.pytorch.org/uploads/default/adab8f15c9051aa71bfbb87dc650238053789b6b" title="Selection_001"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/a/adab8f15c9051aa71bfbb87dc650238053789b6b_2_690x270.png" alt="Selection_001" data-base62-sha1="oMmagUZUudhgZoZpCTtClzLE7vZ" width="690" height="270" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/a/adab8f15c9051aa71bfbb87dc650238053789b6b_2_690x270.png, https://discuss.pytorch.org/uploads/default/optimized/2X/a/adab8f15c9051aa71bfbb87dc650238053789b6b_2_1035x405.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/a/adab8f15c9051aa71bfbb87dc650238053789b6b.png 2x" data-dominant-color="2F3536"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Selection_001</span><span class="informations">1182×463 59.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Suddenly, CPUs are not working:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5adbde341c534354d3d1e3e97fd615283e814f07.png" data-download-href="https://discuss.pytorch.org/uploads/default/5adbde341c534354d3d1e3e97fd615283e814f07" title="Selection_004"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5adbde341c534354d3d1e3e97fd615283e814f07_2_690x31.png" alt="Selection_004" data-base62-sha1="cXM2jszghEzX1ktkPVOmkG7n0Dt" width="690" height="31" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5adbde341c534354d3d1e3e97fd615283e814f07_2_690x31.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5adbde341c534354d3d1e3e97fd615283e814f07_2_1035x46.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/5/5adbde341c534354d3d1e3e97fd615283e814f07.png 2x" data-dominant-color="2F3536"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Selection_004</span><span class="informations">1247×57 5.18 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/c/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48.png" data-download-href="https://discuss.pytorch.org/uploads/default/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48" title="Selection_003"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48_2_690x256.png" alt="Selection_003" data-base62-sha1="rEWF49XJC7mG4h8sTGNDwbppjp6" width="690" height="256" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48_2_690x256.png, https://discuss.pytorch.org/uploads/default/optimized/2X/c/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48_2_1035x384.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/c/c1db9bf59a2e4c1e5398c2e787d24fad84c45f48.png 2x" data-dominant-color="2E3537"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Selection_003</span><span class="informations">1199×445 53.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And it continues like this. Very slow.</p>

username: albanD
cooked: <p>Hi,</p>
<p>How is the GPU usage?<br>
At the beginning, the dataloader preloads many samples so that might explain the high cpu usage.</p>

username: karlTUM
cooked: <p>Hi The GPU is always zero. I think my dataset is too large and I should put it in a mounted storage machine. Maybe it is the reason of IO problem?</p>

username: ptrblck
cooked: <p>Might be unrelated, but how many workers are you using? Also do you set <code>pin_memory=True</code>?</p>

username: hasakii
cooked: <p>Its mainly caused by IO problem. It seems that you used a super computer, but the swap area is too small. When loading data from disk, it would comsumes too much memory for data buffering.</p>
<p>Here is some suggestions:</p>
<ol>
<li>mapping the input feature to output label one by one and contiguously write to disk.</li>
<li>clean the swap area.</li>
<li>use other machine if possible.</li>
</ol>

username: karlTUM
cooked: <p>Right now I use 8 workers. Yes, I set pin_memory=True. After I put the data back into the local disk, the speed becomes normally.</p>