Running Error Message

Rui_Liu: <p>Hi,</p>
<p>I’m trying to finish a multi-classification problem. In detail, I’m trying to determine the speech either contains some specific noise or no noise.</p>
<p>After training my model 100 epochs, the Cuda utilization suddenly jumps down to 0 - 2%. In the previous epochs, that was about 60 -70%.</p>
<p>Even though the error message reports a format issue, I don’t think that’s the file format issue because in the previous epochs I have already successfully loaded the same file with the same code.</p>
<p>I guess this issue just pops out suddenly.</p>
<p>Does anyone have some thoughts on that?</p>
<p>I’ll really appreciate it!<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc112029b6f350a77b0269fe7a2d81d402dd0ae8.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc112029b6f350a77b0269fe7a2d81d402dd0ae8" title="a07a4e226ae93ade256148e5f732e94"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc112029b6f350a77b0269fe7a2d81d402dd0ae8_2_690x351.png" alt="a07a4e226ae93ade256148e5f732e94" data-base62-sha1="t7g5V1ebk49FgKsRB0HJcxayStq" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc112029b6f350a77b0269fe7a2d81d402dd0ae8_2_690x351.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc112029b6f350a77b0269fe7a2d81d402dd0ae8_2_1035x526.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc112029b6f350a77b0269fe7a2d81d402dd0ae8_2_1380x702.png 2x" data-dominant-color="F2F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">a07a4e226ae93ade256148e5f732e94</span><span class="informations">1840×936 78.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

tom: <aside class="quote no-group" data-username="Rui_Liu" data-post="1" data-topic="134828">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rui_liu/48/38437_2.png" class="avatar"> Rui_Liu:</div>
<blockquote>
<p>Even though the error message reports a format issue, I don’t think that’s the file format issue because in the previous epochs I have already successfully loaded the same file with the same code.</p>
<p>I guess this issue just pops out suddenly.</p>
<p>Does anyone have some thoughts on that?</p>
</blockquote>
</aside>
<p>One thing you could check if it is a memory issue. My thinking around this is that “unknown format” might really be something like “something went wrong while decoding” and if your training leaks memory, it might end up not being able to load whatever it needs for decoding.</p>
<p>It’s just a guess, though.</p>
<p>Best regards</p>
<p>Thomas</p>

Rui_Liu: <p>Thanks for your idea!</p>
<p>I’m not familiar with “training leaks memory”, but it does make sense to me.</p>
<p>I will search more on that. Thanks for your help anyway!</p>

tom: <p>So there will be better ways to monitor this, but my very simple recipe for getting a first impression:</p>
<ul>
<li>Start the training,</li>
<li>Every now and then I run the program “top”  in a console window and after hitting “m” it will sort processes by memory use. The column RSS is the one I look at.</li>
<li>If it keeps increasing between the 5th and 10th epoch (or so) and continues to increase between 10th and 20th, you might have something that keeps using memory and never returns it (i.e. a memory “leak”).</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>