Weird spikes in Loss function

iamneerav: <p>My loss function has weird spikes in it that keep on decreasing. I do not understand what inference should I make from this.</p>
<p>This is the plot (zoomed in on y-axis)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/195743ad7e60b041dd0ec50528608bbf6293783c.png" data-download-href="https://discuss.pytorch.org/uploads/default/195743ad7e60b041dd0ec50528608bbf6293783c" title="new"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195743ad7e60b041dd0ec50528608bbf6293783c_2_690x268.png" alt="new" data-base62-sha1="3CaTMQiSh1MHgkZUaT4oMVZCkig" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195743ad7e60b041dd0ec50528608bbf6293783c_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195743ad7e60b041dd0ec50528608bbf6293783c_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/195743ad7e60b041dd0ec50528608bbf6293783c_2_1380x536.png 2x"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">new</span><span class="informations">9000×3500 1.01 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>and the follwing is the original loss plot<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/80835d508d832789d8929e51c9db93a839d74e3e.png" data-download-href="https://discuss.pytorch.org/uploads/default/80835d508d832789d8929e51c9db93a839d74e3e" title="new1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80835d508d832789d8929e51c9db93a839d74e3e_2_690x268.png" alt="new1" data-base62-sha1="ikSyVDVYKv3RWL4OiKZR9j9QZRA" width="690" height="268" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80835d508d832789d8929e51c9db93a839d74e3e_2_690x268.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80835d508d832789d8929e51c9db93a839d74e3e_2_1035x402.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/80835d508d832789d8929e51c9db93a839d74e3e_2_1380x536.png 2x"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">new1</span><span class="informations">9000×3500 499 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using a U-Net architecture to train data of the shape (batch_size, 3, 32, 32, 32).</p>

ptrblck: <p>You could try to capture the iteration loss spikes and manually check, which samples were included in the current batch, which might yield more information. E.g. it could be “bad” or “hard” samples, which create a high loss as well as non-optimal gradients so that the following iterations are used to lower the loss again.<br>
However, this is only pure speculation and it would be interesting to see what you can find out about the data.</p>