Sudden deterioration of model

pointlessone: <p>I’m new to ML and PyTorch in particular so might’ve missed something but please help me understand what’s happening.</p>
<p>I have a simple model. It learns pretty well and pretty quickly reaches 100% accuracy but in the span of just two epochs it completely deteriorates. Loss shoots way up and accuracy becomes 0%. I’ve seen spikiness in my previous attempts and elsewhere on the internet. I’ve also seen a few question about this on StackOverflow and here on this forum but I haven’t seen an explanation of the phenomenon, at least not a one I can understand yet.</p>
<p>I have a Sequential model. I use <code>SoftMarginLoss</code> (I also tried <code>MSELoss</code> with about the same result) for loss function and <code>Adam</code> for optimiser. I have about 65k data points in total. I randomly select a quarter of the dataset for training in each epoch and 1000 points for testing.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7be46ef13ba54019e8f8037efadfe9d2152957ab.png" data-download-href="https://discuss.pytorch.org/uploads/default/7be46ef13ba54019e8f8037efadfe9d2152957ab" title="run1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7be46ef13ba54019e8f8037efadfe9d2152957ab_2_690x230.png" alt="run1" data-base62-sha1="hG08ZLn7SOmUaLoRV79uA4Bm3XR" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7be46ef13ba54019e8f8037efadfe9d2152957ab_2_690x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7be46ef13ba54019e8f8037efadfe9d2152957ab_2_1035x345.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7be46ef13ba54019e8f8037efadfe9d2152957ab_2_1380x460.png 2x" data-dominant-color="FEFEFE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">run1</span><span class="informations">3000×1000 50.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is a plot of Loss and Accuracy. As you can see, on epoch 41 accuracy tanks to 0 and loss shoots up to 0.8, higher even than the first epoch. It recovers to 0.69 but stays there for a long time. Can someone please explain what’s going on here?</p>

ptrblck: <aside class="quote no-group" data-username="pointlessone" data-post="1" data-topic="202562">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pointlessone/48/69951_2.png" class="avatar"> pointlessone:</div>
<blockquote>
<p>on epoch 41 accuracy tanks to 0 and loss shoots up to 0.8</p>
</blockquote>
</aside>
<p>An accuracy of <code>0</code> is not expected even if the model diverges as it would be worse than a random predictor. I don’t know what kind of use case you are working on but e.g. in a binary classification use case you could just invert the predictions to achieve an accuracy of <code>100</code>, so you should check what the random accuracy would be and why your model gets worse than that.</p>

pointlessone: <p>The model is supposed to predict a series of binary outputs. An accuracy of 0% means that at least one of the outputs is not close enough to expectation. It doesn’t mean that all outputs are the opposite but it’s an intriguing idea and I will check if that’s actually the case. I’m probably not using “accuracy” in the industry-standard way so might be a bit confused about that.</p>
<p>I wonder though about the loss. Isn’t optimizer supposed to press it towards 0? Why does loss jump so high and can’t get back down?</p>