Loss explodes is equal to overfitting?

Christian_Ses: <p>For semantic segmentation I trained a network using a learning rate of 5*10**-4, mini-batchsize 2 and a Momentum of 0.8 with image size 256x384. The only custom transformation applied is a center crop. The loss after 20 Epochs looks like the following:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d1358611512b4bec3e7dca80027037ee1b3e8831.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1358611512b4bec3e7dca80027037ee1b3e8831" title="lr5e-4b2x256x384Mom8-noTransforms-reduceUpsample"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1358611512b4bec3e7dca80027037ee1b3e8831_2_666x500.png" alt="lr5e-4b2x256x384Mom8-noTransforms-reduceUpsample" data-base62-sha1="tQKsQIcDOMaRLr4tiHL766smNtn" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1358611512b4bec3e7dca80027037ee1b3e8831_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1358611512b4bec3e7dca80027037ee1b3e8831_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d1358611512b4bec3e7dca80027037ee1b3e8831.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lr5e-4b2x256x384Mom8-noTransforms-reduceUpsample</span><span class="informations">1280×960 54.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Question:</p>
<p>Why is there a peak in the loss function, when there is no custom transformation applied? Is the network to small?</p>

LeviViana: <p>It doesn’t look like over-fitting to me. Its hard to tell without much information. Does it happens every time when you train the model ?</p>

Christian_Ses: <p>Yes it happens ever time training the model.</p>
<p>When the mini batch is greater than 2 it happens even more. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/162dbb420d6ec51785ef063702b3f7261015b929.png" data-download-href="https://discuss.pytorch.org/uploads/default/162dbb420d6ec51785ef063702b3f7261015b929" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/162dbb420d6ec51785ef063702b3f7261015b929_2_666x500.png" alt="loss" data-base62-sha1="3actYUflluQ51OQY6wkal8Mzijv" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/162dbb420d6ec51785ef063702b3f7261015b929_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/162dbb420d6ec51785ef063702b3f7261015b929_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/1/162dbb420d6ec51785ef063702b3f7261015b929.png 2x" data-dominant-color="FBF8F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1280×960 77.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/3436f9a81e7f246b739bee1b3dab42b53ca57dc5.png" data-download-href="https://discuss.pytorch.org/uploads/default/3436f9a81e7f246b739bee1b3dab42b53ca57dc5" title="IoU"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3436f9a81e7f246b739bee1b3dab42b53ca57dc5_2_666x500.png" alt="IoU" data-base62-sha1="7rUBXNVKvakj99XtisGQ7pwZcwt" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/3436f9a81e7f246b739bee1b3dab42b53ca57dc5_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/3436f9a81e7f246b739bee1b3dab42b53ca57dc5_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/3436f9a81e7f246b739bee1b3dab42b53ca57dc5.png 2x" data-dominant-color="FBF9F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IoU</span><span class="informations">1280×960 73 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

LeviViana: <p>Indeed, it is quite odd. My guess is that it will require a deep debug. BTW, did you check that the train and validation data-sets have the same distribution ?</p>
<p>Otherwise, I’d pick two consecutive checkpoints: a good and a bad. Then, I’d pick a sample were both predict good results and a sample where only the good predicts the good results. Then, I’d look at the last activations, and see the differences. It may take a lot of time, and maybe the solution wouldn’t be straightforward.</p>

Oli: <p>What kind of network are you using? Does the network behave differently in train() and valid() mode? How big is your validation dataset?</p>

barthelemymp: <p>Looks like like loss evolution of Elman network … What s is your model?<br>
Problem was back then solved by gates (GRU)</p>

Christian_Ses: <aside class="quote no-group" data-username="Oli" data-post="5" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>What kind of network are you using?</p>
</blockquote>
</aside>
<p>Im using a semantic segmentation network, which is also incorporating the motion state to differentiate between moving and static cars.</p>
<aside class="quote no-group" data-username="Oli" data-post="5" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>How big is your validation dataset?</p>
</blockquote>
</aside>
<p>I used the cityscapes motion dataset [<a href="http://deepmotion.cs.uni-freiburg.de" rel="noopener nofollow ugc">deepmotion</a>], which provides 2975 training images and 500 validation images. Because I train only small images, I splitted the images in 3 overlapping frames, to get 3 times the amount of training images.</p>
<aside class="quote no-group" data-username="Oli" data-post="5" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>Does the network behave differently in train() and valid() mode?</p>
</blockquote>
</aside>
<p>Yes the network behave totally different in  train() and valid() mode. Is there a reason for it ? I used some dropout layers.</p>

Christian_Ses: <p>The model is looking like the following:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f7b0b2374c243a656dec874bad8a03fbb8658607.png" data-download-href="https://discuss.pytorch.org/uploads/default/f7b0b2374c243a656dec874bad8a03fbb8658607" title="model"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7b0b2374c243a656dec874bad8a03fbb8658607_2_690x411.png" alt="model" data-base62-sha1="zlavj6rbFupiwAxxU4QFzUqyWub" width="690" height="411" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7b0b2374c243a656dec874bad8a03fbb8658607_2_690x411.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7b0b2374c243a656dec874bad8a03fbb8658607_2_1035x616.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f7b0b2374c243a656dec874bad8a03fbb8658607_2_1380x822.png 2x" data-dominant-color="E2E5DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">model</span><span class="informations">1703×1015 270 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Im preprocessing the optical flow with FlowGenerationNetwork (PWCNet) and the segmentation features (with Deeplab) and inputting it in the final network, for calculating the Lovasz loss.</p>
<p>…</p>

Christian_Ses: <aside class="quote no-group" data-username="LeviViana" data-post="4" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>Indeed, it is quite odd. My guess is that it will require a deep debug. BTW, did you check that the train and validation data-sets have the same distribution ?</p>
</blockquote>
</aside>
<p>I also switched some training images with the validation set and trained it. It gave me more or less the same results.</p>
<aside class="quote no-group" data-username="LeviViana" data-post="4" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>Otherwise, I’d pick two consecutive checkpoints: a good and a bad. Then, I’d pick a sample were both predict good results and a sample where only the good predicts the good results. Then, I’d look at the last activations, and see the differences. It may take a lot of time, and maybe the solution wouldn’t be straightforward.</p>
</blockquote>
</aside>
<p>Thats a good idea. I have already plotted some results with high loss. Usually the prediction is biased towards the biggest class, which is the background. (I have just 3 classes: moving car, static car and background, see illustration from [<a href="http://deepmotion.cs.uni-freiburg.de" rel="noopener nofollow ugc">DeepMotion</a>])</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/6cc6510754d46f2c1a37b0c59002b62a4601ee0e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6cc6510754d46f2c1a37b0c59002b62a4601ee0e" title="cityscapes"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6cc6510754d46f2c1a37b0c59002b62a4601ee0e_2_345x172.jpeg" alt="cityscapes" data-base62-sha1="fwgsovBayfCf6P5MUepDW1o6uPc" width="345" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/6cc6510754d46f2c1a37b0c59002b62a4601ee0e_2_345x172.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6cc6510754d46f2c1a37b0c59002b62a4601ee0e_2_517x258.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/6/6cc6510754d46f2c1a37b0c59002b62a4601ee0e_2_690x344.jpeg 2x" data-dominant-color="444F55"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cityscapes</span><span class="informations">768×384 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Oli: <p>Ok, 500 images is large enough of a validation set to not produce those big fluctuations. People sometimes leave the model in train() mode during evaluation and a batch size of 1, so the batch norm layers aren’t happy. I don’t think I can offer any insightful help, sorry. Good luck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

LeviViana: <aside class="quote no-group" data-username="Christian_Ses" data-post="9" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christian_ses/48/12490_2.png" class="avatar"> Christian_Ses:</div>
<blockquote>
<p>Usually the prediction is biased towards the biggest class, which is the background.</p>
</blockquote>
</aside>
<p>I imagine you are using Cross-Entropy loss somewhere. You could try to balance the class importance on the loss by setting different weights. Maybe <a href="https://discuss.pytorch.org/t/what-is-the-weight-values-mean-in-torch-nn-crossentropyloss/11455/2">this thread</a> could help a bit.</p>

Christian_Ses: <aside class="quote no-group" data-username="LeviViana" data-post="11" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/leviviana/48/9548_2.png" class="avatar"> LeviViana:</div>
<blockquote>
<p>I imagine you are using Cross-Entropy loss somewhere. You could try to balance the class importance on the loss by setting different weights. Maybe <a href="https://discuss.pytorch.org/t/what-is-the-weight-values-mean-in-torch-nn-crossentropyloss/11455/2">this thread</a> could help a bit.</p>
</blockquote>
</aside>
<p>When I was using the cross-entropy loss, it was even more fluctuating. This is why Iam using the Lovasz loss, which is taking the IoU (L = 1 - IoUc). [<a href="http://openaccess.thecvf.com/content_cvpr_2018/papers/Berman_The_LovaSz-Softmax_Loss_CVPR_2018_paper.pdf" rel="noopener nofollow ugc">Lovasz Softmax Paper</a>] [<a href="http://www.ds3-datascience-polytechnique.fr/wp-content/uploads/2018/06/DS3-366.pdf" rel="noopener nofollow ugc">Lovasz Overview Slide</a>]</p>
<p>Should I also balance the classes for the Lovasz loss function ?</p>

Christian_Ses: <aside class="quote no-group" data-username="Oli" data-post="10" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oli/48/9773_2.png" class="avatar"> Oli:</div>
<blockquote>
<p>Ok, 500 images is large enough of a validation set to not produce those big fluctuations. People sometimes leave the model in train() mode during evaluation and a batch size of 1, so the batch norm layers aren’t happy. I don’t think I can offer any insightful help, sorry. Good luck <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>In training mode I use a batch size of 12 and in evaluation mode I use a batch size of 1. Is this a problem?  But im always calculating the mean of the whole validation epoch for plotting the epochs.</p>

Oli: <aside class="quote no-group" data-username="Christian_Ses" data-post="13" data-topic="48031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/christian_ses/48/12490_2.png" class="avatar"> Christian_Ses:</div>
<blockquote>
<p>In training mode I use a batch size of 12 and in evaluation mode I use a batch size of 1. Is this a problem? But im always calculating the mean of the whole validation epoch for plotting the epochs.</p>
</blockquote>
</aside>
<p>No it shouldn’t be a problem as long as you do <code>model.train()</code> and <code>model.eval()</code></p>

Christian_Ses: <p>One more question, when I want to further train the pertained model with an even smaller dataset with 700 images in the training dataset and 500 images in the validation. Is better to choose a really small learning rate in comparison to the first training?</p>
<p>I guess weight decay should be also much higher.</p>

jmlb: <ol>
<li>Are you using any data augmentation? If you do, turn off all augmentation schemes, and train exclusively on natural images. Goal is to make sure there is nothing wrong with the data augmentation script.</li>
<li>Also, visualize the images in your batch (input and output): to make sure that  the images that are fed to the model are consistent with what you expect.<br>
I have experienced  similar “noisy” validation loss, and it originated either from a bug in the data augmentation, or in the batch generator.</li>
</ol>

Christian_Ses: <p>Thanks for your information. I already did a training without data augmentation.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/f/f541d8c191de1ae01fd5856a83dab1051d5bc692.png" data-download-href="https://discuss.pytorch.org/uploads/default/f541d8c191de1ae01fd5856a83dab1051d5bc692" title="loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f541d8c191de1ae01fd5856a83dab1051d5bc692_2_666x500.png" alt="loss" data-base62-sha1="yZE3tVlm29RYUzHmho0CJoVZplM" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/f/f541d8c191de1ae01fd5856a83dab1051d5bc692_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/f/f541d8c191de1ae01fd5856a83dab1051d5bc692_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/f/f541d8c191de1ae01fd5856a83dab1051d5bc692.png 2x" data-dominant-color="FBFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss</span><span class="informations">1280×960 62.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But its not looking much better.</p>
<p>I used SGD and CosinWithRestarts. Is it better to use other values tmax or a different scheduler?</p>
<pre><code class="lang-auto">
optimizer = optim.SGD(params, 0.001, momentum=0.95, weight_decay=0.001)
scheduler = CosineWithRestarts(optimizer, t_max=30)
</code></pre>

henrique: <p>Sorry for the stupid question, maybe you have also already solved it, but are you really sure you are averaging the validation for all 500 images every epoch?<br>
Your training loss should be way noisier than a mean of all validation images.</p>
<p>Unless there is a problem on your Lovasz loss…<br>
Have you tried to optimize a smoothed IoU or DICE loss instead?</p>