MSE loss function decreasing very slowly

aswamy: <p>I have MSE loss that is computed between ground truth image and the generated image.<br>
The loss is decreasing/converging but very slowlly(below image). Any suggestions in terms of tweaking the optimizer?  I am currently using adam optimizer with lr=1e-5.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/695c26e5f15f5d46f82bbb3d5065f80327d817b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/695c26e5f15f5d46f82bbb3d5065f80327d817b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695c26e5f15f5d46f82bbb3d5065f80327d817b5_2_690x421.png" alt="image" data-base62-sha1="f23zhm4sDZ240TxNbhHjE23WbAN" width="690" height="421" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/9/695c26e5f15f5d46f82bbb3d5065f80327d817b5_2_690x421.png, https://discuss.pytorch.org/uploads/default/original/3X/6/9/695c26e5f15f5d46f82bbb3d5065f80327d817b5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/9/695c26e5f15f5d46f82bbb3d5065f80327d817b5.png 2x" data-dominant-color="F2F3F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">782×478 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

knoriy: <p>The reason for your model converging so slowly is because of your leaning rate (<code>1e-5 == 0.000001</code>), play around with your learning rate. I find default works fine for most cases.</p>
<p>try: <code>1e-2</code></p>
<p>or you can use a learning rate that changes over time as discussed <a href="https://discuss.pytorch.org/t/how-to-do-exponential-learning-rate-decay-in-pytorch/63146/3">here</a></p>

aswamy: <p>I tried a higher learning rate than 1e-5, which leads to a gradient explosion. if you observe up to 2k iterations the rate of decrease of error is pretty good but after that, the rate of decrease slows down, and towards 10k+ iterations it almost dead and not decreasing at all.<br>
Yeah, I will try adapting the learning rate. Is there any guide on how to adapt?</p>

knoriy: <p>Looking at the plot again, your model looks to be about 97-98% accurate. Is that correct?</p>
<p>I’m not aware of any guides that give a comprehensive overview, but you should find other discussion boards that explore this topic, such as the link in my previous reply.</p>

rahulvigneswaran: <p>Hey <a class="mention" href="/u/aswamy">@aswamy</a>,</p>
<p>The replies from <a class="mention" href="/u/knoriy">@knoriy</a> explains your situation better and is something that you should try out first. In case you need something extra, you could look into the learning rate schedulers.</p>
<ul>
<li>
<p>Beginner friendly <a href="https://www.deeplearningwizard.com/deep_learning/boosting_models_pytorch/lr_scheduling/" rel="noopener nofollow ugc">article</a> on it.</p>
</li>
<li>
<p><a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">PyTorch documentation</a> (Scroll to “How to adjust learning rate” header)</p>
</li>
</ul>

aswamy: <p>yeah, Correct! Thanks!</p>

knoriy: <p>Without knowing what your task is, I would say that would be considered close to the state of the art.</p>
<p>Once your model gets close to these figures, in my experience the model finds it hard to find new feature to optimise without overfitting to your dataset. <a href="https://towardsdatascience.com/gradient-descent-explained-9b953fc0d2c" rel="noopener nofollow ugc">Some reading materials</a></p>
<p>You may also want to learn about <a href="https://www.dna-ghost.com/single-post/2018/03/13/Neural-network-Escaping-from-variety-of-non-global-minimum-traps" rel="noopener nofollow ugc">non-global minimum traps</a>.</p>