About learning rate and weight decay "fine-tuning"

gass_jb: <p>I am really confused about choosing the best learning rate and weight decay. I conducted an ablation study and got the following results:</p>
<ul>
<li>
<p>For both experiments a weight decay = 0.001 was used.</p>
</li>
<li>
<p>For the first plot: lr = 0.003  =&gt; best validation loss: 0.405 reached in epoch 4, train_loss = 0.326 at same epoch</p>
</li>
<li>
<p>For the second plot: lr = 0.03 =&gt; best validation loss: 0.403 reached in epoch 9, train_loss = 0.411 also at same epoch</p>
</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8c266cd4ff83c6973f962729f099d3aa804f6613.png" data-download-href="https://discuss.pytorch.org/uploads/default/8c266cd4ff83c6973f962729f099d3aa804f6613" title="lr0,003-wd0,001"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c266cd4ff83c6973f962729f099d3aa804f6613_2_690x364.png" alt="lr0,003-wd0,001" data-base62-sha1="jZPaCwQLPaGMr9yYobyoWC25b0L" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c266cd4ff83c6973f962729f099d3aa804f6613_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c266cd4ff83c6973f962729f099d3aa804f6613_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/c/8c266cd4ff83c6973f962729f099d3aa804f6613_2_1380x728.png 2x" data-dominant-color="FEFEFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lr0,003-wd0,001</span><span class="informations">1848×977 47.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15.png" data-download-href="https://discuss.pytorch.org/uploads/default/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15" title="lr0,03-wd0,001"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15_2_690x364.png" alt="lr0,03-wd0,001" data-base62-sha1="9cAaypnS3Z2W71YHCfBevZy8M1n" width="690" height="364" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15_2_690x364.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15_2_1035x546.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/407fb5210ce6564c5c2ba69dc7ce70cb90f24f15_2_1380x728.png 2x" data-dominant-color="FEFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lr0,03-wd0,001</span><span class="informations">1848×977 53.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sayed_Nadim: <p>I would suggest using hyperparameter search for finding the best values. Try <a href="https://optuna.org/" rel="noopener nofollow ugc">Optuna</a>, <a href="https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html" rel="noopener nofollow ugc">Raytune</a> etc. for tuning. Refer to <a href="https://stackoverflow.com/questions/44260217/hyperparameter-optimization-for-pytorch-model" rel="noopener nofollow ugc">This question</a>.</p>

gass_jb: <p>Thx Sayed for your quick answer, but I would like to get best analysis of the above-shared plots, which one performed better.</p>

Sayed_Nadim: <p>Understood. In my opinion, with smaller lr, you don’t risk “overshoot” during gradient descent. But it may take more time converge. Higher lr can make training faster but also can “oscillate” for convergence.<br>
In the case lr = 0.003, the validation loss seems stable, with no significant change. I am not sure if it will improve further from that. In the second case where lr = 0.03, the validation loss goes high and then drops significantly low. It may indicate a bit of overfitting tendency.</p>
<p>I would recommend training for some more epochs for both cases to see if the validation value improves. Also, are you performing weight decay based on epoch or steps?</p>

gass_jb: <p>I trained for some more epochs, the model starts overfitting and the values went higher for both cases. The weight decay is performed on epoch</p>

gass_jb: <p>This is what what I got,  when I trained for more epochs with lr = 0.003</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d79ba840d363f5524ecd7a6307ee4fe14c94b9b9.png" alt="lr0,003-wd0,001_8-epochs" data-base62-sha1="uLm9jxcRwfjy347izSMZvPFCaCt" width="640" height="480"></p>

Sayed_Nadim: <p>Can you try with linear weight decaying instead of fixed value i.e. 1st epoch - 0.001, 2nd epoch - 0.0008, 3rd epoch - 0.0004 etc?  I guess for lr = 0.003, having a constant weight decay of 0.001 is reducing lr faster than with lr = 0.03.</p>

Sayed_Nadim: <p>Yes, the model is diverging (potentially overfitting) when you train for more epoch.</p>