LR finder for different weight decay values (cycle LR, Hyperparameter optimization)

mayool: <p>Hey Everyone,</p>
<p>I am working on finding the best learning rate (lr) and weight decay (wd) values for my networks and applied the methods described in [1, 2] to find the best lr to help convergence,</p>
<p>I have let the algorithm run for different wd values but are now confused which one to pick:</p>
<p>Here are 2 example plots of different models:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf89700ecc12d689324a8167293b513c31acb277.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf89700ecc12d689324a8167293b513c31acb277" title="lr_analysis"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf89700ecc12d689324a8167293b513c31acb277_2_333x250.png" alt="lr_analysis" data-base62-sha1="tBXijkKCJHRFcnqyHsMfzhTHEvt" width="333" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf89700ecc12d689324a8167293b513c31acb277_2_333x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf89700ecc12d689324a8167293b513c31acb277_2_499x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf89700ecc12d689324a8167293b513c31acb277_2_666x500.png 2x"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lr_analysis</span><span class="informations">5120×3840 851 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> img 1<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/7/b77e960afe208fec9c15664c5d465af9826c2b03.png" data-download-href="https://discuss.pytorch.org/uploads/default/b77e960afe208fec9c15664c5d465af9826c2b03" title="lr_analysis"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77e960afe208fec9c15664c5d465af9826c2b03_2_333x250.png" alt="lr_analysis" data-base62-sha1="qbgAmMuzbBSDEToWFlpnNNuiM5t" width="333" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77e960afe208fec9c15664c5d465af9826c2b03_2_333x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77e960afe208fec9c15664c5d465af9826c2b03_2_499x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/7/b77e960afe208fec9c15664c5d465af9826c2b03_2_666x500.png 2x"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">lr_analysis</span><span class="informations">5120×3840 777 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> img 2</p>
<p>(the algorithm stops early if it detects divergence, therefore some lines are cut of earlier)</p>
<p>what should i consider when picking the lr for the different WD values.<br>
Is it good if the divergence happens at a higher lr (e.g. purple in img 1)?<br>
Is a shallow or a steep drop (around 1e-4 to 1e-1 in img 2) good?<br>
In generell i know i should pick my max lr to be around the lowest point but i struggle to interpret the rest of the plots.<br>
Any help is much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
Cheers, Sven</p>
<p>[1] <a href="https://arxiv.org/abs/1803.09820" class="inline-onebox" rel="noopener nofollow ugc">[1803.09820] A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay</a><br>
[2] <a href="https://arxiv.org/abs/1506.01186" class="inline-onebox" rel="noopener nofollow ugc">[1506.01186] Cyclical Learning Rates for Training Neural Networks</a></p>