How to add L2 Regularization to Parameter groups

Abas1: <p>I have parameter groups down below with different learning rates using the Adam optimizer and I would like to add weight decay. How is this done in pytorch.</p>
<p>Also does adding L2 Regularization become redundant when using a weighted loss function? or could they both be used together</p>
<pre><code class="lang-auto">def define_optimizer():
  optimizer = optim.Adam([
      {'params' : model.classifier.parameters(), 'lr':0.001},
      {'params' : model.features[7].parameters(), 'lr' : 0.00001},
      {'params' : model.features[8].parameters(), 'lr' : 0.00001}
      ])
  return optimizer
</code></pre>

KFrank: <p>Hi Abas!</p>
<aside class="quote no-group" data-username="Abas1" data-post="1" data-topic="219434" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/f9ae1b/48.png" class="avatar"> Abas1:</div>
<blockquote>
<p>I have parameter groups down below with different learning rates using the Adam optimizer and I would like to add weight decay.</p>
</blockquote>
</aside>
<p>Simply use <code>Adam</code>’s <code>weight_decay</code> parameter:</p>
<pre data-code-wrap="python"><code class="lang-python">      {'params' : model.classifier.parameters(), 'lr':0.001, weight_decay:0.00001},
</code></pre>
<p>Note that for optimizers other than plain-vanilla <code>SGD</code>, including <code>Adam</code>, weight decay and<br>
L2 regularization are not technically exactly the same, although they both do pretty much<br>
the same thing.  See for example this discussion <a href="https://www.fast.ai/posts/2018-07-02-adam-weight-decay.html#understanding-adamw-weight-decay-or-l2-regularization" rel="noopener nofollow ugc">“Weight decay or L2 regularization?”</a></p>
<aside class="quote no-group">
<blockquote>
<p>Also does adding L2 Regularization become redundant when using a weighted loss function?</p>
</blockquote>
</aside>
<p>I assume that by “weighted loss” you mean something like class weights such as<br>
<code>CrossEntropyLoss</code>’s <code>weight</code> constructor argument.  If so, this is separate from L2<br>
regularization (or weight decay) and the two can sensibly be used together.</p>
<p>Best.</p>
<p>K. Frank</p>

Abas1: <p>Really appreciate the response Frank. I have a quick question in regards to the Cyclic learning rate scheduler. I do understand according to Leslie n.Smith that before using a scheduler we must determine the min_lr and max_lr. To do this a LR range test is used to find the points where loss begins to increase(as our min) and where loss beings to diverge(our Max).</p>
<p>is there any pytorch implementation for implementing this LR range test?</p>
<p>from what I’ve seen Fast.ai has a implementation but im not sure if its okay to use<br>
from torch_lr_finder import LRFinder</p>
<pre><code class="lang-auto">from torch_lr_finder import LRFinder
model = ...
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.1, weight_decay=1e-2)
lr_finder = LRFinder(model, optimizer, criterion, device="cuda")
lr_finder.range_test(trainloader, val_loader=val_loader, end_lr=1, num_iter=100, step_mode="linear")
lr_finder.plot(log_lr=False)
lr_finder.reset()
</code></pre>
<p>And if I were to use a subset of my data(20% of 35k images) to tune Hyperparameters is the general consensus that the results from these subsets are more likely to reflect the full training data</p>