How to add a L2 regularization term in my loss function

username: JinChengWu
cooked: <h2><a name="hi-im-a-newcomer-i-learned-pytorch-for-a-short-time-and-i-like-it-so-much-1" class="anchor" href="#hi-im-a-newcomer-i-learned-pytorch-for-a-short-time-and-i-like-it-so-much-1"></a>Hi, I’m a newcomer.<br>
I learned Pytorch for a short time and I like it so much.</h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/90af8b995cfab96c5a4415217bc28ead3d08af31.png" alt="%E5%9C%96%E7%89%87" data-base62-sha1="kDWRjcBHAlo9iwG76jcAZmoZFyV" width="454" height="119"></p>
<p>I’m going to compare the difference between with and without regularization, thus I want to custom two loss functions.</p>
<pre><code class="lang-auto">###OPTIMIZER
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr = LR, momentum = MOMENTUM)
</code></pre>
<h2><a name="can-someone-give-me-a-further-example-thanks-a-lot-2" class="anchor" href="#can-someone-give-me-a-further-example-thanks-a-lot-2"></a>Can someone give me a further example?<br>
Thanks a lot!</h2>
<p>BTW, I know that the latest version of TensorFlow can support dynamic graph.<br>
But what is the difference of the dynamic graph between these two frameworks?</p>

username: Separius
cooked: <p>set “weight_decay” parameter to a non zero value in your optimizer(sgd, adam, …)(it’s the alpha in your equation)<br>
edit: I think it’s alpha times two actually</p>

username: JinChengWu
cooked: <aside class="quote no-group quote-modified" data-username="JinChengWu" data-post="1" data-topic="17411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/48/4587_2.png" class="avatar"> JinChengWu:</div>
<blockquote>
<p>def backward</p>
</blockquote>
</aside>
<p>I think I miss one row: def backward<br>
Cuz ‘w’ is the weight. It is updated continuously by steps.<br>
I just wonder if I need to do the grad-decent by myself?</p>

username: Separius
cooked: <p>oh, in that case, iterate over your parameters (for p in self.parameters()) and add (p**2).sum() to your loss</p>

username: JinChengWu
cooked: <p>Can you give me a simple example, ex: MSE<br>
I’d like to know the mechanism of this custom class detaily.<img src="https://discuss.pytorch.org/images/emoji/apple/scream.png?v=5" title=":scream:" class="emoji" alt=":scream:"></p>

username: Separius
cooked: <p>have you seen <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py" rel="nofollow noopener">this</a>?<br>
there are two ways to handle backprop, doing it by hand or using the autograd package (and also a third way which is using both of them, by defining backward)<br>
if you are using the autograd, and your modules are composed of standard operations, you can simply define your loss without the L2 regularizer and in the optimizer define the regularizer</p>
<pre><code class="lang-auto">class custom(nn.Module):
 def __init__(self):
  super().__init__()
 def forward(self, x):
  return x
net = custom()
optimizer = optim.SGD(net.parameters(), lr=0.01, weight_decay=0.01)
criterion = nn.MSELoss()
for batch in batches:
 optimizer.zero_grad()
 y = net(batch['x'])
 loss = criterion(y, batch['y'])
 loss.backward()
 optimizer.step()

</code></pre>

username: JinChengWu
cooked: <p>Sure, I know this is custom neural network.</p>
<p>I use the 3 layers CNN net defined by myself with the nn.MSELoss() before. It’s autograd.<br>
But now I want to compare the results if loss function with or without L2 regularization term.</p>
<p>If I use autograd nn.MSELoss(), I can not make sure if there is a regular term included or not.<br>
p.s.:I checked that parameter ‘weight_decay’ in optim means “add a L2 regular term” to loss function.<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=5" title=":+1:" class="emoji" alt=":+1:"></p>
<p>Furthermore, if I want to add a “L1” norm term in my loss function, I CANNOT USE THE autograd ?</p>

username: Separius
cooked: <p>no, you can always use autograd (even if your function does not have a derivative, you can use something else as derivative and go backward from there), what i meant was that when you have simple functions, there is no need to write backward() yourself<br>
adding L1 loss is simple:</p>
<pre><code class="lang-auto"> loss = mse(pred, target)
 l1 = 0
 for p in net.parameters():
  l1 = l1 + p.abs().sum()
 loss = loss + lambda_l1 * l1
 loss.backward()
 optimizer.step()
</code></pre>

username: Separius
cooked: <aside class="quote no-group" data-username="JinChengWu" data-post="7" data-topic="17411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jinchengwu/48/4587_2.png" class="avatar"> JinChengWu:</div>
<blockquote>
<p>If I use autograd nn.MSELoss(), I can not make sure if there is a regular term included or not.</p>
<p>p.s.:I checked that parameter ‘weight_decay’ in optim means “add a L2 regular term” to loss function.<img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>in general loss of a network has some terms, adding L2 term via optimizer class is really easy and there is no need to explicitly add this term (optimizer does it), so if you want to compare networks, you can simply tune weight_decay</p>

username: derEitel
cooked: <p>I want to follow an implementation of a Keras model in which only on some conv layers an l2 <code>kernel_regularizer</code> has been used. Now I have followed your implementation but am wondering if it suffices to filter for the names of the layers that I want to include my regularization on or not. I.e. along the lines of:</p>
<pre><code class="lang-auto">reg_lambda=0.01
l2_reg = 0
    if isinstance(layer_names, list):
         for W in self.model.named_parameters():
             if "weight" in W[0]:
                 layer_name = W[0].replace(".weight", "")
                 if layer_name in layer_names:
                     l2_reg = l2_reg + W[1].norm(2)
loss = loss + l2_reg * reg_lambda
loss.backward()
</code></pre>

username: fs_z
cooked: <p>I have the same problem that use pytorch to achieve keras’s kernel_regularizer. can you share this part code that have worked .THX</p>

username: mmdbrdrn
cooked: <p>weight decay doesn’t show good results with adam optimizer</p>

username: muammar
cooked: <aside class="quote no-group" data-username="mmdbrdrn" data-post="12" data-topic="17411" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmdbrdrn/48/9966_2.png" class="avatar"> mmdbrdrn:</div>
<blockquote>
<p>weight decay doesn’t show good results with adam optimizer</p>
</blockquote>
</aside>
<p>Why? Could you elaborate more on your reply?</p>

username: hemanth346
cooked: <p>If someone’s landed up here for this, you can check out below answer on stackoverflow</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/44641976/in-pytorch-how-to-add-l1-regularizer-to-activations" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/8126541/bull" target="_blank" rel="nofollow noopener">
    <img alt="Bull" src="https://www.gravatar.com/avatar/5f6cb2a7f9b634936def7f9c831885bd?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/44641976/in-pytorch-how-to-add-l1-regularizer-to-activations" target="_blank" rel="nofollow noopener">In Pytorch, how to add L1 regularizer to activations?</a>
</h4>

<div class="tags">
  <strong>python, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/8126541/bull" target="_blank" rel="nofollow noopener">
    Bull
  </a>
  on <a href="https://stackoverflow.com/questions/44641976/in-pytorch-how-to-add-l1-regularizer-to-activations" target="_blank" rel="nofollow noopener">12:39AM - 20 Jun 17 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Nasim_Shafiee
cooked: <p>optimizer   = optim.Adam(model.parameters(), lr=learning_rate, weight_decay=0.01)</p>

username: avh77
cooked: <p>L2 regularizer and weight_decay in Adam are a bit different thinks, more on that is available here: <a href="https://openreview.net/pdf?id=rk6qdGgCZ" rel="nofollow noopener">https://openreview.net/pdf?id=rk6qdGgCZ</a></p>

username: Prajwal_Sridhar
cooked: <p>How do you experiment with different values for weight_decay? So that you could show the amounts of regularization on the x axis and validation set performance on the y axis</p>

username: JINGHAN_YANG
cooked: <p>May I ask are you sure it is 2<em>alpha instead of 1/2</em>alpha? Where I can find this?</p>

username: Siladittya_Manna
cooked: <p>I recently used this to add L2 regularization with LBFGS optimizer for a linear classification model</p>
<pre><code>       reg_loss = 0
        for mod in self.model.modules():
            if isinstance(mod, _BatchNorm):
                if self.decay_bn:
                    for name, param in mod.named_parameters(recurse=False):
                        reg_loss = reg_loss + param.norm(2)    
            else:
                for name, param in mod.named_parameters(recurse=False):
                    if not name.endswith("bias"):
                        reg_loss = reg_loss + param.norm(2)
                    else:
                        if self.decay_bias:
                            reg_loss = reg_loss + param.norm(2)
</code></pre>