Simple L2 regularization?

Kalamaya: <p>Hi, does simple L2 / L1 regularization exist in pyTorch? I did not see anything like that in the losses.</p>
<p>I guess the way we could do it is simply have the data_loss + reg_loss computed, (I guess using nn.MSEloss for the L2), but is an explicit way we can use it without doing it this way?</p>
<p>Thanks</p>

fmassa: <p>Hi,</p>
<p>The L2 regularization on the parameters of the model is already included in most optimizers, including <code>optim.SGD</code> and can be controlled with the <code>weight_decay</code> parameter as can be seen in <a href="http://pytorch.org/docs/optim.html#torch.optim.SGD">the SGD documentation</a>.</p>
<p>L1 regularization is not included by default in the optimizers, but could be added by including an extra loss <code>nn.L1Loss</code> in the weights of the model.</p>
<pre><code class="lang-python">l1_crit = nn.L1Loss(size_average=False)
reg_loss = 0
for param in model.parameters():
    reg_loss += l1_crit(param)

factor = 0.0005
loss += factor * reg_loss
</code></pre>
<p>Note that this might not be the best way of enforcing sparsity on the model though.</p>

Kalamaya: <p>Thanks <a class="mention" href="/u/fmassa">@fmassa</a> - although I must say that’s odd that a regularization loss in included in the <em>optimizer</em> here.  0_0</p>

apaszke: <p>Yeah, that’s been added there as an optimization, as L2 regularization is often used.</p>

Kalamaya: <p>Got it thanks! <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ecolss: <p>what do you recommend which would be a better way to enforce sparsity instead of L1?</p>

fmassa: <p>This comment might be helpful <a href="https://github.com/torch/optim/pull/41#issuecomment-73935805">https://github.com/torch/optim/pull/41#issuecomment-73935805</a></p>

ecolss: <p>Thanks for the note <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ncullen93: <p><a class="mention" href="/u/fmassa">@fmassa</a> does this still work? It seems that <code>nn.L1Loss</code> requires a target - giving the error <code>TypeError: forward() missing 1 required positional argument: 'target'</code></p>
<p>example:</p>
<pre><code class="lang-python">xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx)
</code></pre>

fmassa: <p>I forgot to add the <code>target</code>, which in some cases would be a zero-tensor.<br>
So something like</p>
<pre><code class="lang-python">xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
target = Variable(torch.from_numpy(np.zeros((3,3))))
l1_crit = nn.L1Loss()
l1_crit(xx, target)
</code></pre>

Ja-Keoung_Koo: <p>I have two questions about L1 regularization:</p>
<ul>
<li>
<p>How do we backpropagate for L1 regularization? I wonder it because the term is not differentiable.</p>
</li>
<li>
<p>Where can I see the implementation of L1 regularization? In the following link, there is only <code>pass</code>.</p>
</li>
</ul>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/ecd51f8510bb1c593b0613f3dc7caf31dc29e16b/torch/nn/modules/loss.py#L39" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/ecd51f8510bb1c593b0613f3dc7caf31dc29e16b/torch/nn/modules/loss.py#L39" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/ecd51f8510bb1c593b0613f3dc7caf31dc29e16b/torch/nn/modules/loss.py#L39</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="29" style="counter-reset: li-counter 28 ;">
<li>def __init__(self, weight=None, size_average=True):</li>
<li>    super(_WeightedLoss, self).__init__(size_average)</li>
<li>    self.register_buffer('weight', weight)</li>
<li>
</li>
<li>def forward(self, input, target):</li>
<li>    _assert_no_grad(target)</li>
<li>    backend_fn = getattr(self._backend, type(self).__name__)</li>
<li>    return backend_fn(self.size_average, weight=self.weight)(input, target)</li>
<li>
</li>
<li>
</li>
<li class="selected">class L1Loss(_Loss):</li>
<li>r"""Creates a criterion that measures the mean absolute value of the</li>
<li>element-wise difference between input `x` and target `y`:</li>
<li>
</li>
<li>:math:`{loss}(x, y)  = 1/n \sum |x_i - y_i|`</li>
<li>
</li>
<li>`x` and `y` arbitrary shapes with a total of `n` elements each.</li>
<li>
</li>
<li>The sum operation still operates over all the elements, and divides by `n`.</li>
<li>
</li>
<li>The division by `n` can be avoided if one sets the constructor argument `size_average=False`</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


smth: <p>you can see the implementation of L1Loss here: <a href="https://github.com/pytorch/pytorch/blob/ecd51f8510bb1c593b0613f3dc7caf31dc29e16b/torch/lib/THNN/generic/L1Cost.c">https://github.com/pytorch/pytorch/blob/ecd51f8510bb1c593b0613f3dc7caf31dc29e16b/torch/lib/THNN/generic/L1Cost.c</a></p>

gwg: <p><a class="mention" href="/u/fmassa">@fmassa</a>, so because of PyTorch’s autograd functionality, we do not need to worry about L1 regularization during the backward pass, i.e. applying the derivative of the L1 regularization term to the gradient of the output? That will be handled by the autograd variables?</p>

fmassa: <p>If you add the L1 regularization to the loss as I explained in a previous post, the gradients will be handled by autograd automatically.</p>

yibo: <p>I like to use <code>l1_loss=F.l1_loss(xx, target=torch.zeros_like(xx), size_average=False)</code></p>

Brando_Miranda: <p>if I want to use a custom Regularizer <code>R</code> can is the following code good:</p>
<pre><code>batch_loss = loss(input=y_pred,target=batch_ys)
batch_loss += lambda*R
</code></pre>
<p>?</p>

hughperkins: <p><a class="mention" href="/u/fmassa">@fmassa</a> You say “this might not be the best way of enforcing sparsity on the model”, and “This comment might be helpful <a href="https://github.com/torch/optim/pull/41#issuecomment-73935805" rel="nofollow noopener">https://github.com/torch/optim/pull/41#issuecomment-73935805</a> 1.2k”, which comment explains to explicitly set to 0 any weights changing sign. Does this mean that you feel that L1 with explicit zeroing of weights crossing zero is an appropriate way of encouraging sparsity? Or do you mean, there are some other approach(es) that can work well?</p>

Neta_Zmora: <p>Hi <a class="mention" href="/u/hughperkins">@hughperkins</a>,</p>
<p>If you are interested in inducing sparsity, you might want to checkout <a href="https://github.com/NervanaSystems/distiller" rel="nofollow noopener">this project</a> from Intel AI Labs.  The <a href="https://nervanasystems.github.io/distiller/index.html" rel="nofollow noopener">documentation</a> tries to shed some light on recent research related to sparsity inducing methods.</p>
<p>The project includes a stand-alone Jupyter <a href="https://github.com/NervanaSystems/distiller/blob/master/jupyter/L1-regularization.ipynb" rel="nofollow noopener">notebook that attempts to show how L1 regularization</a> can be used to induce sparsity (by “stand-alone” I mean that the notebook does not import any code from Distiller, so you can just try it out).</p>
<p>Cheers,<br>
Neta</p>

shirui-japina: <p>Sorry for question here.<br>
It is said that when regularization L2, it should <strong>only</strong> for <strong>weight parameters</strong>, but not <strong>bias parameters</strong>.(Is it right?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20">)<br>
But the L2 regularization included in most optimizers in PyTorch, is for all of the parameters in the model (weight and bias).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32a7aae3a3acf117c289c5561aca95ca064c3fe3.png" alt="least_squares_l2" data-base62-sha1="7e76aIjGlAYsMkRrI28Jq2JXGLx" width="518" height="74"><br>
I mean the parameters in the red box should be weight parameters only. (If what I heard of is right.)<br>
How can I deal with it?</p>
<pre><code class="lang-auto">weight_p, bias_p = [],[]
for name, p in model.named_parameters():
  if 'bias' in name:
    bias_p += [p]
  else:
    weight_p += [p]

optim.SGD(
  [
    {'params': weight_p, 'weight_decay':1e -5},
    {'params': bias_p, 'weight_decay':0}
  ],
  lr=1e-2, momentum=0.9
)
</code></pre>
<p>Code here can deal with the problem above, is it right?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"></p>

Deepak: <p>We do regularization to handle the high variance problem (overfitting). It’s good if the regularization includes all the learnable parameters (both weight and bias). But since bias is only a single parameter out of the large number of parameters, it’s usually not included in the regularization; and exclusion of bias hardly affects the results.</p>