L2 regularization with only weight parameters

username: shirui-japina
cooked: <p>I asked <a href="https://discuss.pytorch.org/t/simple-l2-regularization/139/19">here</a> but got no answer. Maybe it’s a topic which is many years ago, and my question is a little bit inconsistent with the original one. So I’m sorry but ask again here.</p>
<p>My question is,<br>
it is said that when regularization L2, it should  <strong>only</strong>  for  <strong>weight parameters</strong> , but not  <strong>bias parameters</strong> .(if regularization L2 is for all parameters, it’s very easy for the model to become overfitting, is it right?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20">)<br>
But the L2 regularization included in most optimizers in PyTorch, is for all of the parameters in the model (weight and bias).<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/2/32a7aae3a3acf117c289c5561aca95ca064c3fe3.png" alt="least_squares_l2" data-base62-sha1="7e76aIjGlAYsMkRrI28Jq2JXGLx" width="518" height="74"><br>
I mean the parameters in the red box should be weight parameters only. (If what I heard of is right.)</p>
<p>And the way to deal with it is code here:</p>
<pre><code class="lang-auto">weight_p, bias_p = [],[]
for name, p in model.named_parameters():
  if 'bias' in name:
    bias_p += [p]
  else:
    weight_p += [p]

optim.SGD(
  [
    {'params': weight_p, 'weight_decay':1e -5},
    {'params': bias_p, 'weight_decay':0}
  ],
  lr=1e-2, momentum=0.9
)
</code></pre>
<p>is it right?<img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"><img src="https://discuss.pytorch.org/images/emoji/apple/flushed.png?v=12" title=":flushed:" class="emoji" alt=":flushed:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>The code looks fine.<br>
I’m not sure, if an L2 regularization of the bias terms leads to overfitting.<br>
Do you have any references on it, as it’s quite interesting?</p>

username: shirui-japina
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="56853" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>The code looks fine.</p>
</blockquote>
</aside>
<p>Thanks for your checking code for me <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20">.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="56853" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not sure, if an L2 regularization of the bias terms leads to overfitting.<br>
Do you have any references on it, as it’s quite interesting?</p>
</blockquote>
</aside>
<p>The contents in <em><strong>Updata 2019-9-30</strong></em> <a href="https://blog.csdn.net/loseinvain/article/details/81708474#commentsedit" rel="noopener nofollow ugc">here</a>.<br>
But that’s written in Chinese, you may need Google Translate.</p>

username: fly2mars
cooked: <p>Good post!<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a>  as the blog tells,  an L2 regularization of the bias terms leads to <strong>underfitting</strong>, ragher than overfitting. You can have a look at the plot it shows.</p>

username: zeakey
cooked: <p>In the article <a class="mention" href="/u/shirui-japina">@shirui-japina</a> gave, the author refers the Deep learning book by Goodfellow et.al.</p>
<p>And it says: “regularizing the bias leads to under-fitting”, <strong>not overfitting</strong>.</p>