Weight Decay Implementation

prlz77: <p>Hi,</p>
<p>I have reproduced <a href="https://github.com/prlz77/ResNeXt.pytorch" rel="nofollow noopener">ResNeXt with pytorch on CIFAR</a> and results are always slightly below the <a href="https://github.com/facebookresearch/ResNeXt" rel="nofollow noopener">original torch implementation</a> .</p>
<p>I have been in contact with the author of the paper and he thinks it might be due to differences with the original ResNet implementation, such as applying weight decay to the biases.</p>
<p>I have gone through the pytorch implementation of the weight decay and I have now some doubts I would like to ask:</p>
<ul>
<li>Is weight decay affecting the batchnorm biases by default?</li>
<li>Taking a look into the current implementation in <a href="http://sgd.py" rel="nofollow noopener">sgd.py</a>, could anyone explain what is line 85 doing?</li>
</ul>
<p><code>d_p.add_(weight_decay, p.data)</code></p>
<p>Thank you in advance!</p>

ncullen93: <p>For 1. yes it looks like weight decay is being applied on the biases as well as the weights.</p>
<p>For 2. It is multiplying <code>p.data</code> by <code>weight_decay</code> then adding it to <code>d_p</code> (the gradients).</p>
<p>Example:</p>
<pre><code class="lang-python"># 3x3 matrix of ones
xx = nn.Parameter(torch.from_numpy(np.ones((3,3))))
print(xx)
#Parameter containing:
# 1  1  1
# 1  1  1
# 1  1  1
#[torch.DoubleTensor of size 3x3]

# sum the values just so we can backprop a gradient
y = torch.sum(xx**2)
y.backward()
# gradient of size 3x3
d_p = xx.grad.data
print(d_p)
# 2  2  2
# 2  2  2
# 2  2  2
#[torch.DoubleTensor of size 3x3]

# do this operation
weight_decay = 3.0
d_p.add_(weight_decay, xx.data)
print(d_p)
# 5  5  5
# 5  5  5
# 5  5  5
#[torch.DoubleTensor of size 3x3]
</code></pre>
<p>You see the matrix was ones, the gradient matrix was twos, then we multiplied the data by weight decay (3.0), then added it to the gradient to get fives.</p>
<p>You can see on <a href="http://stats.stackexchange.com/questions/29130/difference-between-neural-net-weight-decay-and-learning-rate" rel="nofollow noopener">stackexchange</a> for a further explanation, where you’re essentially doing the following update: wi←wi−η∂E∂wi−ηλwi. The weight decay is that final term</p>

prlz77: <p>Thank you, really nice explanation <img src="//discuss.pytorch.org/images/emoji/apple/smile.png?v=5" title=":smile:" class="emoji" alt=":smile:"></p>

smth: <p>if you want to filter out weight decay only for biases (i.e. have weight decay for weights, but no weight decay for biases), then you can use the per-parameter optimization options, like described here: <a href="http://pytorch.org/docs/optim.html#per-parameter-options">http://pytorch.org/docs/optim.html#per-parameter-options</a></p>
<p>To get the biases out of the model, you can use <a href="http://pytorch.org/docs/nn.html#torch.nn.Module.named_parameters">model.named_parameters()</a></p>
<p>Here is an example that might help:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L461-L487" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L461-L487" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/master/test/test_nn.py#L461-L487</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="461" style="counter-reset: li-counter 460 ;">
<li>    module.zero_grad()</li>
<li>    self.assertEqual(module.weight.grad.data, module.weight.data.clone().zero_())</li>
<li>    self.assertEqual(module.bias.grad.data, module.bias.data.clone().zero_())</li>
<li>
</li>
<li>def test_no_grad(self):</li>
<li>    module = nn.Conv2d(2, 5, kernel_size=3, padding=1)</li>
<li>    input = torch.randn(1, 2, 10, 10)</li>
<li>    x = Variable(input)</li>
<li>    y = Variable(input.clone())</li>
<li>
</li>
<li>    output = module(x)</li>
<li>    self.assertTrue(output.requires_grad)</li>
<li>    output.backward(torch.ones(1, 5, 10, 10))</li>
<li>
</li>
<li>    with torch.no_grad():</li>
<li>        output2 = module(y)</li>
<li>        self.assertFalse(output2.requires_grad)</li>
<li>        self.assertRaises(RuntimeError, lambda: output2.backward(torch.ones(1, 5, 10, 10)))</li>
<li>
</li>
<li>def _test_dropout(self, cls, input):</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/master/test/test_nn.py#L461-L487" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


prlz77: <p>I want the opposite (decay everywhere), so the problem is solved <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> Actually your answer helps me in another project.</p>
<p>Thank you!</p>

himanshurobo: <p>Well explained… !!!</p>