How does SGD weight_decay work?

username: BrightLamp
cooked: <p>Hello,<br>
i write a toy code to check SGD weight_decay.<br>
but it seems to have no effect to the gradient update.<br>
am i misunderstand the meaning of weight_decay?<br>
thank you very much.</p>
<p>PyTorch 1.0</p>
<hr>
<pre><code class="lang-auto">import torch
import numpy as np

np.random.seed(123)
np.set_printoptions(8, suppress=True)

x_numpy = np.random.random((3, 4)).astype(np.double)
w_numpy = np.random.random((4, 5)).astype(np.double)
x_torch = torch.tensor(x_numpy, requires_grad=True)
w_torch = torch.tensor(w_numpy, requires_grad=True)

lr = 0.1
sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=0.9)

y_torch = torch.matmul(x_torch, w_torch)
loss = y_torch.sum()

print("w_torch before SGD")
print(w_torch.data.numpy())

sgd.zero_grad()
loss.backward()
sgd.step()

w_grad = w_torch.grad.data.numpy()

print("w_torch after SGD")
print(w_torch.data.numpy())

print("check_weight_decay")
print(w_numpy - lr * w_grad)
</code></pre>
<pre><code class="lang-auto">"""
code output :

w_torch before SGD
[[ 0.43857224  0.0596779   0.39804426  0.73799541  0.18249173]
 [ 0.17545176  0.53155137  0.53182759  0.63440096  0.84943179]
 [ 0.72445532  0.61102351  0.72244338  0.32295891  0.36178866]
 [ 0.22826323  0.29371405  0.63097612  0.09210494  0.43370117]]
w_torch after SGD
[[ 0.20941374 -0.13538012  0.17253327  0.48188881 -0.02361953]
 [ 0.04952477  0.37357542  0.37382677  0.46716854  0.6628466 ]
 [ 0.50417498  0.40095203  0.50234411  0.13881324  0.17414831]
 [ 0.01120012  0.07076036  0.37766885 -0.11270393  0.19814865]]
check_weight_decay
[[ 0.20941374 -0.13538012  0.17253327  0.48188881 -0.02361953]
 [ 0.04952477  0.37357542  0.37382677  0.46716854  0.6628466 ]
 [ 0.50417498  0.40095203  0.50234411  0.13881324  0.17414831]
 [ 0.01120012  0.07076036  0.37766885 -0.11270393  0.19814865]]
"""
</code></pre>

username: rasbt
cooked: <p>The <code>weight_decay</code> parameter adds a L2 penalty to the cost which can effectively lead to to smaller model weights. It seems to work in my case:</p>
<pre><code class="lang-python">import torch
import numpy as np

np.random.seed(123)
np.set_printoptions(8, suppress=True)

x_numpy = np.random.random((3, 4)).astype(np.double)
w_numpy = np.random.random((4, 5)).astype(np.double)
x_torch = torch.tensor(x_numpy, requires_grad=True)
w_torch = torch.tensor(w_numpy, requires_grad=True)

#######################################################

print('Original weights', w_torch)

lr = 0.1
sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=0)

y_torch = torch.matmul(x_torch, w_torch)
loss = y_torch.sum()

sgd.zero_grad()
loss.backward()
sgd.step()

w_grad = w_torch.grad.data.numpy()
print('0 weight decay', w_torch)


#######################################################

w_torch = torch.tensor(w_numpy, requires_grad=True)

print('Reset Original weights', w_torch)

sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=1)

y_torch = torch.matmul(x_torch, w_torch)
loss = y_torch.sum()

sgd.zero_grad()
loss.backward()
sgd.step()

w_grad = w_torch.grad.data.numpy()
print('1 weight decay', w_torch)
</code></pre>
<p>This returns</p>
<pre><code class="lang-auto">Original weights tensor([[0.4386, 0.0597, 0.3980, 0.7380, 0.1825],
        [0.1755, 0.5316, 0.5318, 0.6344, 0.8494],
        [0.7245, 0.6110, 0.7224, 0.3230, 0.3618],
        [0.2283, 0.2937, 0.6310, 0.0921, 0.4337]],
       dtype=torch.float64, requires_grad=True)
0 weight decay tensor([[ 0.2489, -0.1300,  0.2084,  0.5483, -0.0072],
        [ 0.0653,  0.4214,  0.4217,  0.5243,  0.7393],
        [ 0.5694,  0.4559,  0.5674,  0.1679,  0.2067],
        [ 0.0317,  0.0972,  0.4345, -0.1044,  0.2372]],
       dtype=torch.float64, requires_grad=True)
Reset Original weights tensor([[0.4386, 0.0597, 0.3980, 0.7380, 0.1825],
        [0.1755, 0.5316, 0.5318, 0.6344, 0.8494],
        [0.7245, 0.6110, 0.7224, 0.3230, 0.3618],
        [0.2283, 0.2937, 0.6310, 0.0921, 0.4337]],
       dtype=torch.float64, requires_grad=True)
1 weight decay tensor([[ 0.2050, -0.1360,  0.1686,  0.4745, -0.0254],
        [ 0.0478,  0.3683,  0.3685,  0.4608,  0.6544],
        [ 0.4969,  0.3948,  0.4951,  0.1356,  0.1705],
        [ 0.0089,  0.0678,  0.3714, -0.1136,  0.1938]],
       dtype=torch.float64, requires_grad=True)
</code></pre>
<p>As you can see, the weights are smaller when I use <code>weight_decay=1</code> compared to <code>weight_decay=0</code></p>

username: BrightLamp
cooked: <p>Hello, rasbt</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/39a9f417cb836820174d71e9a9318d5ccc6191f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/39a9f417cb836820174d71e9a9318d5ccc6191f3" title="reply"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/39a9f417cb836820174d71e9a9318d5ccc6191f3_2_690x367.png" alt="reply" data-base62-sha1="8e7l2YEVryLZ3fiKpDGkJ0Rvrgf" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/39a9f417cb836820174d71e9a9318d5ccc6191f3_2_690x367.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/39a9f417cb836820174d71e9a9318d5ccc6191f3_2_1035x550.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/39a9f417cb836820174d71e9a9318d5ccc6191f3.png 2x" data-dominant-color="F9F9F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">reply</span><span class="informations">1379×734 53.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>here is another check code :</p>
<pre><code class="lang-auto">import torch
import numpy as np

np.random.seed(123)
np.set_printoptions(8, suppress=True)

x_numpy = np.random.random((3, 4)).astype(np.double)
x_torch = torch.tensor(x_numpy, requires_grad=True)
x_torch2 = torch.tensor(x_numpy, requires_grad=True)

w_numpy = np.random.random((4, 5)).astype(np.double)
w_torch = torch.tensor(w_numpy, requires_grad=True)
w_torch2 = torch.tensor(w_numpy, requires_grad=True)

lr = 0.1
weight_decay = 0.9
sgd = torch.optim.SGD([w_torch], lr=lr, weight_decay=0)
sgd2 = torch.optim.SGD([w_torch2], lr=lr, weight_decay=weight_decay)

y_torch = torch.matmul(x_torch, w_torch)
y_torch2 = torch.matmul(x_torch2, w_torch2)

loss = y_torch.sum()
loss2 = y_torch2.sum()

sgd.zero_grad()
sgd2.zero_grad()

loss.backward()
loss2.backward()

sgd.step()
sgd2.step()

w_grad = w_torch.grad.data.numpy()
w_grad2 = w_torch2.grad.data.numpy()

print("check_grad")
print(w_grad + weight_decay * w_numpy)
print(w_grad2)

"""
check_grad
[[ 2.29158508  1.95058016  2.25510989  2.56106592  2.06111261]
 [ 1.25926989  1.57975955  1.58000814  1.67232418  1.86585193]
 [ 2.20280346  2.10071483  2.20099271  1.84145669  1.87640346]
 [ 2.17063112  2.22953686  2.53307273  2.04808866  2.35552527]]
[[ 2.29158508  1.95058016  2.25510989  2.56106592  2.06111261]
 [ 1.25926989  1.57975955  1.58000814  1.67232418  1.86585193]
 [ 2.20280346  2.10071483  2.20099271  1.84145669  1.87640346]
 [ 2.17063112  2.22953686  2.53307273  2.04808866  2.35552527]]
 """
</code></pre>

username: rasbt
cooked: <p>The part that I circled doesn’t seem right to me:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70.png" data-download-href="https://discuss.pytorch.org/uploads/default/e1761b0ce81dd7cff5679379168d2e5f1b6bec70" title="39a9f417cb836820174d71e9a9318d5ccc6191f3"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70_2_690x367.png" alt="39a9f417cb836820174d71e9a9318d5ccc6191f3" data-base62-sha1="wawtcNPx4b5rUeOpv85Y3dXqI36" width="690" height="367" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70_2_690x367.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70_2_1035x550.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/e/e1761b0ce81dd7cff5679379168d2e5f1b6bec70.png 2x" data-dominant-color="F9F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">39a9f417cb836820174d71e9a9318d5ccc6191f3</span><span class="informations">1379×734 65.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>In L2 regularization, you modify the cost as follows</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69.png" data-download-href="https://discuss.pytorch.org/uploads/default/44ad32d9502e88846fc36bb31c06c7df8ef97c69" title="text_Regularized"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69_2_690x99.png" alt="text_Regularized" data-base62-sha1="9NxxKrBerJIAwSJCrEi7yN1vtgt" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69_2_690x99.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69_2_1035x148.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/4/44ad32d9502e88846fc36bb31c06c7df8ef97c69_2_1380x198.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">text_Regularized</span><span class="informations">2262×325 21.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The weight update should be then</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57.png" data-download-href="https://discuss.pytorch.org/uploads/default/e831c3e6a017fcf2fd3ae177c71c928274772d57" title="text_updatedWeig"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57_2_690x35.png" alt="text_updatedWeig" data-base62-sha1="x85ot7YtD9GryPy5EJWiaNIJLg3" width="690" height="35" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57_2_690x35.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57_2_1035x52.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/e831c3e6a017fcf2fd3ae177c71c928274772d57_2_1380x70.png 2x" data-dominant-color="DFDFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">text_updatedWeig</span><span class="informations">2954×154 25.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The way PyTorch applied the weight decay seems correct to me (you can drop the factor 2)</p>

username: InnovArul
cooked: <p>If you look closely,  the formula that you circled is just a rearrangement of the usual SGD weight decay formula I guess.</p>

username: rasbt
cooked: <p>Ooops, you are right, they are exactly the same.</p>

username: Dhorka
cooked: <p>I have a doubt here. In pytorch the weigh decay,  is it only applied to the weighs or to all the parameters that requires gradient? I mean for instance if I use this piece of code:</p>
<p><code>optimizer = torch.optim.Adam(model.parameters(), lr=args.lr, betas=args.betas, weight_decay=args.wd)</code></p>
<p>Will be the weight decay applied to all the parameters of the model including bias and batchnorm parameters?</p>
<p>Thanks</p>

username: bintonto
cooked: <pre><code class="lang-auto">   def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            weight_decay = group['weight_decay']
            momentum = group['momentum']
            dampening = group['dampening']
            nesterov = group['nesterov']

            for p in group['params']:
                if p.grad is None:
                    continue
                d_p = p.grad.data
                if weight_decay != 0:
                    d_p.add_(weight_decay, p.data)
                if momentum != 0:
                    param_state = self.state[p]
                    if 'momentum_buffer' not in param_state:
                        buf = param_state['momentum_buffer'] = torch.clone(d_p).detach()
                    else:
                        buf = param_state['momentum_buffer']
                        buf.mul_(momentum).add_(1 - dampening, d_p)
                    if nesterov:
                        d_p = d_p.add(momentum, buf)
                    else:
                        d_p = buf

                p.data.add_(-group['lr'], d_p)

        return loss
</code></pre>
<pre><code class="lang-python"> d_p = p.grad.data
 if weight_decay != 0:
     d_p.add_(weight_decay, p.data)
</code></pre>
<p>so… weight_decay * w is already included in w_grad</p>

username: Dhorka
cooked: <p>Thanks for your reply but it is not answering my question <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"> . I will try to explain better. My concern is to know if the weigh_decay functionality is able to distinguish between weighs, bias and learning parameters of batchnorm. Because, Normally weight decay is only applied to the weights and not to the bias and batchnorm parameters (do not make sense to apply a weight decay to the batchnorm parameters). For this reason I am asking if the weigh decay is able to distinguish between this kind of parameters. My feelings after see the code is that weight_decay functionality is not able to distinguis between these parameters, but I would like to have the confirmation.</p>

username: Konstantin_Burlachen
cooked: <p>So let me summarize.<br>
From experiments and from <a href="https://github.com/pytorch/pytorch/blob/62809dc0628aee628120e40c12ef39b9d905a51d/torch/optim/_functional.py#L176" rel="noopener nofollow ugc">F.sgd</a> code we may observe that weight_decay is in fact add $\ambda w_i$</p>
<p>So:</p>
<ol>
<li>
<p>Original modified code<br>
<strong>"print(w_numpy - lr * (w_grad+0.9*w_numpy))"</strong><br>
produce the same result as<br>
<strong>"print(w_torch.data.numpy())"</strong><br>
Error in the question was in ignoring the weight_decay parameter.</p>
</li>
<li>
<p>"<em>The <code>weight_decay</code> parameter adds an L2 penalty to the cost".</em> Such a response is too vague, such a thing as the L2 penalty does not exist in mathematics.</p>
</li>
</ol>
<p>Please don’t blame me, but you use several words from math:</p>
<ul>
<li>L2 is a name from math that stands for the norm. And it’s not true that you use the L2 norm at all. What you use is the L2 norm square of all vectors of parameters. It’s not L2 norm.</li>
<li>You use the word “Penalty” penalty can be as a constraint or extra term in objective. If it is an extra term then different technics for scalarization are existed.</li>
</ul>
<p>Next to such responses like “L2 penalty” lead to real problems.</p>
<ul>
<li>First of all - you multiply it by “\lambda/2”, not by “\lambda”<br>
It’s why such responses add “L2” penalty is too vague, and it only brings too much confusion. It’s better to say everything exactly.</li>
</ul>
<ol start="3">
<li>The formula with “2 \lambda” is incorrect as well. And bring only confusion.</li>
</ol>
<p>So what is implemented in PyTorch is weight_decay corresponds to implicit additive term in the objective function with form $(\lambda/2) |parameters|_2^2$.</p>

username: dhruvbird
cooked: <aside class="quote no-group" data-username="Dhorka" data-post="9" data-topic="33105">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/b2d939/48.png" class="avatar"> Dhorka:</div>
<blockquote>
<p>My concern is to know if the weigh_decay functionality is able to distinguish between weighs, bias and learning parameters of batchnorm.</p>
</blockquote>
</aside>
<p>Weight decaying is applied to all the learnable/trainable parameters of the model. If you want to explicitly exclude some parameters from being weight decayed, you will need to do that manually. See the code for <code>configure_optimizers</code> at <a href="https://github.com/karpathy/nanoGPT/blob/master/model.py#L270" rel="noopener nofollow ugc">https://github.com/karpathy/nanoGPT/blob/master/model.py#L270</a> for an example.</p>

username: Miro_Mlynarik
cooked: <p>Batchnorm parameters are not learnable parameters. They are registered as buffer.</p>

username: ptrblck
cooked: <p>That’s not correct since BatchNorm layers use trainable affine parameters and buffers collecting the running stats as seen here:</p>
<pre><code class="lang-python">bn = nn.BatchNorm2d(3)
print(dict(bn.named_parameters()))
# {'weight': Parameter containing:
# tensor([1., 1., 1.], requires_grad=True), 'bias': Parameter containing:
# tensor([0., 0., 0.], requires_grad=True)}
print(dict(bn.named_buffers()))
# {'running_mean': tensor([0., 0., 0.]), 'running_var': tensor([1., 1., 1.]), 'num_batches_tracked': tensor(0)}
</code></pre>