Weight decay - Does it change layers with no gradients?

TomerF: <p>I optimize some parts of a model, while other parts are set fixed with requires_grad_(False).<br>
Doesn’t weight decay change the weights of the layers, whose requires_grad is set to False, even if inputs are forwarded through these layers?<br>
What about the case of a layer, whose requires_grad is set to True, but inputs are not forwarded through it during this optimization step?<br>
Thanks</p>

pascal_notsawo: <p>I don’t think so.<br>
When I look <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/adam.py#L107" rel="noopener nofollow ugc">here</a>, I notice that they just pass the paremeters that require the grandient to <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/_functional.py#L54" rel="noopener nofollow ugc">F.adam</a>, who in turn uses it for its <a href="https://github.com/pytorch/pytorch/blob/master/torch/optim/_functional.py#L83" rel="noopener nofollow ugc">computations</a>.</p>

shivammehta007: <p>Yeah as stated by <a class="mention" href="/u/pascal_notsawo">@pascal_notsawo</a> , you pass <code>model.parameters()</code> into the optimizer it automatically removes the tensors with <code>requires_grad = False</code>. So the weight decay will not effect them as they are no longer part of the optimization process.<br>
For the second case, if the inputs are not forwarded then the weight decay will still effect them, this you can check by plotting them on tensorboard or any other visualization</p>

TomerF: <p>Thank you.<br>
I guess that the second case does not occur when the optimization step is done after zero_grad(set_to_none=True), does it?</p>

AlphaBetaGamma96: <p>The <code>set_to_none=True</code> is just a way to speed-up the zeroing of your gradient attributes. So, it shouldn’t change anything to do with weight decay.</p>
<p>It’s quicker to set all your param.grad attributes to <code>None</code> rather than <code>torch.zeros_like</code>!</p>

TomerF: <p>Thank you.<br>
How does the weight decay still affect parameters of a layer, which no inputs are forwarded through it as <a class="mention" href="/u/shivammehta007">@shivammehta007</a> said?<br>
It seems that the gradients of this layer will remain None after the backward process, so these parameters will no be appended to <code>params_with_grad</code> in the <a href="https://github.com/pytorch/pytorch/blob/2e6221a232d39917e2736b248c53fa85dfb8986e/torch/optim/adam.py#L79" rel="noopener nofollow ugc">step</a> function. Then, it seems that these parameters will not be iterated over in <a href="https://github.com/pytorch/pytorch/blob/2e6221a232d39917e2736b248c53fa85dfb8986e/torch/optim/_functional.py#L72" rel="noopener nofollow ugc">F.adam</a> and <a href="https://github.com/pytorch/pytorch/blob/2e6221a232d39917e2736b248c53fa85dfb8986e/torch/optim/_functional.py#L82" rel="noopener nofollow ugc">weight decay</a> will not be applied on these parameters as <a class="mention" href="/u/pascal_notsawo">@pascal_notsawo</a> said.<br>
By the way, according to <a href="https://github.com/pytorch/pytorch/blob/2e6221a232d39917e2736b248c53fa85dfb8986e/torch/optim/optimizer.py#L208" rel="noopener nofollow ugc">zero_grad</a>, it indeed does not zero parameters, which are already set to None, so <code>set_to_none</code> indeed does not change the situation.</p>

AlphaBetaGamma96: <p>I’m not sure 100% on the phasing of the statement but I assume you mean ‘forwarding’ as in calling a particular Module/Layer within your model? If this is true, then a particular layer within your network that doesn’t take any input data and just applies arithmetic to intermediate layers might not have a <code>grad_fn</code> (depending on the arithmetic at play) in the case of no <code>grad_fn</code> it’d be skipped by the optimizer step method (as you said)</p>