Requires_grad doesn't propagate to the parameters of the module

username: nkundiushuti
cooked: <p>hi!</p>
<p>I am trying to have different layers of the network trained during specific epochs. So I set require_grad for those layers to True and then False. I noticed that :<br>
self.model.conv2[0].requires_grad is True<br>
self.model.conv2[0].weight.requires_grad is False<br>
although I set self.model.conv2[0].requires_grad = False<br>
where self.model.conv2[0] is Conv2d(12, 25, kernel_size=(3, 3), stride=(1, 1))</p>
<p>shouldn’t require_grad propagate to the parameters of the layers (weights and biases)?</p>
<p>thank you!</p>

username: albanD
cooked: <p><code>nn.Module</code>s don’t have a <code>requires_grad</code> field. So creating one and setting it to <code>True</code> won’t change anything.<br>
You can use <code>zero_grad()</code> on it to set all gradients to zero. Or use <code>parameters()</code> to get an iterator over the parameters of your module so that you can set <code>requires_grad</code> to <code>True</code> for each of them.</p>

username: nkundiushuti
cooked: <p>thank you for the quick reply. that’s interesting.<br>
maybe the require_grad should be removed from the nn.Modules since it’s a bit misleading.</p>

username: albanD
cooked: <p><code>nn.Module</code>s do not have a <code>requires_grad</code> field… It think you are creating it yourself.</p>

username: nkundiushuti
cooked: <p>yes! you’re right.<br>
thank you</p>

username: f3ba
cooked: <p>Sorry to revive an old topic. But I have kind of the same question:</p>
<p>Is there an easy way to set <code>requires_grad</code> simultaneously to all the parameters associated with a module (recursively if said module contains other modules)?</p>

username: ptrblck
cooked: <p>You could write a method, which accepts a module, checks for valid parameters (<code>weight</code> and <code>bias</code>) and manipulates the <code>requires_grad</code> attribute, similar to a <code>weight_init</code> method:</p>
<pre><code class="lang-python">def set_requires_grad(m, requires_grad):
    if hasattr(m, 'weight') and m.weight is not None:
        m.weight.requires_grad_(requires_grad)
    if hasattr(m, 'bias') and m.bias is not None:
        m.bias.requires_grad_(requires_grad)


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.module1 = nn.Sequential(
            nn.Linear(1, 1),
            nn.ReLU(),
            nn.Linear(1,1, bias=False)
        )
        self.module2 = nn.Sequential(
            nn.Linear(1, 1),
            nn.ReLU(),
            nn.Linear(1,1, bias=False)
        )
    

model = MyModel()
model.module1.apply(lambda m: set_requires_grad(m, False))
print(model.module1[0].weight.requires_grad)
&gt; False
print(model.module1[0].bias.requires_grad)
&gt; False
print(model.module2[2].weight.requires_grad)
&gt; True
</code></pre>

username: f3ba
cooked: <p>Is there a more generic way, that doesn’t assume that all parameters are called “weight” or “bias”? Thanks!</p>

username: ptrblck
cooked: <pre><code class="lang-python">def set_requires_grad(m, requires_grad):
    for param in m.parameters():
        param.requires_grad_(requires_grad)
</code></pre>
<p>should work as well.</p>

username: ptrblck
cooked: <p><code>model.parameters()</code> and <code>model.named_parameters()</code> will return all parameters recursively from all submodules as seen here:</p>
<pre><code class="lang-python">class SubSubModule(nn.Module):
    def __init__(self):
        super(SubSubModule, self).__init__()
        self.sub_sub_param = nn.Parameter(torch.randn(1))

class SubModule(nn.Module):
    def __init__(self):
        super(SubModule, self).__init__()
        self.sub_param = nn.Parameter(torch.randn(1))
        self.sub_sub_module = SubSubModule()


class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.param = nn.Parameter(torch.randn(1))
        self.sub_module = SubModule()
        
model = MyModel()
for name, param in model.named_parameters():
    print(name, param)
&gt; param Parameter containing:
tensor([1.8495], requires_grad=True)
sub_module.sub_param Parameter containing:
tensor([1.8534], requires_grad=True)
sub_module.sub_sub_module.sub_sub_param Parameter containing:
tensor([0.0247], requires_grad=True)


for param in model.parameters():
    print(param)
&gt; Parameter containing:
tensor([1.8495], requires_grad=True)
Parameter containing:
tensor([1.8534], requires_grad=True)
Parameter containing:
tensor([0.0247], requires_grad=True)
</code></pre>
<p>Could you clarify, when this doesn’t work?</p>

username: drevicko
cooked: <p>I’m not able to replicate it now… Not sure what that’s about? Perhaps I was confused… Deleting my previous post.</p>

username: banton
cooked: <p>The problem I have with using m.parameters() or m.named_parameters() is that it does not return the affine parameters of the batch norm layers if I am not mistaken.<br>
Could you explain how to set our batch norm layers to requires_grad=False?<br>
Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="banton" data-post="14" data-topic="9979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/eada6e/48.png" class="avatar"> banton:</div>
<blockquote>
<p>is that it does not return the affine parameters of the batch norm layers if I am not mistaken.</p>
</blockquote>
</aside>
<p>No, the trainable affine parameters from batchnorm layers will be returned as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">bn = nn.BatchNorm2d(3, affine=False)
print(dict(bn.named_parameters()))
# {}

bn = nn.BatchNorm2d(3, affine=True)
print(dict(bn.named_parameters()))
# {'weight': Parameter containing:
# tensor([1., 1., 1.], requires_grad=True), 'bias': Parameter containing:
# tensor([0., 0., 0.], requires_grad=True)}
</code></pre>
<p>However, <code>named_parameters</code> will not return the running stats as these are buffers:</p>
<pre data-code-wrap="python"><code class="lang-python">bn = nn.BatchNorm2d(3, affine=True)
print(dict(bn.named_buffers()))
# {'running_mean': tensor([0., 0., 0.]), 'running_var': tensor([1., 1., 1.]), 'num_batches_tracked': tensor(0)}
</code></pre>
<aside class="quote no-group" data-username="banton" data-post="14" data-topic="9979">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/eada6e/48.png" class="avatar"> banton:</div>
<blockquote>
<p>Could you explain how to set our batch norm layers to requires_grad=False?</p>
</blockquote>
</aside>
<p>Set the <code>.requires_grad</code> attribute of the trainable, affine parameters to <code>False</code>.</p>