Add a layer in __init__ function but not in forward function problem

Erfan_Dejband: <p>Hi,<br>
I define a layer in <strong>init</strong> function but I don’t use it in forward function. why it still cause the number of trainable parameter change?<br>
and what it mean? (dose it mean this layer is not in forward path but in backpropagation path exist?)<br>
and what if I want use a layer conditionally (i.e. each 20 epoch, or if error is less than … )<br>
i put the code here( its part of pytorch tutorial  <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html</a>)</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)  # 5*5 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.test_layer = nn.Linear(84, 84)   #add or remove this layer which not exict in forward pass 
        self.fc3 = nn.Linear(84, 10)
        
        

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square, you can specify with a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = torch.flatten(x, 1) # flatten all dimensions except the batch dimension
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


model = Net()
print(model)
print ('model trainable parameters',sum(p.numel() for p in model.parameters() if p.requires_grad))
print ('model non-trainable parameters',sum(p.numel() for p in model.parameters() if p.requires_grad==False))
</code></pre>
<p>you can see with  <code>self.test_layer = nn.Linear(84, 84)</code> output is</p>
<blockquote>
<p>Net(<br>
(conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))<br>
(conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))<br>
(fc1): Linear(in_features=400, out_features=120, bias=True)<br>
(fc2): Linear(in_features=120, out_features=84, bias=True)<br>
(test_layer): Linear(in_features=84, out_features=84, bias=True)<br>
(fc3): Linear(in_features=84, out_features=10, bias=True)<br>
)<br>
model trainable parameters 68846<br>
model non-trainable parameters 0</p>
</blockquote>
<p>while without it</p>
<blockquote>
<p>Net(<br>
(conv1): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))<br>
(conv2): Conv2d(6, 16, kernel_size=(5, 5), stride=(1, 1))<br>
(fc1): Linear(in_features=400, out_features=120, bias=True)<br>
(fc2): Linear(in_features=120, out_features=84, bias=True)<br>
(fc3): Linear(in_features=84, out_features=10, bias=True)<br>
)<br>
model trainable parameters 61706<br>
model non-trainable parameters 0</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="Erfan_Dejband" data-post="1" data-topic="177297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erfan_dejband/48/57486_2.png" class="avatar"> Erfan_Dejband:</div>
<blockquote>
<p>why it still cause the number of trainable parameter change?</p>
</blockquote>
</aside>
<p><code>model.parameters()</code> will return all properly registered parameters of the model and all of its children regardless if these parameters/modules are used in the <code>forward</code> or not.</p>
<aside class="quote no-group" data-username="Erfan_Dejband" data-post="1" data-topic="177297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erfan_dejband/48/57486_2.png" class="avatar"> Erfan_Dejband:</div>
<blockquote>
<p>and what it mean? (dose it mean this layer is not in forward path but in backpropagation path exist?)</p>
</blockquote>
</aside>
<p>No, the computation graph will be created during the forward pass, not during the initialization.</p>
<aside class="quote no-group" data-username="Erfan_Dejband" data-post="1" data-topic="177297">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/erfan_dejband/48/57486_2.png" class="avatar"> Erfan_Dejband:</div>
<blockquote>
<p>and what if I want use a layer conditionally (i.e. each 20 epoch, or if error is less than … )</p>
</blockquote>
</aside>
<p>You can use conditions in the <code>forward</code> method and use the layer as you want.</p>

Erfan_Dejband: <p>thank you for your explanation my conflict solved</p>