Do unused layers that are not included in the forward pass affect training/evaluating?

username: Fawaz_Sammani
cooked: <p>Hi, I would like to ask. If I define some layers in the <strong>init</strong> funtion of a class, but in the forward function, I do not use these layers (not call them), does that affect anything?</p>
<p>For example, take the code below:</p>
<pre><code class="lang-auto">class Net(nn.Module):

    def __init__(self, input_size, hidden_size, output_size):
        super(Net, self).__init__()
        self.input_layer = nn.Linear(input_size, hidden_size)-
        self.hidden_layer_1 = nn.Linear(hidden_size, hidden_size)
        self.hidden_layer_2 = nn.Linear(hidden_size, hidden_size)     # not used/not called in forward pass
        self.output_layer = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        out = F.relu(self.input_layer(x))
        out = F.relu(self.hidden_layer_1(out))
        out = torch.sigmoid(self.output_layer(out))
        return out
</code></pre>

username: alex.veuthey
cooked: <p>The best way to evaluate that is to try it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> But here are my thoughts:</p>
<p>Since it’s been instantiated, and probably sent to GPU, it will consume some memory, as the parameters are created and must be kept somewhere. However, gradients won’t be computed, as it’s not present in the forward method. Also, if you send the whole <code>model.parameters()</code> to the optimizer, it might consume a bit more memory again? Not sure about this one.</p>
<p>In such cases, I would try to pass an additional parameter to the model, which defines if the layers have to be created or not, to save a bit of memory. The parameter won’t be kept if it’s just for initialization.</p>

username: Eta_C
cooked: <aside class="quote no-group" data-username="alex.veuthey" data-post="2" data-topic="63705">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex.veuthey/48/12176_2.png" class="avatar"> alex.veuthey:</div>
<blockquote>
<p>The best way to evaluate that is to try it</p>
</blockquote>
</aside>
<p>In addition, I suggest</p>
<pre data-code-wrap="python"><code class="lang-python">class Net(nn.Module):
    def __init__(self, use_linear):
        super().__init__()
        self.use_linear = use_linear
        self.conv_layer = nn.Conv2d(3, 3, 1)
        if self.use_linear:
            self.linear_layer = nn.Linear(12, 8)
    def forward(self, x):
        x = self.conv_layer(x)
        return self.linear_layer if self.use_linear else x
</code></pre>

username: Fawaz_Sammani
cooked: <p>Hi <a class="mention" href="/u/alex.veuthey">@alex.veuthey</a> and thanks for your answer.<br>
I did already. Even though I had to repeat training from the beginning, but I wanted to give it a try. Comparing the results from the first epoch w and w/o these extra parameters, I find that it does not affect. However, if I save the optimizer using torch.save and then load it back again using torch.load, there is an error. Most probably because the optimizer has the parameters saved in its dictionary but isn’t updating them. Not sure about this.</p>

username: mohit_kaushik
cooked: <p>Hi, I recently went through an error where I created one nn.Linear layer but didn’t use it in forward.<br>
It turns out that net.parameters() generates parameters for this layer also. So, when I tried to update using <code>for p in model.parameters(): p -= p.grad * 0.001</code> this gave me error <code>TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'</code>. I resolved this by commenting the unused layer in init.</p>
<p>Since my loss was not depending upon unused layer gradients were none.</p>
<p>This is for pytorch_1.16_cpu</p>