Why does Pytorch updates weights of a layer if it does not contribute to output?

mohit117: <p>Hello,<br>
I have a CNN model with a layer which though takes input from its previous layer and computes an output, this output is not used at all. Hence I expect that back propagation should not affects its weight.</p>
<pre><code class="lang-auto">
... My Model init() part

   self.alpha = alpha_model

def forward(self, x, h, hist):

   alpha = self.alpha(hist)

   #x = torch.pow(x,alpha)
   #h = torch.pow(h,alpha)
....
</code></pre>
<p>As you can see <code>alpha_model</code> which is some other CNN which this CNN calls. <code>alpha_model</code> is doing some computation but I am no longer using its outputs. See the commented section. That is the only place where <code>alpha_model</code> had some usage.</p>
<p>Why then if print the weights of last layer of <code>alpha_model</code>, the weights are changing by 0.0001(which is the my learning rate)? Since it is not contributing to the output the gradients should be 0. Is this because I use Adam optimiser?</p>
<p>Here is a sample output. Here <code>fc4</code> is the last layer of <code>alpha_model</code></p>
<pre><code class="lang-auto">fc4 weights Parameter containing:
tensor([[ 0.1044, -0.0683, -0.1153, -0.0387, -0.0276,  0.1335, -0.0946,  0.0927,
          0.0041,  0.1364, -0.0192, -0.0634, -0.0361,  0.0884,  0.1091, -0.0954,
          0.1241,  0.1089, -0.0930,  0.0839,  0.0144,  0.0735,  0.0217,  0.0746,
         -0.0384, -0.0422, -0.0879,  0.0786,  0.0737, -0.0474,  0.1309, -0.0705,
         -0.0487, -0.1311, -0.0782, -0.0974,  0.0303,  0.0652,  0.0628, -0.0315,
         -0.0909, -0.0865, -0.0575, -0.1176,  0.0899, -0.0818,  0.0181, -0.1335,
          0.1153,  0.0833]], requires_grad=True)
--
fc4 grad tensor([[ 1.0000e-06, -1.0000e-06, -1.0000e-06, -1.0000e-06, -1.0000e-06,
          1.0000e-06, -1.0000e-06,  1.0000e-06,  1.0000e-06,  1.0000e-06,
         -1.0000e-06, -1.0000e-06, -1.0000e-06,  1.0000e-06,  1.0000e-06,
         -1.0000e-06,  1.0000e-06,  1.0000e-06, -1.0000e-06,  1.0000e-06,
          1.0000e-06,  1.0000e-06,  1.0000e-06,  1.0000e-06, -1.0000e-06,
         -1.0000e-06, -1.0000e-06,  1.0000e-06,  1.0000e-06, -1.0000e-06,
          1.0000e-06, -1.0000e-06, -1.0000e-06, -1.0000e-06, -1.0000e-06,
         -1.0000e-06,  1.0000e-06,  1.0000e-06,  1.0000e-06, -1.0000e-06,
         -1.0000e-06, -1.0000e-06, -1.0000e-06, -1.0000e-06,  1.0000e-06,
         -1.0000e-06,  1.0000e-06, -1.0000e-06,  1.0000e-06,  1.0000e-06]])
---------------------
fc4 weights Parameter containing:
tensor([[ 0.1043, -0.0682, -0.1152, -0.0386, -0.0275,  0.1334, -0.0945,  0.0926,
          0.0040,  0.1363, -0.0191, -0.0633, -0.0360,  0.0883,  0.1090, -0.0953,
          0.1240,  0.1088, -0.0929,  0.0838,  0.0143,  0.0734,  0.0216,  0.0745,
         -0.0383, -0.0421, -0.0878,  0.0785,  0.0736, -0.0474,  0.1308, -0.0704,
         -0.0486, -0.1310, -0.0781, -0.0973,  0.0302,  0.0652,  0.0627, -0.0314,
         -0.0908, -0.0864, -0.0574, -0.1175,  0.0898, -0.0817,  0.0180, -0.1334,
          0.1152,  0.0832]], requires_grad=True)
--
fc4 grad tensor([[ 2.0000e-06, -2.0000e-06, -2.0000e-06, -2.0000e-06, -2.0000e-06,
          2.0000e-06, -2.0000e-06,  2.0000e-06,  2.0000e-06,  2.0000e-06,
         -2.0000e-06, -2.0000e-06, -2.0000e-06,  2.0000e-06,  2.0000e-06,
         -2.0000e-06,  2.0000e-06,  2.0000e-06, -2.0000e-06,  2.0000e-06,
          2.0000e-06,  2.0000e-06,  2.0000e-06,  2.0000e-06, -2.0000e-06,
         -2.0000e-06, -2.0000e-06,  2.0000e-06,  2.0000e-06, -2.0000e-06,
          2.0000e-06, -2.0000e-06, -2.0000e-06, -2.0000e-06, -2.0000e-06,
         -2.0000e-06,  2.0000e-06,  2.0000e-06,  2.0000e-06, -2.0000e-06,
         -2.0000e-06, -2.0000e-06, -2.0000e-06, -2.0000e-06,  2.0000e-06,
         -2.0000e-06,  2.0000e-06, -2.0000e-06,  2.0000e-06,  2.0000e-06]])
</code></pre>

Nikronic: <aside class="quote no-group quote-modified" data-username="mohit117" data-post="1" data-topic="89768">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mohit117/48/13447_2.png" class="avatar"> mohit117:</div>
<blockquote>
<p>Hi,</p>
<p>If you are using <code>alpha</code> which is the output of <code>alpha_model</code> in <code>x</code> and <code>x</code> the only way that <code>alpha_model</code> won’t contribute is that you throw away <code>x</code> and <code>h</code> too, if not, <code>alpha_model</code> is part of the definition of <code>x</code> so gradients will be calculated.</p>
<pre><code class="lang-auto">   #x = torch.pow(x,alpha)
   #h = torch.pow(h,alpha)
</code></pre>
</blockquote>
</aside>
<p>If you want to omit a part of graph from updating, just freeze their parameters using <code>requires_grad = False</code>. Then optimizer will skip it.</p>
<p>Bests</p>

mohit117: <p>I cannot throw away x and h as they are being used in downstream tasks. Anyways I have commented out alpha,</p>
<pre><code class="lang-auto">   #x = torch.pow(x,alpha)
   #h = torch.pow(h,alpha)
</code></pre>
<p>and although it is computed it is not contributing. So should not the dervitaive of loss wrt any paramter in alpha be always 0.</p>

albanD: <p>Hi,</p>
<p>This most likely happens because the optimizer you use has weight decay. And so you still update the weights even when the gradient is 0.</p>

mohit117: <p>Thankyou for pointing out this possibility.</p>
<p>So you mean to say that, Adam optimizer which I am using by default has weight decay (L2 penalty on weights tensor )switched on which thus penalises the weights of the sub-module which does not contribute to the output.</p>
<p>But if this is the case i am simply using,<br>
<code>torch.optim.</code> <code>Adam</code> ( <em>model.parameters</em> , <em>lr=0.0001</em> )</p>
<p>And by default i think Adam has weight decay set to 0.</p>
<p>Thankyou</p>

albanD: <p>In Adam, you actually have the exponential average term from the algorithm that can have a similar effect no?</p>

mohit117: <p>Aaa. sorry but not an expert. Anyways I was just curious to know after I made this observation. Anyhow now I have removed the sub-module which I am not using. I guess that is the best solution <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>
<p>Thank you anyways for the suggestions and thoughts.</p>