Update some weight with torch.no_grad and type(weight.grad) is Nonetype

username: Mandy
cooked: <p>Hi all,</p>
<p>I’m trying to freeze some weights(value = 0) from updating during backward, but I failed for several times <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>The method one I tried is using trying to zero out the gradient with below codes:</p>
<pre><code class="lang-auto">    def hook(self, model, inputs):
        with torch.no_grad():  
            print('*********************************forward***********************')
            print(model)
            print(model.weight.data )
            model.weight.data = model.weight.data * self.sparse_mask[self.type[model]]
         

    def back_hook(self, model,grad_input,grad_output):
        print(model)
        print(model.weight)
        print('im back',type(model.weight.grad))
       
        model.weight.grad = model.weight.grad * self.sparse_mask[self.type[model]]

    def register_hook(self,module):
        self.handle = module.register_forward_pre_hook(self.hook)
        self.backhandle = module.register_backward_hook(self.back_hook)

    for m in model.modules():
         if isinstance(m, nn.Conv2d):
               sp.register_hook(m)

    pred = model(data)
    loss = criterion(pred, target)
    losses.update(loss.data.cpu().numpy())
    optimizer.zero_grad()
    loss.backward()
</code></pre>
<p>But I got this error</p>
<blockquote>
<p>TypeError: mul(): argument ‘other’ (position 1) must be Tensor, not NoneType</p>
</blockquote>
<p>And my output of <code>back_hook()</code> is</p>
<blockquote>
<p>im back  <strong>NoneType</strong> which means my <strong>weight.grad is None</strong></p>
</blockquote>
<p>I don’t have any clue why this happened!</p>
<p>The method 2 I tried is</p>
<pre><code class="lang-auto">    def back_hook(self, model,grad_input,grad_output):      
        with torch.no_grad():
               model.weight.data = model.weight.data * self.sparse_mask[self.type[model]]
       
</code></pre>
<p>And keeps others as the same.</p>
<p>There is no bug and error this time, however, the output is not correct.<br>
This is the backward of iter2<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b9239b169f2e8aa4b3f181b669446c6a3fd17858.png" alt="ss2" data-base62-sha1="qpOCghawNp0REvVP7Rry8HBd9fa" width="341" height="218"><br>
These two are the forward of iter3<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/8/841741fe23e74d9fc0e0c0060428260ba461977e.png" alt="ss3" data-base62-sha1="iQwR3cAaWVpNO7zVgVcnfcPmw7c" width="340" height="196"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d040fc38c5eeafbc7cdc44f14f50fb2661423695.png" alt="ss4" data-base62-sha1="tIixJu3DN21td4vITUEh44mlS05" width="321" height="190"><br>
This is the backward of iter3<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0b7f5d5cc7f241ff36941d1fbf99cc47a5d4bc31.png" alt="ss9" data-base62-sha1="1DI82S8Ek6b9PrSxplNEhBDAGU9" width="330" height="201"></p>
<p>I can’t figure out <strong>Why in the first pic of forward part the others weight are not updating except the left-top one, from 0 to -1.1226e-3?</strong></p>
<p>But what I want is freeze that weight which value = 0 and update the others non-zero weight.</p>
<p>Anyone would like to help me solve that problem?</p>
<p>Really appreciate!</p>