ERROR IN setting a conv filter in a particular layer as 0

username: ANKUR_GUPTA1
cooked: <pre><code class="lang-auto">class my_model(nn.Module):
  def __init__(self):
    super(my_model,self).__init__()
    self.conv1 = nn.Conv2d(3,16,kernel_size=3,stride=1,padding=1)
    self.conv2 = nn.Conv2d(16,32,kernel_size=3,stride=1,padding=1)
    self.conv3 = nn.Conv2d(32,64,kernel_size=3,stride=1,padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(4*4*64,64)
    self.fc2 = nn.Linear(64,10)
  def forward(self,inp):
    ab = self.pool(F.relu(self.conv1(inp)))
    ab = self.pool(F.relu(self.conv2(ab)))
    ab = self.pool(F.relu(self.conv3(ab)))
    ab = ab.view(ab.shape[0],-1)
    ab = F.relu(self.fc1(ab))
    ab = F.relu(self.fc2(ab))
    return ab
    
test_model = my_model()
test_model.cuda()
</code></pre>
<p>I trained my model and saved it and loaded it</p>
<pre><code class="lang-auto">new_test_model = my_model()
a = torch.load("/content/cifar_net.pth")

new_test_model.load_state_dict(a)

&gt; &lt;all keys matched&gt;

</code></pre>
<p>after that I set one filter as zero</p>
<p><code>new_test_model.conv1.weight[0]=torch.zeros_like(new_test_model.conv1.weight[0].data)</code></p>
<p>and wanted to check that after training is the filter revived<br>
so I initialized my optimizer again<br>
but am getting this error</p>
<blockquote>
<p><strong>ValueError: can’t optimize a non-leaf Tensor</strong></p>
</blockquote>
<p>I am not able to understand why<br>
<strong>Please Help Me</strong></p>

username: ptrblck
cooked: <p>Wrapping the operation into a <code>torch.no_grad()</code> block and using <code>copy_</code> should work:</p>
<pre><code class="lang-python">with torch.no_grad():
    model.conv1.weight[0].copy_(torch.zeros_like(model.conv1.weight[0]))
</code></pre>
<p>Also, don’t use the <code>.data</code> attribute, as it might cause unwanted side effects.</p>

username: ANKUR_GUPTA1
cooked: <p>is there any way by which I can set the filters as zero and make them such that they are not revived.<br>
I actually want to remove the filters.</p>

username: ptrblck
cooked: <p>If you want so set some filters as zeros, you could either remove them from your model completely or alternatively multiply your filters with a mask containing 1s for the valid filters and 0s for the filters you want to zero out, and apply the same mask to the gradients of these filters.</p>

username: ANKUR_GUPTA1
cooked: <p>can i use something internally so that the gradient is not computed by itself<br>
i tried using</p>
<p><code>        new_test_model.conv1.weight[0].requires_grad = False</code></p>
<p>but got</p>
<blockquote>
<p>RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn’t require differentiation use var_no_grad = var.detach().</p>
</blockquote>
<p>Then I tried using</p>
<p><code>        new_test_model.conv1.weight[0] = new_test_model.conv1.weight[0].detach()</code></p>
<p>but then I am getting</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/2489be51f5b80be14bf6b04faf342a5a18067384.png" data-download-href="https://discuss.pytorch.org/uploads/default/2489be51f5b80be14bf6b04faf342a5a18067384" title="Screenshot (275)"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/2489be51f5b80be14bf6b04faf342a5a18067384.png" alt="Screenshot (275)" data-base62-sha1="5deiMjt2QDQHQVBOkGkpxmgzHgg" width="690" height="361" data-dominant-color="FAFAFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot (275)</span><span class="informations">1380×723 23.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>No, you can set the <code>requires_grad</code> attribute for the entire tensor, not just a specific part of it.</p>

username: ANKUR_GUPTA1
cooked: <p><strong>Sir This is my model</strong><br>
<strong>I want to fix some filters to zero and don’t want them to update when the model learns</strong><br>
<strong>so I want to make their gradients as zero as well</strong></p>
<pre><code class="lang-auto">class my_model(nn.Module):
  def __init__(self):
    super(my_model,self).__init__()
    self.conv1 = nn.Conv2d(3,16,kernel_size=3,stride=1,padding=1)
    self.conv2 = nn.Conv2d(16,32,kernel_size=3,stride=1,padding=1)
    self.conv3 = nn.Conv2d(32,64,kernel_size=3,stride=1,padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.fc1 = nn.Linear(4*4*64,64)
    self.fc2 = nn.Linear(64,10)
  def forward(self,inp):
    ab = self.pool(F.relu(self.conv1(inp)))
    ab = self.pool(F.relu(self.conv2(ab)))
    ab = self.pool(F.relu(self.conv3(ab)))
    ab = ab.view(ab.shape[0],-1)
    ab = F.relu(self.fc1(ab))
    ab = F.relu(self.fc2(ab))
    return ab
</code></pre>
<p><strong>&gt; I loaded the model from a previous checkpoint</strong></p>
<pre><code class="lang-auto">new_test_model = my_model()
a = torch.load("/content/cifar_net.pth")

new_test_model.load_state_dict(a)

new_test_model.cuda()
</code></pre>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(new_test_model.parameters(), lr=0.001)
</code></pre>
<p><strong>&gt; I executed once to check if the model was updating weights or not</strong></p>
<pre><code class="lang-auto">optimizer.zero_grad()
inputs, labels = iter(trainloader).next()
inputs, labels = inputs.cuda(), labels.cuda()
outputs = new_test_model(inputs)
loss = criterion(outputs, labels)
print("before loss backward")
loss.backward()
print("after loss backward")
optimizer.step()
print("after opt step")
</code></pre>
<p><strong>&gt; I want to make the weights of these layers as zero according to the dictionary below</strong></p>
<pre><code class="lang-auto">indices = {
    'conv1':[0,3,9,5,7,2],
    'conv2':[10,13,19,25,17,22],
    'conv3':[20,31,29,15,27,12],
}
</code></pre>
<p><strong>As you suggested I used this code to make the filters zero</strong></p>
<pre><code class="lang-auto"># code to make the filters zero
for index, item in enumerate(new_test_model.named_children()):
  with torch.no_grad():
    if(isinstance(item[1],nn.Conv2d)):
      wt_list = indices[item[0]]
      for i in wt_list:
        item[1].weight[i].copy_(torch.zeros_like(item[1].weight[i]))
        item[1].bias[i].copy_(torch.zeros_like(item[1].bias[i]))
</code></pre>
<p><strong>Then I checked the weights, they were zero, as I wanted them to be</strong></p>
<pre><code class="lang-auto"># code to check the filters if they turned to zero
for index, item in enumerate(new_test_model.named_children()):
  with torch.no_grad():
    if(isinstance(item[1],nn.Conv2d)):
      wt_list = indices[item[0]]
      for i in wt_list:
        print(item[1].weight[i])
</code></pre>
<p><strong>Then I executed the code below to make the gradients again</strong></p>
<pre><code class="lang-auto">optimizer.zero_grad()

inputs, labels = iter(trainloader).next()

inputs, labels = inputs.cuda(), labels.cuda()

outputs = new_test_model(inputs)

loss = criterion(outputs, labels)

loss.backward()
</code></pre>
<p><strong>And then I checked the gradients</strong></p>
<pre><code class="lang-auto">for index, item in enumerate(new_test_model.named_children()):
    if(isinstance(item[1],nn.Conv2d)):
      wt_list = indices[item[0]]
      for i in wt_list:
        print(item[1].weight.grad[i,:,:,:])
</code></pre>
<p><strong>They turned out to be non zero as expected</strong><br>
<strong>because till now I have not made them as zero</strong></p>
<p>Then I turned the gradients to be zero using the code below</p>
<pre><code class="lang-auto">print("after loss backward")
for index, item in enumerate(new_test_model.named_children()):
  if(isinstance(item[1],nn.Conv2d)):
    wt_list = indices[item[0]]
    for i in wt_list:
      item[1].weight.grad[i,:,:,:]=0
      item[1].bias.grad[i]=0
</code></pre>
<p><strong>And checked the gradients again and they were zero as expected</strong></p>
<p>so when the gradients were Zero and the filters were themselves Zero there should be NO update in the filters but the filters were updated</p>
<p><strong>The filters from being zero despite the gradients being zero got updated to non zero.</strong></p>
<p>and to add to this even after I did optimizer.zero_grad()<br>
without calculating the loss again and executing loss.backward()<br>
just to check that even when all gradients are zero if then also the model is updated,<br>
then also using just <strong>optimizer.zero_grad() and then optimizer.step()</strong><br>
the model got updated again.</p>
<p><strong>Can you please explain what is happening or what am i missing or doing wrong.</strong></p>
<p><strong>I just want to make the filters zero and don’t want them to update</strong></p>

username: ptrblck
cooked: <p>If these parameters were updated in the past and if you are using an optimizer with internal running states for all parameters (e.g. <code>Adam</code>), this behavior is expected.<br>
<a href="https://gist.github.com/ptrblck/d9abccd4f52b1aa6d242da3338533169">Here</a> is a small example.</p>
<p>Could this be the case, i.e. are you training these parameters and freezing them after some iterations?</p>