Block the gradient flow in a certain path beckward

username: mojzaar
cooked: <p>Hi,</p>
<p>I wanted to creat an architecture that gradient flow is blocked from a certain layer beckward and the former layer will not update. I don’t want freeze that part. I want to put gradient of the intended layer be zero and block the gradient flow somehow the former layer will not be updated in this path. Is there any solution? I tried to put the grad of the intended output’s layer to  be zero manually in the forward path of the model but I still have the gradient values for the layers before that.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1" title="Screenshot_20191212-103709_Samsung%20Notes"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1_2_506x500.jpeg" alt="Screenshot_20191212-103709_Samsung%20Notes" data-base62-sha1="zOOV5eO0kUrOBqUmY2BgQGypSzT" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb0ac53dcf74f3fa6e57643e0725d373f503f8a1_2_1012x1000.jpeg 2x" data-dominant-color="F7F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_20191212-103709_Samsung%20Notes</span><span class="informations">1079×1065 131 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>If you know during the forward which part you want to block the gradients from, you can use <code>.detach()</code> on the output of this block to exclude it from the backward.</p>
<p>If you only know after the forward which part you want to block, you will need to add a hook to the Tensor that is the output of your block as:</p>
<pre><code class="lang-auto">out.register_hook(lambda grad: torch.zeros_like(grad))
</code></pre>

username: mojzaar
cooked: <p>Thanks for your reply. Yes, I know which part should block it.</p>
<p>I tried with the following code, but when I check the weights, all of them does not have the gradient.</p>
<pre><code class="lang-auto">class simm(nn.Module):
    def __init__(self, input_channels=1, output_channels=32):
        super(simm, self).__init__()
        self.layer1 =  nn.Conv3d(in_channels=input_channels, out_channels=output_channels, kernel_size=(3, 3, 3), padding=1)
        self.layer2 = nn.Conv3d(in_channels=output_channels, out_channels=2*output_channels, kernel_size=(3, 3, 3), padding=1)
        self.layer3 = nn.Conv3d(in_channels=2 * output_channels + 1, out_channels=1, kernel_size=(3, 3, 3), padding=1)
        self.layer4 =  nn.ReLU(inplace=False)

    def forward(self, x):
        o1 = self.layer1(x)
        o2 = self.layer2(o1)
        x1 = o2.detach()
        x2 = torch.cat((x1,x),1)
        o3 = self.layer3(x2)
        return self.layer4(o3)

ceriterion = nn.MSELoss()

model = simm().to(device)
w1 = model.layer1.weight.data
w2 = model.layer2.weight.data
w3 = model.layer3.weight.data

optimizer = optim.Adam(model.parameters(), lr=0.1)
    
data = torch.rand(5,1,10,12,13).to(device)

model.train()

output_b = model(data)

lablel = 10*torch.ones_like(output_b)
loss = ceriterion(output_b,lablel)
optimizer.zero_grad()
loss.backward()
optimizer.step() 
print(torch.mean(abs(model.layer1.weight.data - w1)))
print(torch.mean(abs(model.layer2.weight.data - w2)))
print(torch.mean(abs(model.layer3.weight.data - w3)))

tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
tensor(0., device='cuda:0')
</code></pre>
<p>I think the weights at the last layer should change, am I right?</p>

username: albanD
cooked: <p>So</p>
<ul>
<li>Do not use <code>.data</code>. That is not a thing anymore.</li>
<li>In this case, you want to clone the weights, otherwise you comparison compare twice the same Tensor: <code>w3 = model.layer3.weight.detach().clone()</code>.</li>
<li>layer1 and layer2 are expected not to change. layer3 will see change after you add the clone.</li>
</ul>

username: mojzaar
cooked: <p>Thanks for your help. You are right.</p>

username: jakelevi1996
cooked: <blockquote>
<p>Do not use <code>.data</code>. That is not a thing anymore.</p>
</blockquote>
<p>Please could you explain what you mean by this? If I am implementing my own optimiser, how else should my optimiser modify the underlying data of a tensor, without modifying the <code>data</code> attribute of that tensor?</p>

username: soulitzer
cooked: <p>Using <code>.data</code> bypasses certain checks around mutations which can lead to incorrect gradients<br>
If you are building your own optimizer you can modify the parameters in-place (e.g., torch.add_) in no-grad mode (this is also how optim writes their optimizers)</p>