Set a portion of a tensor to have requires_grad=False

Qwwq: <p>I want to set a portion of a tensor (such as <code>x[1:3, 5:10, :]</code>) to have <code>requires_grad=False</code>. Is this possible?</p>

ptrblck: <p>No, that’s not directly possible since the <code>.requires_grad</code> attribute is used for the entire tensor.<br>
You could either zero out the gradients of the frozen tensor part, restore their values after the rest was updated, or recreate the <code>x</code> tensor from smaller tensors having different <code>.requires_grad</code> settings via <code>torch.cat</code> or <code>torch.stack</code>.</p>

Qwwq: <p>That’s what I though too. I wish it was possible though. For example, if the <code>requires_grad</code> flag is instead made to contain indices of the tensor, then <code>optimizer.step</code> could perform the gradient update to only those indices. This wouldn’t hurt performance as the optimizer could fallback on the existing way of performing the update if <code>requires_grad</code> takes a boolean value.</p>

whubaichuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<ol>
<li>You could either zero out the gradients of the frozen tensor part, restore their values after the rest was updated.</li>
</ol>
<p>I think this solution has a problem. Because in principal, the frozen tensor will not contribute to the rest part. But here if we do this, the frozen tensor will contribute to the rest in terms of weight update.</p>
<ol start="2">
<li>recreate the <code>x</code> tensor from smaller tensors having different <code>.requires_grad</code> settings via <code>torch.cat</code> or <code>torch.stack</code></li>
</ol>
<p>Is this possible to has one official function in Pytorch?</p>

ptrblck: <aside class="quote no-group" data-username="whubaichuan" data-post="4" data-topic="157856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whubaichuan/48/69247_2.png" class="avatar"> whubaichuan:</div>
<blockquote>
<p>I think this solution has a problem. Because in principal, the frozen tensor will not contribute to the rest part. But here if we do this, the frozen tensor will contribute to the rest in terms of weight update.</p>
</blockquote>
</aside>
<p>I’m not sure I understand the description correctly, so could you add more details or an example?<br>
Frozen parameters generally still contribute to the <code>dgrad</code> computation, but won’t receive a <code>wgrad</code> (so their own <code>.grad</code> attribute remains empty). Here is a small example:</p>
<pre data-code-wrap="python"><code class="lang-python">lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)
for param in lin2.parameters():
    param.requires_grad = False


x = torch.randn(1, 1)
intermediate = lin1(x)
intermediate.retain_grad()
out = lin2(intermediate)
out.mean().backward()

print({n: p.grad for n, p in lin1.named_parameters()})
# {'weight': tensor([[0.3544]]), 'bias': tensor([0.8607])}
print({n: p.grad for n, p in lin2.named_parameters()})
# {'weight': None, 'bias': None}
print(intermediate.grad)
# tensor([[0.8607]])
</code></pre>

whubaichuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your quick reply. I put my code below. The question is that if we zero out the gradients of the lin2 part, but the lin1 part is still the same as before. In principal, if lin2 is frozen, then the weight of line2 is not updated. Then for lin1, the error is passed back by the old weight of lin2. But here, even we zero out the gradients of lin2 part, the error for lin1 seems to be passed back by the updated weight of lin2.</p>
<pre><code class="lang-auto">

import torch
import torch.nn as nn

lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)
lin3 = nn.Linear(1,1)
# for param in lin2.parameters():
#    param.requires_grad = False


x = torch.randn(1, 1)
intermediate = lin1(x)
intermediate.retain_grad()
out = lin2(intermediate)
final = lin3(out)
final.mean().backward()

print({n: p.grad for n, p in lin1.named_parameters()})
#{'weight': tensor([[0.5468]]), 'bias': tensor([0.5331])}
print({n: p.grad for n, p in lin2.named_parameters()})
#{'weight': tensor([[-0.8191]]), 'bias': tensor([0.8629])}
print(intermediate.grad)
#tensor([[0.5331]])

for name, module in lin2.named_modules():
   module.weight.grad = torch.zeros_like(module.weight.grad)
   module.bias.grad = torch.zeros_like(module.bias.grad)


print({n: p.grad for n, p in lin1.named_parameters()})
#{'weight': tensor([[0.5468]]), 'bias': tensor([0.5331])}
print({n: p.grad for n, p in lin2.named_parameters()})
#{'weight': tensor([[0.]]), 'bias': tensor([0.])}
print(intermediate.grad)
# tensor([[0.5331]])
</code></pre>

ptrblck: <aside class="quote no-group" data-username="whubaichuan" data-post="6" data-topic="157856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whubaichuan/48/69247_2.png" class="avatar"> whubaichuan:</div>
<blockquote>
<p>The question is that if we zero out the gradients of the lin2 part, but the lin1 part is still the same as before.</p>
</blockquote>
</aside>
<p>Yes, since the <code>dgrad</code> computation won’t be changed as mentioned before.</p>
<aside class="quote no-group" data-username="whubaichuan" data-post="6" data-topic="157856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whubaichuan/48/69247_2.png" class="avatar"> whubaichuan:</div>
<blockquote>
<p>In principal, if lin2 is frozen, then the weight of line2 is not updated.</p>
</blockquote>
</aside>
<p>That’s also correct and zeroing out the gradients will also not update the corresponding weights unless the optimizer has calculated running stats from previous iterations for this parameter, which is why restoring the weights might be considered the safe option.</p>
<aside class="quote no-group" data-username="whubaichuan" data-post="6" data-topic="157856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whubaichuan/48/69247_2.png" class="avatar"> whubaichuan:</div>
<blockquote>
<p>Then for lin1, the error is passed back by the old weight of lin2. But here, even we zero out the gradients of lin2 part, the error for lin1 seems to be passed back by the updated weight of lin2.</p>
</blockquote>
</aside>
<p>Where does the updated weight in <code>lin2</code> come from? The gradient computation is performed before calling <code>optimizer.step()</code> and performing the parameter updates.</p>

whubaichuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your detailed and quick reply. Ok, let me make it clear.</p>
<blockquote>
<p>Where does the updated weight in <code>lin2</code> come from? The gradient computation is performed before calling <code>optimizer.step()</code> and performing the parameter updates.</p>
</blockquote>
<p>In this code, as I wrote before, in principle, if the lin2 is <code>requires_grad = False </code>, the gradient will not be passed back to lin1 in the chain rule, because lin2 is not in the computation graph. In this computation graph, the connection between lin1 and lin2 is cut off.</p>
<p>However, if we manually zero out the gradients of the lin2 part after the <code>backward()</code>, the gradient is already passed back to lin1, which means that the <code>requires_grad</code> of lin2 is True although the <code>grad.data=0</code>. This situation actually will not update the weight of lin2, but this is not exactly the <code>requires_grad = False</code>.</p>
<p>What do you think?</p>

ptrblck: <aside class="quote no-group" data-username="whubaichuan" data-post="8" data-topic="157856">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whubaichuan/48/69247_2.png" class="avatar"> whubaichuan:</div>
<blockquote>
<p>In this code, as I wrote before, in principle, if the lin2 is <code>requires_grad = False </code>, the gradient will not be passed back to lin1 in the chain rule, because lin2 is not in the computation graph. In this computation graph, the connection between lin1 and lin2 is cut off.</p>
</blockquote>
</aside>
<p>No, this shouldn’t be the case, as freezing intermediate parameters would otherwise detach the computation graph and stop the training of previous layers.<br>
My code snippet also demonstrates that the gradient is still properly passed to previous layers.</p>
<p>I.e. in my previous post <code>lin1.parameters()</code> receive a valid gradient:</p>
<pre data-code-wrap="python"><code class="lang-python">print({n: p.grad for n, p in lin1.named_parameters()})
# {'weight': tensor([[0.3544]]), 'bias': tensor([0.8607])}
</code></pre>
<p>as well as the <code>intermediate</code> even though <code>lin2</code> is frozen.</p>
<p>Let me know if I misunderstand you post.</p>

whubaichuan: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, thanks for the explanation. You are right. Freezing the weight will not interrupt the error/gradient backpropagation.</p>
<p>Require_grad=false is not equal to .detach()</p>