How to freeze or fix the specific(subset, partial) weight in convolution filter

username: hongjunchoi
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff0dd54b806beefe14711cb973f4205b63549c90.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff0dd54b806beefe14711cb973f4205b63549c90" title="torch forum1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff0dd54b806beefe14711cb973f4205b63549c90.png" alt="torch forum1" data-base62-sha1="Aojoktp9VqsUrP1AG7fZbW3glLG" width="690" height="261" data-dominant-color="F3F4F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch forum1</span><span class="informations">1183×449 13.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Above is my code for trying to fix, freeze the subset(temp_5.values ) of 25th convolution layer’s filter. And the result.<br>
I try to set the subset filter’s gradient as 0 so not to update. And the weight has been changed.</p>
<p>So two questions</p>
<ol>
<li>
<p>I wonder why the gradient becomes not  zero?</p>
</li>
<li>
<p>Is there other solution to fix the subset convolution filter?</p>
</li>
</ol>

username: ptrblck
cooked: <p>Which optimizer are you using?<br>
If you are using an optimizer with running estimates, note that even with a zero gradient, the parameters might still be updated, if the running estimates were previously calculated.<br>
E.g. <code>optim.Adam</code> would show such a behavior.</p>
<p>PS: It’s better to post the code directly by wrapping it into three backticks ``` as it makes debugging easier. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: hongjunchoi
cooked: <p>Thanks for the reply this is my code. I use the SGD optimizer</p>
<blockquote>
<p>‘’’<br>
optimizer = optim.SGD(im_net.parameters(), lr=0.1, momentum=0.9,<br>
weight_decay=5e-4)<br>
weight_reference = im_net.features[25].weight[temp_4].clone()<br>
for batch_idx, (inputs, targets) in enumerate(testloader):</p>
</blockquote>
<pre><code>    inputs = inputs.float().cuda()
    inputs, targets = inputs.to(device), targets.to(device)
        
        
    outputs = im_net(inputs)
                       
    ce_loss = criterion(outputs, targets)
    
    optimizer.zero_grad()
    loss = criterion(outputs, targets)
        
    
    #im_net.features[25].weight[temp_4].register_hook(lambda grad : grad * 0)
    loss.backward()
    im_net.features[25].weight.grad[temp_4] = 0.
    
    
    optimizer.step()
    print(weight_reference == im_net.features[25].weight[temp_4])
</code></pre>
<blockquote>
<p>‘’’</p>
</blockquote>
<p>By the way, could you explain about ‘optimizer with running estimates’? I don’t know what it means.</p>

username: ptrblck
cooked: <p><code>momentum</code> would be such an internal state, which uses the previous gradients to update the parameters even with a zero gradients.<br>
Here is a small example showing this behavior:</p>
<pre><code class="lang-python"># Setup
model = nn.Conv2d(1, 10, 3, 1, 1)
weight_reference = model.weight.clone()
optimizer = torch.optim.SGD(model.parameters(), lr=1., momentum=0.0)

# Should fail
model(torch.randn(1, 1, 3, 3)).mean().backward()
optimizer.step()
print((model.weight == weight_reference).all())

# Should work with momentum = 0.0
optimizer.zero_grad()
weight_reference = model.weight.clone()

model.weight.register_hook(lambda grad: grad * 0)
model(torch.randn(1, 1, 3, 3)).mean().backward()
optimizer.step()

print((model.weight == weight_reference).all())
</code></pre>
<p>The parameters should not be updated using <code>mementum=0.0</code>, while any other value for <code>momentum</code> will still update the parameters.</p>

username: hongjunchoi
cooked: <p>Thanks for the reply again <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"><br>
I change the momentum to zero but it still changes the gradient and the weight still updates.<br>
So I made the weight decay to zero and it works! I wonder why weight decay makes this difference.<br>
P.S If i have to set weight decay to zero, then is there other solution to give regularization like the weight decay do?</p>

username: ptrblck
cooked: <p>I think you would have to define the weight decay manually, e.g. as shown <a href="https://discuss.pytorch.org/t/weight-decay-in-the-optimizers-is-a-bad-idea-especially-with-batchnorm/16994/2">here</a>, and filter out the parameters, which should not be added to weight decay.</p>

username: SofiaCP
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Regarding the fact that parameters are still updated even when the grad is zero (momentum, decay, etc), will setting the grads to none solve this issue? ie, <code>optimizer.zero_grad(set_to_none=True)</code>?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Yes, this should be the case, since the optimizers should check if the gradients are even set (e.g. <a href="https://github.com/pytorch/pytorch/blob/73a6c36f1bfbf9aff04ba41cfe6ab06aa99883d9/torch/optim/adam.py#L111">here</a> for <code>Adam</code>). In any case, I would always double check the behavior especially if you are using custom optimizers, as they might use another workflow.</p>

username: SofiaCP
cooked: <p>Okay, this is the behaviour I was looking for. Thank you for your time!</p>

username: SinHanYang
cooked: <p>Hi, I would like to ask whether this method works for only freezing a part of the parameter.<br>
Since the code of <code>Adam</code> <a href="https://github.com/pytorch/pytorch/blob/73a6c36f1bfbf9aff04ba41cfe6ab06aa99883d9/torch/optim/adam.py#L111" rel="noopener nofollow ugc">here</a> is probably checking whether the whole parameter is <code>None</code>.</p>
<pre><code class="lang-auto">for p in group['params']:
    if p.grad is not None:
</code></pre>
<p>Then if I only want to freeze the subset of the parameter as <a class="mention" href="/u/hongjunchoi">@hongjunchoi</a> does, does <code>optimizer.zero_grad(set_to_none=True)</code> works when setting gradient of the subset to zero?</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>I’m unsure what “subset” refers to, but note that you cannot set some gradients to <code>None</code> and others to valid values for the same parameter. Each parameter has a <code>.grad</code> attribute which is either set to <code>None</code> or a tensor.</p>

username: SinHanYang
cooked: <p>Thanks for the reply!</p>
<p>My definition of the “subset” is same as <a class="mention" href="/u/hongjunchoi">@hongjunchoi</a> .<br>
My goal is to freeze the subset of the parameter, and the subset is given by a list of index <code>id=[0,1,2,...]</code>. Therefore, I set the subset’s gradient to zero, and the code is like:</p>
<pre><code class="lang-auto">param.grad[id]=0
</code></pre>
<p>This method doesn’t work if the optimizer has momentum part.<br>
However, I’m not sure whether <code>optimizer.zero_grad(set_to_none=True)</code> would prevent <code>Adam</code> from updating the subset since  <code>param.grad</code> is not <code>None</code> in this case.</p>

username: ptrblck
cooked: <p><code>optimizer.zero_grad(set_to_none=True)</code> is irrelevant, since you are setting the gradient to zero <em>after</em> a valid gradient was already calculated.<br>
The <code>optimizer.zero_grad()</code> call could have set the <code>.grad</code> to <code>None</code> before, but since a <code>backward()</code> pass was executed afterwards, the <code>.grad</code> attribute is already populated.<br>
As described in my previous post, it’s not possible to set an attribute to <code>None</code> <em>and</em> a valid value.</p>

username: SinHanYang
cooked: <p>I see. Is it impossible to freeze the subset of the parameter if I use <code>Adam</code> optimizer? Or there are others ways to do so instead of setting subset’s gradient to zero.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>I would recommend to explicitly restore the “frozen” values of the parameter as described in <a href="https://discuss.pytorch.org/t/is-there-any-way-to-set-the-requires-grad-as-false-of-particular-weights-instead-of-whole-layer/144495/2">this post</a>.</p>