How to stop updating the parameters of a part of a layer in a CNN model (not the parameters of the whole layer)?

username: MonKeyBoy
cooked: <p>For example, there are ten parameters(filters) in a CNN layer, how I can do to only update five of them and keep the rest unchanged(while considering efficiency)?</p>

username: KFrank
cooked: <p>Hi Monkey!</p>
<aside class="quote no-group" data-username="MonKeyBoy" data-post="1" data-topic="153173" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e68b1a/48.png" class="avatar"> MonKeyBoy:</div>
<blockquote>
<p>there are ten parameters(filters) in a CNN layer, how I can do to only update five of them</p>
</blockquote>
</aside>
<p>If your “ten parameters” are separate <code>Tensor</code>s (or <code>Tensor</code>s wrapped<br>
in <code>Parameter</code>s), set the <code>requires_grad</code> property of the five you wish<br>
to keep unchanged to <code>False</code> (and don’t add them to your optimizer).</p>
<p>If your “ten parameters” are all values in the same <code>Tensor</code>, the best<br>
approach, in my mind, is to store the values before the optimizer update<br>
and then, after the update, restore the values you wish to keep unchanged.</p>
<p>This approach is outlined in the following post:</p>
<aside class="quote quote-modified" data-post="2" data-topic="153040">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/set-requires-grad-according-to-a-binary-mask/153040/2">Set requires_grad according to a binary mask</a> 
  </div>
  <blockquote>
    Hi Megh! 

There is no way to do what you want – with or without using loops. 
The requires_grad property applies to the entire tensor, not to elements 
of the tensor individually. 
If your use case for this is to “freeze” just certain elements of the Conv2d 
weight, that is, to only update some of the elements of weight when 
training, the safest way is to store a copy of the elements prior to calling 
the update, and then restore them after the update.  E.g., something like 
this: 
with torch.…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: MonKeyBoy
cooked: <p>Thanks, K. Frank</p>
<p>Yes, you provide a useful method.<br>
But if we only update some parameters through “store and restore”, it will bring extra time consumption compare to the original training process.<br>
Is there some methods that don’t bring extra computation?</p>
<p>Best,<br>
MonKeyBoy</p>

username: KFrank
cooked: <p>Hi Monkey!</p>
<p>The short story is that floating-point operations != computation time.</p>
<aside class="quote no-group" data-username="MonKeyBoy" data-post="3" data-topic="153173" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e68b1a/48.png" class="avatar"> MonKeyBoy:</div>
<blockquote>
<p>But if we only update some parameters through “store and restore”, it will bring extra time consumption compare to the original training process.</p>
</blockquote>
</aside>
<p>Yes, for the typical use case, updating only some elements of, for example,<br>
a <code>weight</code> tensor will take longer than updating all of them.</p>
<aside class="quote no-group">
<blockquote>
<p>Is there some methods that don’t bring extra computation?</p>
</blockquote>
</aside>
<p>It is possible to avoid the extra <em>computation.</em>  But the key point is that<br>
avoiding computation doesn’t necessarily save time – and can actually<br>
take more time.</p>
<p>This is because the floating-point pipelines in the gpu (and cpu, for<br>
that matter), as well as the gpu “kernels” (the code that performs the<br>
computations), are optimized for performing certain highly-structured<br>
sequences of floating-point computations.</p>
<p>As a basic example, consider multiplying two large matrices together<br>
where one of the matrices has many zero elements.  You could avoid<br>
many floating-point operations by skipping the multiply-by-zeros.  But<br>
the logic to do so would break up the optimized way in which those<br>
structured floating-point operations are streamed through the gpu<br>
pipeline, and the pipeline would end up spending a lot of its time not<br>
performing <em>any</em> floating-point operations.</p>
<p>So unless your matrix is <em>very</em> sparse (has a very large percentage of<br>
zero elements), it is (much) cheaper in time – on this kind of optimized<br>
hardware – to multiply the full matrices together, rather than to use a<br>
sparse-matrix-multiplication algorithm to perform the computation.</p>
<p>Best.</p>
<p>K. Frank</p>