Freezing intermediate layers while training top and bottom layers

t.g: <p>I am trying to understand how to get the “freeze” weights functionality work. Suppose I have a multi-layer network:  <code>x --&gt; L1 --&gt; L2 --&gt; L3 --&gt; y</code> .  (where <code>x</code> is input and <code>y</code> is output)<br>
Suppose I want to freeze <code>L2</code> layer in pytorch (only the <code>L2</code>, keeping <code>L1</code> and <code>L3</code> trainable).<br>
As discussed in [1] and bunch other posts, I simply set <code>requires_grad=False</code> for all params in L2, which disables gradient computation for L2 params. The observation there is, they are making layers close to the loss_function as trainable and layers away from loss_func are frozen (i.e. all trainable params are connected to the loss in the gradient graph,)</p>
<p>In my case, if I freeze L2,  does L1 still get trained in the back prop? If yes, how?</p>
<p>[1] <a href="https://discuss.pytorch.org/t/how-the-pytorch-freeze-network-in-some-layers-only-the-rest-of-the-training/7088" class="inline-onebox">How the pytorch freeze network in some layers, only the rest of the training?</a></p>

t.g: <p>maybe, in my case, I should not be setting <code>requires_grad=False</code> to the <code>L2</code> parameters,  instead I must exclude all L2 parameters from optimizer. That way, right amount of gradients will flow back to L1’s params, but optimizer does not update L2 parameters (which is analogous to freezing L2, yet keeping L1 trainable)</p>
<p>Is this a correct approach <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ?</p>

likethevegetable: <p>Sorry to not answer your question, but I’m struggling to reconcile this as well. I asked about it <a href="https://discuss.pytorch.org/t/using-two-neural-network-modules-to-optimize-only-one/39561/8">here</a> and <a href="https://discuss.pytorch.org/t/understanding-of-requires-grad-false/39765">here</a></p>

gsp-27: <aside class="quote no-group" data-username="t.g" data-post="1" data-topic="39776">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/t.g/48/74955_2.png" class="avatar"> t.g:</div>
<blockquote>
<p>In my case, if I freeze L2, does L1 still get trained in the back prop? If yes, how?</p>
</blockquote>
</aside>
<p>One of the simple thing you can try is just not include <code>L2</code> layer in the optimizer, so the gradients will still be computed but it will not update the parameters. Here I presume that you are not using batch-norm or dropout.</p>
<p><a href="https://pytorch.org/docs/stable/optim.html" rel="noopener nofollow ugc">See per parameter options</a></p>

t.g: <p>Thanks for the links <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=6" title=":+1:" class="emoji" alt=":+1:">  I will keep an eye on those threads as well.</p>

likethevegetable: <p>Hey t.g., I reached to a fellow named <a href="https://www.youtube.com/watch?v=MswxJw-8PvE" rel="nofollow noopener">Elliot Waite</a> on YoutTube and his video and answer to my comment very much clarified things for me.</p>
<p>In summary, think of setting requires_grad = False as a way of telling the L2’s parameters to not care about gradients (since the gradients are 0, it will never update). Now, what I was missing, is that it helps out greatly to think of L2 as a bunch of input parameters (those which you are freezing) connected to a bunch of computation nodes, and some of those computation nodes will have the output of L1 as another input. When you set the requires_grad = False, you are NOT freezing the gradients on the computation blocks, therefore the necessary gradients still get passed backward to L1. Elliot pointed out that you will get an error if you try to set requires_grad on a computation.</p>

t.g: <p>Hi Kale, thanks, just caught with your discussion on <a href="https://www.youtube.com/watch?v=MswxJw-8PvE&amp;lc=UgyUPrKsKQh8FC-0RIh4AaABAg.8sQJec8Agkf8sRZpw45V3Q" rel="noopener nofollow ugc">youtube comment thread</a>  and I am<br>
glad to learn that there is a checker built in pytorch that raises Error if we try to set requires_grad=False for interim nodes.</p>
<p>In summary:</p>
<ol>
<li>we <strong>do not (and can not)</strong> set <code>requires_grad=False</code> on intermediate layers that we want to freeze because their gradients are needed for calculating gradients of other connected nodes (i.e. the definition of a leaf node in grad network).</li>
<li>but intermediate nodes that we want to freeze can be excluded from the optimizer.</li>
</ol>
<p>So,</p><aside class="quote" data-post="2" data-topic="39776">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/t.g/48/74955_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/freezing-intermediate-layers-while-training-top-and-bottom-layers/39776/2">Freezing intermediate layers while training top and bottom layers</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    maybe, in my case, I should not be setting requires_grad=False to the L2 parameters,  instead I must exclude all L2 parameters from optimizer. That way, right amount of gradients will flow back to L1’s params, but optimizer does not update L2 parameters (which is analogous to freezing L2, yet keeping L1 trainable) 
Is this a correct approach <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title="slight_smile" alt="slight_smile" class="emoji"> ?
  </blockquote>
</aside>
<p>
is the right approach!</p>
<p>let me go and try this stuff.<br>
Cheers and thanks for the help,<br>
TG</p>

likethevegetable: <blockquote>
<p>I must exclude all L2 parameters from optimizer</p>
</blockquote>
<p>That is sufficient. But from a memory/time standpoint, the recommended practice would be to set <code>requires_grad=False</code> as well. Remember, you’re only freezing the <em>weights (parameters)</em> of L2, not the intermediate computations in that network. When backprop does its thing, the gradients of the loss function WRT the <em>weights (parameters)</em> of L2 are never required for updating the weights of L1 or L3.</p>
<p>I think you and I had the same misunderstanding of <code>requires_grad=False</code>. It helped me a lot to watch Elliot’s video and visualize the intermediate layer (i.e. L2) not as a lumped network, but as a connection of parameters being fed to computations.</p>

sezer: <p>Hi,</p>
<p>Probably I am getting you or <a class="mention" href="/u/t.g">@t.g</a> wrong, but is it possible that what you both say are different?</p>
<p>As I understand from what you say, we can set requires_grad=False for L2 in order to avoid training its parameters while we keep L1’s and L3’s parameters trainable.</p>
<p>From <a class="mention" href="/u/t.g">@t.g</a> I understand that we cannot set requires_grad=False for L2 and instead we need to exclude it from the optimizer.</p>
<p>So which is the proper way to not update a specific layer while updating all the others? Preferably without excluding it from the optimizer?</p>