Some questions about require_grad = False

username: Vincent
cooked: <p>Hello,everyone.</p>
<p>When I tried to freeze part of my model in the process, I set require_grad to False, but I found that memory increased by more than two times at this point. I wonder if PyTorch will save the weight at once and copy it to continue execution, and if so, is there a better way to save my memory since GPU cannot work now.</p>
<p>I tracked my memory usage and it worked until it was frozen.</p>
<pre><code class="lang-auto">At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 52 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 44 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 42 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 50 * Size:()                   | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 54 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 46 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 52 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 44 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 48 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 56 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 54 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 46 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8269.6 Mb

+ | 50 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 58 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 48 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 56 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 60 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 52 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 50 * Size:(1,)                 | Memory: 0.0001 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 58 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 62 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 54 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 60 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 52 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 56 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 64 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 62 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 54 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 58 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 66 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 56 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 64 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 68 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 60 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 58 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 66 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:8267.4 Mb

+ | 70 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
+ | 62 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 68 * Size:()                   | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32
- | 60 * Size:(1,)                 | Memory: 0.0002 M | &lt;class 'torch.Tensor'&gt; | torch.float32

At Backbone_idsub &lt;module&gt;: line 202                 Total Used Memory:20112.1Mb
</code></pre>
<p>Looking forward to your reply :).</p>

username: sio277
cooked: <p>This is not the answer to your question, but how did you get the memory info? Just the memory of parameters?</p>

username: Vincent
cooked: <p>hi, sio277.<br>
I use the following script based on pynvml to track memory usage.</p>
<blockquote>
<p><a href="https://github.com/Oldpan/Pytorch-Memory-Utils/blob/master/gpu_mem_track.py" rel="noopener nofollow ugc">https://github.com/Oldpan/Pytorch-Memory-Utils/blob/master/gpu_mem_track.py</a></p>
</blockquote>
<p>I now want to use different loss values during the process. At the beginning of the training I use the following function to freeze the model, but when I unfreeze it during the training, it will report an error out of memory.</p>
<pre><code class="lang-auto">    @staticmethod
    def set_grads(mod, state):
        for para in mod.parameters():
            para.requires_grad = state
</code></pre>
<p>By the way, I set a flag (True or False) to select different loss values, but it will get error info (out of memory), and when the flag is canceled it will be fine.</p>
<p>I wonder why that is.</p>
<p>Looking forward to your reply :).</p>

username: sio277
cooked: <p>Thanks. I don’t know the exact situation because I can’t see the whole code, but how about using <code>detach()</code> or <code>torch.no_grad()</code> to freeze part of your model? I think it is a safer approach.</p>

username: Vincent
cooked: <aside class="quote no-group" data-username="sio277" data-post="4" data-topic="118031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sio277/48/37684_2.png" class="avatar"> sio277:</div>
<blockquote>
<p><code>detach()</code> or <code>torch.no_grad()</code></p>
</blockquote>
</aside>
<p>Thank you for your reply!</p>
<p><code>detach</code> will create a new tensor for modules that do not require gradients, which seems even less applicable. Regarding <code>torch.no_grad()</code>, it seems that can only be applied to the serialized training process below. If I need to unfrozen the model during training, what should I do with <code>torch.no_grad()</code>.</p>
<pre><code class="lang-auto">class xxnet(nn.Module):
    def __init__():
        ....
        self.layer1 = xx
        self.layer2 = xx
        self.fc = xx

    def forward(self.x):
        with torch.no_grad():
            x = self.layer1(x)
            x = self.layer2(x)
        x = self.fc(x)
        return x
</code></pre>
<p>Thanks for patiently answering my questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: sio277
cooked: <aside class="quote no-group" data-username="Vincent" data-post="5" data-topic="118031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vincent/48/37056_2.png" class="avatar"> Vincent:</div>
<blockquote>
<p><code>detach</code> will create a new tensor for modules</p>
</blockquote>
</aside>
<p>Then, consider using <code>detach_()</code>, which is an inplace operation.</p>
<p>If you want <code>torch.no_grad()</code> for defrosting the model during training, how about something like this?:</p>
<pre><code class="lang-auto">if freeze is True:
    with torch.no_grad():
        xxnet()(input_tensor)
else:
    xxnet()(input_tensor)
</code></pre>

username: el_youssfi_azeddine
cooked: <p>hello, If you use torch.no_grad(): it uses same size of the memory?<br>
this line freezes all gradients.</p>

username: sio277
cooked: <aside class="quote no-group" data-username="el_youssfi_azeddine" data-post="7" data-topic="118031">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/el_youssfi_azeddine/48/36808_2.png" class="avatar"> el_youssfi_azeddine:</div>
<blockquote>
<p>If you use torch.no_grad(): it uses same size of the memory?</p>
</blockquote>
</aside>
<p>It reduces the memory consumption, because no forward tensors needed to calculate the gradient are saved.</p>

username: Vincent
cooked: <p>Thanks a lot, I used this structure and solved my problem ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:"></p>