Zero grad optimizer or net?

username: vvanirudh
cooked: <p>What should we use to clear out the gradients accumulated for the parameters of the network?</p>
<pre><code> optimizer.zero_grad()
 net.zero_grad()
</code></pre>
<p>I have seen tutorials use them interchangeably. Are they the same or different? If different, what is the difference and do you need to execute both?</p>

username: chenyuntc
cooked: <p>if <code>optimizer = optim.Optimizer(net.parameters())</code>,they are the same.</p>
<pre><code class="lang-python">    def zero_grad(self):
        """Sets gradients of all model parameters to zero."""
        for p in self.parameters():
            if p.grad is not None:
                p.grad.data.zero_()
</code></pre>

username: chenjus
cooked: <p>We’re supposed to clear the gradients each iteration before calling <code>loss.backward()</code> and <code>optimizer.step()</code>, correct?</p>

username: dgriff
cooked: <p>Yes, that’s is the suggested usage</p>

username: QuantScientist
cooked: <p>Hi,<br>
I am trying an example adapted from here:![nn02|478x500]</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/adamlerer/pytorch-examples">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/adamlerer/pytorch-examples" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/119dc525aa88423b9d055b174913c82f76e0f1199f36c62d6bf97f3a6a832ecb/adamlerer/pytorch-examples" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/adamlerer/pytorch-examples" target="_blank" rel="noopener nofollow ugc">GitHub - adamlerer/pytorch-examples: Simple examples to introduce PyTorch</a></h3>

  <p>Simple examples to introduce PyTorch. Contribute to adamlerer/pytorch-examples development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I copy pasted the example and this exception is thrown:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/7/7e4f0d3521ca303535dcfef8fefbddad5d813db9.png" data-download-href="https://discuss.pytorch.org/uploads/default/7e4f0d3521ca303535dcfef8fefbddad5d813db9" title="nn02"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e4f0d3521ca303535dcfef8fefbddad5d813db9_2_478x500.png" alt="nn02" width="478" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e4f0d3521ca303535dcfef8fefbddad5d813db9_2_478x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/7/7e4f0d3521ca303535dcfef8fefbddad5d813db9_2_717x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/7/7e4f0d3521ca303535dcfef8fefbddad5d813db9.png 2x" data-dominant-color="ECEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">nn02</span><span class="informations">843×881 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Has the API changed?<br>
Thanks,</p>

username: tom
cooked: <p>These days, if there isn’t a gradient (yet), <code>.grad</code> can be <code>None</code>.<br>
When I had that while trying some code, I fixed it with  <code>if w1.grad is not None:</code> .</p>
<p>Best regards</p>
<p>Thomas</p>

username: griver
cooked: <p>Hi,<br>
I’m trying to use an optimizer only on a part of the module parameters. Should I call module.zero_grad() instead of optimizer.zero_grad() if there are other layers between the loss and layer that i’m training?</p>
<p>And what if I need to train only the last layer(right before the loss) of a module? The gradients for the previous layers wouldn’t be computed at all. So there is no difference between calling module.zero_grad() and calling optimizer.zero_grads(), right?</p>

username: isalirezag
cooked: <p>any solution regarding this question?</p>

username: fermat97
cooked: <p>I also think there is no difference between model.zero_grad() and model_optimizer.zero_grad()</p>

username: ptrblck
cooked: <p>As <a class="mention" href="/u/chenyuntc">@chenyuntc</a> explained, if you pass all parameters of your model to the optimizer, both calls will be equal.<br>
However, there might be use cases where you would like to use different optimizers for different parts of your model. In such a case, <code>model.zero_grad()</code> would clear all parameters of the model, while the <code>optimizerX.zero_grad()</code> call will just clean the gradients of the parameters that were passed to it.</p>

username: Ankit_Taxak
cooked: <p>But if when we added a extra layer  like  this</p>
<p>classifier = nn.Linear(128, 200,bias=False)<br>
nn.init.xavier_uniform_(classifier.weight)<br>
model.add_module(“classify”,classifier)</p>
<p>but we do not want to update the parameter of this layer. So in this case what we should do?</p>

username: tom
cooked: <p>Something like</p>
<pre><code class="lang-auto">for p in classifier.parameters():
    p.requires_grad_(False)
</code></pre>
<p>before training should do the trick. Also filter the parameter from what you pass to the optimizer.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Joker
cooked: <p>For example ,when we create an optimizer like this:<br>
optimizer_ft=optim.SGD(model_ft.parameters(),lr=0.001,momentum=0.9)<br>
we will  add  the model’s  parameters to  this  optimizer,so when we call the function   optimizer.zero_grad()   it  will update  these  parameters ,but if  there is another  model  which  did’t register  this optimizer,   optimizer.zero_grad()    will  not update   it’s    grad</p>

username: Purushothaman_Srikan
cooked: <p><a class="mention" href="/u/chenyuntc">@chenyuntc</a> Can u please elaborate your answer?</p>
<p>Thanks in advance…</p>