What step(), backward(), and zero_grad() do

isalirezag: <p>Can someone explain what is the difference between step() and  backward() and what they do?<br>
also can you explain when we should use zero_grad()? should we use it whenever we want to do backward()?</p>

tom: <p>Hopefully, you use them in the other order - <code>opt.zero_grad()</code>, <code>loss.backward()</code>, <code>opt.step()</code>.</p>
<ul>
<li>
<code>zero_grad</code> clears old gradients from the last step (otherwise you’d just accumulate the gradients from all <code>loss.backward()</code> calls).</li>
<li>
<code>loss.backward()</code> computes the derivative of the loss w.r.t. the parameters (or anything requiring gradients) using backpropagation.</li>
<li>
<code>opt.step()</code> causes the optimizer to take a step based on the gradients of the parameters.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

Mahmoud_Housam: <p><a class="mention" href="/u/tom">@tom</a> I can’t express my thanks to you man. The most clearer explanation <img src="https://discuss.pytorch.org/images/emoji/apple/heart_eyes.png?v=9" title=":heart_eyes:" class="emoji" alt=":heart_eyes:"></p>

GoingMyWay: <p>Very gooooooooooooooood, man!</p>

John_Grabner: <p>Does zero_grad clear all state information in the optimizer or can the optimizer maintain some information from the previous minibatch?<br>
<strong>For example 1</strong>: if I change batch size as a function of epoch do I need to recreate the optimizer to zero out its internal state?<br>
<strong>For example 2</strong>: if I have a common model (cnn layers), and 2 different heads (FC layers) with a small amount of data for head 1 and head 2, and a lot of data for head 1. I can train head 1, then train head 2.  Can I ping pong between training head 1 and head 2 or with the optimizer get all confused because of an internal state or discontinuous loss from the previous minibatch?</p>
<p>Note: the above examples are only to understand the optimizer and if it has any state information.</p>

Nathan_Wood: <p>Both examples should work with a single optimizer*.</p>
<p>Basically, every tensor stores some information about how to calculate the gradient, and the gradient. The gradient is (when initialized), the same shape but full of 0s. When you do <code>backward</code>, this info is used to calculate the gradients. These gradients are added to each tensor’s <code>.grad</code>. When you do <code>step</code>, the optimizer updates the weights based on the gradients. When you do <code>zero_grad</code>, the gradients are set to 0.</p>
<p>Most optimizers store extra data (eg momentum of the weights). However, this is the same size as the weights, and so is independent of the batch size.</p>
<p>*If you have two different models or parts of a model that you train separately, it makes sense to create two different optimizers. As long as you provide the correct parameters to the optimizer, this will work.</p>

John_Grabner: <p>Thank you, your response is very helpfull in my quest to learn pytorch.</p>

jkf: <p>Would you elaborate on what <code>zero_grad</code> is doing to 'accumulate the gradients from all <code>loss.backward()</code> calls`? The current gradients apply math ±*/ on previous gradients? Or stacking those tensors together occupying memory which in most cases we would avoid? Thanks</p>

Megh_Bhalerao: <p><a class="mention" href="/u/jkf">@jkf</a> - If we do not do <code>zero_grad</code>, then what will happen is that the the gradients of all your previous batches will keep adding up for a particular weight say <code>w</code> and the weight will get updated with the sum of all the previous gradients instead of the gradient of the current batch.</p>