Basic Explanation of torch gradients - '.backward()' vs autograd

username: aviasd
cooked: <p>Hi,<br>
I am trying to understand the way torch functions.<br>
I have tried to search the docs, but though torch docs are very descriptive, I couldn’t find a concrete answer.<br>
I will write this question differently, I will write what I have understood, and I will be glad if someone will correct me.</p>
<p>I understood that the way PyTorch and the autograd works is as follows:</p>
<ol>
<li>
<p>The computational graph is being built from the ground up in every <code>.forward()</code> pass.</p>
</li>
<li>
<p>The <code>.forward()</code> pass is adding the <code>.grad</code> attribute to all the parameters with the <code>.requires_grad</code> attribute set to <code>True</code> (but not attaching any value to it).</p>
</li>
<li>
<p>Then, when we use <code>.backward()</code>, we are backpropagating through the graph and adding the gradients to the <code>.grad</code> attributes of the parameters with the <code>.requires_grad=True</code>.</p>
</li>
<li>
<p>If we won’t do <code>.backward()</code>, the ‘.grad’ will be <code>None</code> (even if the <code>.requires_grad</code> is equal to <code>True</code>).</p>
</li>
<li>
<p>If we do multiple <code>.backward()</code> one after the other (even with no <code>.forward()</code> in between), the <code>.grad</code> attribute for every parameter will be accumulated.</p>
</li>
<li>
<p>If we do <code>.forward()</code> pass <code>with torch.no_grad()</code>, we are building the graph with <code>.grad</code> of all the parameters, set to <code>None</code>. So <code>.backward()</code> will have no effect.</p>
</li>
<li>
<p>If we do <code>optimizer.zero_grad()</code> we are setting to 0, the <code>.grad</code> attributes of all the parameters inside the optimizer (or inside the model, if we set the optimizer on all the model parameters).</p>
</li>
<li>
<p>The <code>optimizer.step()</code> is changing the weights of the parameters based on their <code>.grad</code> attribute.</p>
</li>
<li>
<p>I’m sure I have a mistake in few of these, maybe in all of them, but I want to understand why the autograd is being called <code>auto</code>… maybe it is calculating the gradients automatically when we do the <code>.forward()</code> pass? if so, why do we need the <code>.backward()</code>? we can do just the <code>optimizer.step()</code> and update the weights based on the gradients that has been calculated by the autograd…</p>
</li>
</ol>
<p>I hope someone will clarify some of this for me…<br>
Thank you,</p>

username: DJ_1992
cooked: <p>As far as I understand, <code>torch.autograd.</code> <code>backward()</code> calculates the  sum of gradients of the tensors w.r.t. graph leaves. And <code>optimizer.step()</code> calculates the new parameter(weights) values based on these gradients.</p>

username: albanD
cooked: <p>Hi,</p>
<p>I think all of these are pretty correct yes.<br>
Just keep in mind the distinction between autograd (that just runs the backward pass and compute gradients) and torch.nn (that is a set of utilities to work with neural nets).</p>
<ol start="2">
<li>
<p>We actually build the graph under the hook. The only place where you can see it is from the <code>.grad_fn</code> field.  The <code>.grad</code> attribute exists for every Tensor, it is just None by default.</p>
</li>
<li>
<p>We accumulate in the .grad fields of all the leaf Tensors (t.is_leaf()) that require gradients yes. (Or the ones on which you called retain_grad() that can be used to get the .grad field populated for non-leaf Tensors).</p>
</li>
<li>
<p>In no_grad mode, no graph is actually built, new Tensors will have requires_grad=False and grad_fn=None.</p>
</li>
<li>
<p>Here we enter the torch.nn realm. Your description is correct</p>
</li>
<li>
<p>Autograd is called auto from Automatic Differentiation. In comparison to Symbolic Differentiation or Numerical Differentiation that are other ways to compute gradients.</p>
</li>
</ol>

username: aviasd
cooked: <p>Thank you for your answer,</p>
<blockquote>
<ol start="2">
<li>We actually build the graph under the hook. The only place where you can see it is from the  <code>.grad_fn</code>  field. The  <code>.grad</code>  attribute exists for every Tensor, it is just None by default.</li>
</ol>
</blockquote>
<blockquote>
<ol start="3">
<li>We accumulate in the .grad fields of all the leaf Tensors (t.is_leaf()) that require gradients yes. (Or the ones on which you called retain_grad() that can be used to get the .grad field populated for non-leaf Tensors).</li>
</ol>
</blockquote>
<p>So as I understand it, in every <code>.forward()</code> pass, we build the graph under the hook with the <code>.grad_fn</code> that connects every variable to the function and variables that creates it (this is done by the <code>autograd</code>).<br>
When we do <code>.backward()</code>, we are calculating the <code>.grad</code> attribute for every leaf variables, from the <code>.grad_fn</code> attribute.<br>
But, how we calculate the gradients for non-leaf variables? If we are not doing so, how does the weights for those variables change by the <code>optimizer.step()</code>?</p>
<blockquote>
<ol start="4">
<li>In no_grad mode, no graph is actually built, new Tensors will have requires_grad=False and grad_fn=None.</li>
</ol>
</blockquote>
<p>So, in <code>no_grad</code> mode, <code>.backward()</code> and <code>optimizer.step()</code> will have no effect?</p>
<p>Thank you,</p>

username: albanD
cooked: <blockquote>
<p>But, how we calculate the gradients for non-leaf variables?</p>
</blockquote>
<p>You don’t need it because all the <code>nn.Parameter()</code> are leafs. So <code>.backward()</code> will always compute their gradient. And so the optimizer will always have the <code>.grad</code> to work with.</p>
<blockquote>
<p>So, in <code>no_grad</code> mode, <code>.backward()</code> and <code>optimizer.step()</code> will have no effect?</p>
</blockquote>
<p>Since the Tensor have requires_grad=False, you cannot call <code>.backward()</code> on it (you will get an error).<br>
You can still do <code>optimizer.step()</code> (remember this is a nn construct, unrelated to the autograd) and it will perform its update based on the current <code>.grad</code> (these can be None if no <code>.backward()</code> was called before).</p>
<p>Side note, for some optimizers (like Adam or SGD with momentum or SGD with weight decay), even with a gradient of <code>0</code>, the optimizer step will still change the weights.</p>