Simple understanding of backprop / gradient descent

username: snip3r77
cooked: <p>Please let me know if my understanding is correct</p>
<p>We  <strong>requires_grad= True</strong>  for  <strong>a</strong>  is because we want to optimize this parameter.<br>
We  <strong>loss.backward()</strong>  because we want to minimize loss to find  <strong>a</strong></p>
<p>thanks</p>
<blockquote>
<p>a = torch.tensor([-5.,5.],requires_grad=True)<br>
def update():<br>
<span class="hashtag-raw">#calculate</span> y_hat<br>
y_hat = x@a<br>
<span class="hashtag-raw">#calculate</span> loss<br>
loss = mse(y, y_hat)<br>
<span class="hashtag-raw">#print</span> loss every 10 loops<br>
if t % 10 == 0: print(loss)<br>
<span class="hashtag-raw">#compute</span> the derivatives, you can call .backward()<br>
loss.backward()<br>
<span class="hashtag-raw">#To</span> prevent tracking history (and using memory), you can also wrap the code block in with torch.no_grad():.<br>
with torch.no_grad():<br>
a.sub_(lr * a.grad) # w(t) = w(t-1) - lr dL/dw(t-1)<br>
a.grad.zero_()</p>
</blockquote>
<p>full code<br>
<a href="https://i.imgur.com/UCqEGal.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/UCqEGal.png</a></p>

username: albanD
cooked: <p>It’s mostly correct yes, I would say it slightly differently:</p>
<ul>
<li>a.requires_grad=True means that we will ask for gradients later and so the autograd should save everything that it needs to be able to do so.</li>
<li>a.is_leaf=True means that it was not computed in a differentiable manner (either created by the user or from Tensors that don’t require gradients). For such Tensor, if we compute their gradient, it should be saved in <code>a.grad</code>.</li>
<li>loss.backward() means compute the gradients for all the Tensor that require gradients that were used to create loss. By side effect, this will populate the .grad field of all the leafs that require gradients.</li>
<li>optimizer.step() is then used to update a using a.grad (note that a.grad gives a direction of descent for the loss above).</li>
</ul>

username: snip3r77
cooked: <p>Can you please explain a.is_leaf= True as I did not use this.<br>
How do I incorporate to my code?</p>
<p>Also, what do you mean by optimizer.step() ?</p>
<blockquote>
<p>with torch.no_grad():<br>
# does a.optimizer.step() replaces this line below ?<br>
a.sub_(lr * a.grad) # w(t) = w(t-1) - lr dL/dw(t-1)</p>
<pre><code>    a.grad.zero_()
</code></pre>
</blockquote>
<p>Thanks.</p>

username: albanD
cooked: <blockquote>
<p>Can you please explain a.is_leaf= True as I did not use this.</p>
</blockquote>
<p>A Tensor is a leaf if it does not have any “history” in the autograd sense. Meaning that we cannot propagate gradients from this Tensor back to its parents.<br>
There are two main cases here:</p>
<ul>
<li>leaf that does not require gradient mean a Tensor for which gradients are not tracked.</li>
<li>leaf that requires gradient mean that the user explicitly asked for gradients here and so the <code>.grad</code> field will be populated when you call .backward().</li>
</ul>
<blockquote>
<p>How do I incorporate to my code?</p>
</blockquote>
<p>You don’t need to <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> It is just in case you want to look in more details, you can check if a Tensor is a leaf or not.</p>
<blockquote>
<p>Also, what do you mean by optimizer.step() ?</p>
</blockquote>
<p>Yes exactly, we have builtin optimizer to do these kind of updates, but you can do them by hand as well.</p>