In this grad example, interesting how loss affects the value of w

GGPYTORCH000: <p>In the grad example below, x, y, w and b initialized to tensor for input, output, weight and bias respectively.<br>
and then in line 21, z is computed for “forward” pass to be compared against y later</p>
<p>z=torch.add(torch.mul(w, x), b)</p>
<p>and loss is computed by comparing z against y:<br>
loss = (y-z).pow(2).sum()</p>
<p>This backward seems to compute the gradient. WHere it is being updated, it seems it is updating w.grad and b.grad.<br>
loss.backward()</p>
<p>Interesting thing is how computing loss affects the w. How would loss know to update the w.grad and b.grad in this example?</p>
<p>Here is how I knew by stepping through line by line</p>
<blockquote>
<p>/gg/git/codelab/gpu/ml/pt/ml-with-pt-and-sk/ch13-pytorch-mechanics/p415-vector-grad-basic.py(24)-&gt;<br>
(Pdb) print(w.grad)<br>
None<br>
(Pdb) print(b.grad)<br>
None<br>
(Pdb) l<br>
19  	for i in range(0, 5):<br>
20  	    print("-------- “, i, " ---------”)<br>
21  	    z=torch.add(torch.mul(w, x), b)<br>
22  	    loss = (y-z).pow(2).sum()<br>
23  	<br>
24  →     loss.backward()<br>
25  	    print("loss: ", loss)<br>
26  	    print("w: ", w, type(w))<br>
27  	    print("b: ", b, type(b))<br>
28  	    print('dL/dw : ', w.grad, type(w.grad))<br>
29  	    print('dL/db : ', b.grad, type(b.grad))<br>
(Pdb) n<br>
/gg/git/codelab/gpu/ml/pt/ml-with-pt-and-sk/ch13-pytorch-mechanics/p415-vector-grad-basic.py(25)()<br>
 → print("loss: ", loss)<br>
(Pdb) print(w.grad)<br>
tensor([ 0.6732, -0.0102, -0.3356,  0.1399, -0.0511], device=‘cuda:0’)<br>
(Pdb) print(b.grad)<br>
tensor([ 0.7562, -0.3724, -0.3716,  0.2598, -0.0698], device=‘cuda:0’)<br>
(Pdb)</p>
</blockquote>
<p>CODE EXAMPLE:</p>
<p>cat -n p415-vector-grad-basic.py<br>
1	import torch<br>
2	import code<br>
3	cuda = torch.device(‘cuda’)<br>
4	<br>
5	# Create weight and bias values.<br>
6	<br>
7	TENSOR_SIZE=5<br>
8	w=torch.rand(TENSOR_SIZE, requires_grad=True, device=‘cuda’)<br>
9	b=torch.rand(TENSOR_SIZE, requires_grad=True, device=‘cuda’)<br>
10	<br>
11	torch.manual_seed(1)<br>
12	<br>
13	# Create input(x), output (y, expected).<br>
14	# input(x) used for forward pass: z=w<em>x+b, z will be computed y rather than expected y. diff=(z-y)<br>
15	<br>
16	x=torch.rand(TENSOR_SIZE, device=‘cuda’)<br>
17	y=torch.rand(TENSOR_SIZE, device=‘cuda’)<br>
18	<br>
19	for i in range(0, 5):<br>
20	    print("-------- “, i, " ---------”)<br>
21	    z=torch.add(torch.mul(w, x), b)<br>
22	    loss = (y-z).pow(2).sum()<br>
23	<br>
24	    loss.backward()<br>
25	    print("loss: ", loss)<br>
26	    print("w: ", w, type(w))<br>
27	    print("b: ", b, type(b))<br>
28	    print('dL/dw : ', w.grad, type(w.grad))<br>
29	    print('dL/db : ', b.grad, type(b.grad))<br>
30	<br>
31	    # verifying output of loss.backward…<br>
32	<br>
33	    print(“verifying output of loss.backward…(compare with DL/DW)”)<br>
34	    test1=2 * x * ((w</em>x+b)-y)<br>
35	    print("dL/dw    : ", w.grad)<br>
36	    print("t        : ", test1[:5])<br>
37	<br>
38	    # update weights<br>
39	<br>
40	    w1 = w + w.grad<br>
41	    b1 = b + b.grad<br>
42	    w=w1.detach()<br>
43	    w.requires_grad=True<br>
44	    b=b1.detach()<br>
45	    b.requires_grad=True<br>
46	<br>
47	    print("new updated w1/b1: ")<br>
48	    print("w: ", w, type(w))<br>
49	    print("b: ", b, type(b))<br>
50</p>

srishti-git1110: <p>Hi,<br>
W and b are leaf nodes in the computational graph of loss.</p>
<p>As you call backward on the loss, gradients flow through the backward graph and the grad attribute of the leaf nodes is populated.</p>
<p>Does that answer your question?</p>

GGPYTORCH000: <p>yes, very nice answer and direction. Any good documentation on computational graph?<br>
Just quickly searched and found following:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/overview-of-pytorch-autograd-engine/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/overview-of-pytorch-autograd-engine/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/blog/overview-of-pytorch-autograd-engine/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/computational-graphs-constructed-in-pytorch/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/computational-graphs-constructed-in-pytorch/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:584/500;"><img src="https://pytorch.org/assets/images/augmented_computational_graph.png" class="thumbnail" width="584" height="500"></div>

<h3><a href="https://pytorch.org/blog/computational-graphs-constructed-in-pytorch/" target="_blank" rel="noopener nofollow ugc">How Computational Graphs are Constructed in PyTorch</a></h3>

  <p>In the previous post we went over the theoretical foundations of automatic differentiation and reviewed the implementation in PyTorch. In this post, we will be showing the parts of PyTorch involved in creating the graph and executing it. In order to...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


srishti-git1110: <p>The official docs suffice. Feel free to post here if you have any doubts.</p>