How I can interpret the autograd graph?

Mahdi_Amrollahi: <p>How can I perceive the autograd grad which has been made by pytorch?<br>
Here I have a sample. I do not understand what arrows are or what AccumulatedGrads are and how I can interpret this graph?<br>
Here is the actual function:<br>
<code> y = x*z + x + z</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/3442bd83e10e548068c972eb3871e6984b46b3d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/3442bd83e10e548068c972eb3871e6984b46b3d5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3442bd83e10e548068c972eb3871e6984b46b3d5_2_365x500.png" alt="image" data-base62-sha1="7sjOLMI91msW5fBCpB1cLxkgltr" width="365" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3442bd83e10e548068c972eb3871e6984b46b3d5_2_365x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/3442bd83e10e548068c972eb3871e6984b46b3d5_2_547x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/4/3442bd83e10e548068c972eb3871e6984b46b3d5.png 2x" data-dominant-color="E8EBE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">644×882 33.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Mahdi!</p>
<aside class="quote no-group" data-username="Mahdi_Amrollahi" data-post="1" data-topic="157903" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahdi_amrollahi/48/62481_2.png" class="avatar"> Mahdi_Amrollahi:</div>
<blockquote>
<p>how I can interpret this graph?<br>
Here is the actual function:<br>
<code> y = x*z + x + z</code></p>
</blockquote>
</aside>
<p>The arrows indicate the direction of the forward pass.  Note, that the terms<br>
“root” and “leaves” of the graph correspond to the <em>backward</em> pass starting<br>
at the root and ending at the leaves.</p>
<p>The two top (blue) boxes are the two leaves, and indicate that <code>x</code> and <code>z</code><br>
have shape <code>[2]</code>.</p>
<p>“AccumulateGrad” is a special-purpose “grad_fn” that indicates that the<br>
backward pass should accumulate the computed gradient into the <code>.grad</code><br>
of the leaf tensor.</p>
<p>Python parses expressions with multiplication (<code>*</code>) taking precedence over<br>
addition (<code>+</code>), and with operators of equal precedence parsed left to right.</p>
<p>So (in the forward pass that builds the graph), first you get two arrows<br>
linking <code>x</code> and <code>z</code> to “MulBackward0”.  This is the <code>*</code> in the expression.</p>
<p>Then you get two arrows linking the result of the “MulBackward0” box<br>
(namely <code>x*z</code>) and <code>x</code> itself to an “AddBackward0”.  This is the first <code>+</code> in<br>
the expression, reading left to right.</p>
<p>Then you get two arrows linking the first “AddBackward0” box (namely<br>
<code>x*z + x</code>) and <code>z</code> to a second “AddBackward0”.  This is the second <code>+</code>.</p>
<p>The result of the second “AddBackward0” box is the whole expression,<br>
<code>x*z + x + z</code>, and is linked by an arrow to the bottom (green) box – the<br>
root – that indicates that <code>y</code> is also a tensor of shape <code>[2]</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Mahdi_Amrollahi: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="157903">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>ward pass should accumulate the compu</p>
</blockquote>
</aside>
<p>Thanks,<br>
I saw that there are two arrows from AccumulatedGrad to AddBackward0s, so may I know what they point to?</p>

srishti-git1110: <p>Which two arrows from AccumulatedGrad to AddBackward0s are you referring to?<br>
There is one direct arrow from x (AccumulatedGrad) to the first AddBackward0. Similarly, one direct arrow from z (AccumulatedGrad) to the second AddBackward0.</p>
<p>Are you pondering why there are two AddBackward0s in the graph? - It is because xz ‘+’ x is the first addition operation, while xz + x ‘+’ z is the second addition op.</p>
<p>Anything else you were referring to?</p>

Mahdi_Amrollahi: <p>Actually no,<br>
There are two AccumulatedGrad in the graph and each has an arrow to any AddBackward. Why the graph pointed from AccumulatedGrad to AddBackward nodes?<br>
Thanks</p>

srishti-git1110: <p>Each tensor requiring gradients (<code>required_grad=True</code>) is associated with a special <code>grad_fn</code> called <code>AccumulatedGrad</code> which is where the calculated gradients are stored (accumulated) after a backward call.</p>
<blockquote>
<p>Why the graph pointed from AccumulatedGrad to AddBackward nodes?</p>
</blockquote>
<p>The two <code>AccumulatedGrad</code> are associated with the tensors <code>x</code> and <code>z</code>. The first arrow from x’s <code>AccumulatedGrad</code> to the first <code>AddBackward0</code> indicates <code>x</code> getting added to <code>xz</code> (the arrow coming from <code>MulBackward0</code>).</p>
<p>The 2nd arrow from z’s <code>AccumulatedGrad</code> to the second <code>AddBackward0</code> indicates <code>z</code> getting added to <code>xz + x</code> (the arrow coming from the 1st <code>AddBackward0</code>).</p>
<p>Hope that helped.<br>
Srishti</p>

Mahdi_Amrollahi: <p>Thanks, now I got how this graph works.<br>
Here is just another sample for anyone in future:</p>
<p><code>y = x*z + x+z + torch.sin(x) + torch.cos(z)</code></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7249cf0d4fde41f65eac57215c6c099504512db.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7249cf0d4fde41f65eac57215c6c099504512db" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7249cf0d4fde41f65eac57215c6c099504512db_2_330x500.png" alt="image" data-base62-sha1="nQCbcsUEU76JaoOFvtMXHK4CBQv" width="330" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7249cf0d4fde41f65eac57215c6c099504512db_2_330x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7249cf0d4fde41f65eac57215c6c099504512db_2_495x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7249cf0d4fde41f65eac57215c6c099504512db.png 2x" data-dominant-color="ECEDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">650×984 41.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>