What is the difference between autograd.backward() and autograd.grad()

username: Kevin96
cooked: <p>It seems like both functions can be used to compute gradient. In fact, in their source code they call the same function_execution_engine.run_backward().</p>
<p>So what is the difference of these functions in terms of use cases? I would assume autograd.grad is more general?</p>

username: albanD
cooked: <p>Hi,</p>
<p><code>.backward()</code> is very convenient when working with torch.nn because it does not require the user to specify which Tensors we want the gradients for (and the optimizer knows where to look for these gradients).<br>
<code>.grad()</code> requires you to specify the Tensors you want the gradients for and won’t populate the .grad field.</p>
<p>You could re-implement one with the other but they are just slightly different APIs for different use cases.</p>

username: Sudarshan_Babu
cooked: <p>Is there any reason for these functions to yield different gradients ?</p>
<p>I have been getting slightly different gradients, when everything else is maintained constant.</p>
<p>Any help would be much appreciated, thanks !</p>

username: albanD
cooked: <p>Hi,</p>
<p>If the <code>.grad</code> fields are not full of zero before you call <code>.backward()</code> then you can end up with different values.<br>
Also what do you mean by “slightly different” exactly?</p>

username: Sudarshan_Babu
cooked: <p>Hi,</p>
<p>Thank you for your quick response!</p>
<p>I figured that the difference was having <code>create_graph</code> set to <code>True</code> while using <code>.backward()</code>, and set to <code>False</code> while using <code>.grad()</code></p>
<p>My understanding of <code>create_graph</code> is that  it enables differentiation through the gradient computation step. I dont understand why this should affect the values of the computed gradients, any help clarifying this would be much appreciated. Thank you.</p>

username: albanD
cooked: <p>Hi,</p>
<p>Again it depends how big the difference. If the difference is at the level of numerical precision (or one or two orders of magnitude larger if you test a full network), then this is because create_graph forces us to use a backward that is differentiable.<br>
And because floating point arithmetic is not associative, using different implementations can lead to bit-wise difference in the result. These differences are then amplified by the later computations in the backward pass.</p>

username: Sudarshan_Babu
cooked: <p>Thanks for your explanation<br>
Below I have given the gradients computed with and without create_graph.</p>
<pre><code class="lang-auto">[ 17.8867,  21.4154, -18.7954,  ...,  -8.7862, -14.6281,  -4.4456] #computed with create_graph = False
[ 19.2660,  24.0623, -22.0106,  ..., -11.3593, -16.9430,  -4.7851] # computed with  create_graph = True
</code></pre>
<p>Could numerical precision lead to such a difference ?</p>

username: albanD
cooked: <p>They look quite correlated, so for a very deep network, it is not impossible. But quite improbable indeed.</p>
<p>Can you share the structure of your network? Or even better, a small test script that we can use to reproduce the issue?</p>

username: Kevin96
cooked: <p>create_graph is usually false unless you want to do second order differentiation.</p>

username: Sudarshan_Babu
cooked: <p><a class="mention" href="/u/alband">@albanD</a>  Thank you for helping me out, would you mind if I share a github link with the minimal example ?</p>
<p>I am using a model where the first layer is the bert embedding layer, and its attached to a classifier. The gradients for the classifier  however match with and without create_graph</p>

username: Sudarshan_Babu
cooked: <p><a class="mention" href="/u/kevin96">@Kevin96</a>, thanks, but my use case requires that I compute a higher order derivative</p>

username: albanD
cooked: <aside class="quote no-group" data-username="Sudarshan_Babu" data-post="10" data-topic="74663">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudarshan_babu/48/14962_2.png" class="avatar"> Sudarshan_Babu:</div>
<blockquote>
<p>Thank you for helping me out, would you mind if I share a github link with the minimal example ?</p>
</blockquote>
</aside>
<p>Sure, but we need to be able to easily run it and it shouldn’t take long <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"><br>
If you just use the bert embedding you see the same behavior?</p>

username: Sudarshan_Babu
cooked: <p>Hey !</p>
<p>I have the minimal example in this repo:<a href="https://github.com/sudarshan1994/create_graph_debug" rel="nofollow noopener">https://github.com/sudarshan1994/create_graph_debug</a></p>
<p>Its easy to run : <code>python test.py</code> and does not call any weird libraries haha !</p>
<p>The pre-trained model is also uploaded in the repo, and its essential for seeing the discrepancy. For randomly initialized models we dont see the discrepancy.</p>
<p>The whole thing takes a minute to run ! All the relevant action takes place between line 146 to line 170, the rest is just setting up the Bert model.</p>
<p>Thanks a lot for agreeing to take a look at the code.</p>

username: Sudarshan_Babu
cooked: <p>Hey <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>Were you able to look at it, or was the code too clunky (sorry if so! )?</p>
<p>Thanks !</p>

username: albanD
cooked: <p>Didn’t had time yet. I hopefully will have a bit of time this week end <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Sudarshan_Babu
cooked: <p>Thank you so much !!</p>

username: albanD
cooked: <p>Hey,</p>
<p>After doing the change to run on CPU (I don’t have a GPU on my machine), I can’t seem to reproduce this:</p>
<pre><code class="lang-auto">when create_graph is True
loss                     :18.81757354736328
/data/users/albandes/pytorch/3.6_debug_source/torch/functional.py:1242: UserWarning: torch.norm is deprecated and may be removed in a future PyTorch release. Use torch.linalg.norm instead.
  "torch.norm is deprecated and may be removed in a future PyTorch release. "
classifier grad norm     :394.94049072265625
bert embedding grad norm :20.38188934326172
When create_graph is False
loss                     :18.81757354736328
classifier grad norm     :394.94049072265625
bert embedding grad norm :20.38188934326172
</code></pre>
<p>So I will guess that some of the cuda libraries change and lead to different numerical value.<br>
Can you try to:</p>
<ul>
<li>disable cudnn: <code>torch.backends.cudnn.enabled=False</code>
</li>
<li>increase the precision to double: <code>input = input.double()</code> and <code>net.double()</code>
</li>
<li>test on cpu (remove the .cuda() and add a <code>map_location=torch.device('cpu')</code> to the model loading)</li>
</ul>

username: Sudarshan_Babu
cooked: <p>Hey thanks a lot for running the code !</p>
<p>But I ran the code on cpu and made the changes you asked me to make yet I got different results based on create_graph <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>I also fixed the random seed <code>torch.manual_seed(0)</code></p>
<pre><code class="lang-auto">when create_graph is True
loss                     :15.955138471719597
classifier grad norm     :394.79707122347384
bert embedding grad norm :23.128852966924228
When create_graph is False
loss                     :15.955138471719597
classifier grad norm     :394.79707122347384
bert embedding grad norm :19.880596727123276
</code></pre>
<p>I have pushed my cpu runable version of my code in the git repo, could you run that please.</p>
<p>Could the difference be due to different version of pytorch ?<br>
torch related libraries that I am using: <code>torch==1.6.0, torchfile==0.1.0, torchvision==0.7.0</code></p>
<p>Thanks a lot  for spending your time on this !!</p>

username: albanD
cooked: <p>Hey,</p>
<p>I can’t reproduce on master or with the nightly builds:</p>
<pre><code class="lang-auto">[albandes ~/tmp/create_graph_debug] git pull
Updating 15a5a85..169926d
Fast-forward
 test.py | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)
[albandes@devvm138.atn0 ~/tmp/create_graph_debug] . ../../local/pytorch/3.6_debug_source_env/bin/activate
(3.6_debug_source_env) [albandes ~/tmp/create_graph_debug] python test.py
when create_graph is True
loss                     :15.955138471719597
/data/users/albandes/pytorch/3.6_debug_source/torch/functional.py:1242: UserWarning: torch.norm is deprecated and may be removed in a future PyTorch release. Use torch.linalg.norm instead.
  "torch.norm is deprecated and may be removed in a future PyTorch release. "
classifier grad norm     :394.79707122347384
bert embedding grad norm :19.880596727129774
When create_graph is False
loss                     :15.955138471719597
classifier grad norm     :394.79707122347384
bert embedding grad norm :19.880596727129774
</code></pre>
<p>But I can on 1.6:</p>
<pre><code class="lang-auto">(3.6_release_binary_env) [albandes@devvm138.atn0 ~/tmp/create_graph_debug] python test.py
when create_graph is True
loss                     :15.955138471719597
classifier grad norm     :394.7970712234737
bert embedding grad norm :23.12885296693177
When create_graph is False
loss                     :15.955138471719597
classifier grad norm     :394.7970712234737
bert embedding grad norm :19.88059672712989
</code></pre>
<p>Can you install the nightly build from here <a href="https://pytorch.org/get-started/locally/">https://pytorch.org/get-started/locally/</a> and confirm this is working fine for you?</p>

username: Sudarshan_Babu
cooked: <p>Yeah on nightly it works  ! I get the same results as you, thanks a lot !</p>