Second order differentiation doesn't work for "x + 1"

kgg: <p>Hi,</p>
<p>I’m trying out second order differentiation using autograd, and the following example works:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

x = torch.tensor(2.0, requires_grad=True)

# Define the function
y = (x ** 2)

# Compute the first derivative
dy_dx = torch.autograd.grad(y, x, create_graph=True)[0]

# Compute the second derivative
d2y_dx2 = torch.autograd.grad(dy_dx, x, create_graph=True)[0]

print(d2y_dx2.item()) # Prints 12.0

</code></pre>
<p>HOWEVER, when I simply change “(x ** 2)” to “(x + 1)”, it throws error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53d74543c7ce100051809703c8dc0ef8797663b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/53d74543c7ce100051809703c8dc0ef8797663b6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53d74543c7ce100051809703c8dc0ef8797663b6.png" alt="image" data-base62-sha1="bXGQoABiojhRLBvdqOhjylr0FkG" width="690" height="469" data-dominant-color="F2E2DA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1064×724 38 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t understand why the second example doesn’t work if the first does, I was wondering if someone could explain this to me.<br>
( Pytorch: 2.0.0 stable with working CUDA, Ubuntu 18, Anaconda, docker)</p>

AlphaBetaGamma96: <p>Because <code>x+1</code> doesn’t have a 2nd order derivative, it’s a linear function.</p>

kgg: <p>Thank you very much for your answer!<br>
This behavior of autograd surprises me because I understand that the second order derivative of a linear function is always zero, however at the same time it does have a second order derivative. My concern is that if I write a generic function that uses the second derivative of its argument function, then I would prefer not to have to treat linear functions as a special case.<br>
Is there a way to change this default behavior?<br>
A “hack” I currently found is writing something like “(x ** 1) + 1” which it doesn’t complain about.</p>

AlphaBetaGamma96: <p>Could have a look at <code>torch.func</code> library, that should give zeros instead (I think)</p>

kgg: <p>Thank you!</p>
<p>Now that I think more about it, could it be that autograd is intended to be mainly used for gradient descent and it’s designed this way because if a derivative is constant zero then it’s likely a bug in the model?</p>

srishti-git1110: <p>Hi <a class="mention" href="/u/kgg">@kgg</a><br>
The function <code>x+1</code> is linear, and the second order gradient of it is zero.</p>
<p>When such is the case, many of the times, autograd disconnects the computation graph. That to say, autograd is unable to differentiate whether, for a tensor, it’s the case of zero gradient (constant function) or no gradient (<code>requires_grad=False</code> or no graph). In your code:</p>
<pre><code class="lang-auto">import torch
x = torch.tensor(2.0, requires_grad=True)

# Define the function
y = (x+1)

# Compute the first derivative
dy_dx = torch.autograd.grad(y, x, create_graph=True)[0]
print(dy_dx.requires_grad) # False
print(dy_dx.grad_fn) # None

# Compute the second derivative
d2y_dx2 = torch.autograd.grad(dy_dx, x, create_graph=True)[0] # error
</code></pre>
<aside class="quote no-group" data-username="kgg" data-post="3" data-topic="177459">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f14d63/48.png" class="avatar"> kgg:</div>
<blockquote>
<p>A “hack” I currently found is writing something like “(x ** 1) + 1” which it doesn’t complain about.</p>
</blockquote>
</aside>
<p>That is expected. I point you to the <a href="https://pytorch.org/tutorials/beginner/introyt/autogradyt_tutorial.html" rel="noopener nofollow ugc">docs</a> to understand why. Specifically, you might benefit from digging into how the computation graph is actually constructed during tensor calculations. Feel free to ask any questions on this part.</p>
<aside class="quote no-group" data-username="kgg" data-post="5" data-topic="177459">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/f14d63/48.png" class="avatar"> kgg:</div>
<blockquote>
<p>Now that I think more about it, could it be that autograd is intended to be mainly used for gradient descent and it’s designed this way because if a derivative is constant zero then it’s likely a bug in the model?</p>
</blockquote>
</aside>
<p>Not sure! While trainings NNs, we usually do not require/use second order optimizations, so I am just guessing that this behaviour of autograd is intended to aggressively free as much memory as possible while the model trains.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> might help clarify this behaviour better.</p>

ptrblck: <p>Yes, I think this is expected behavior as explained <a href="https://discuss.pytorch.org/t/differentiation-of-a-constant-with-autograd-grad/4415/2">here</a>.<br>
I.e. <code>torch.autograd.grad</code> expects to find <code>x</code> in the computation graph, which won’t be there since the output does not have any dependency on <code>x</code> anymore as it’s a constant.</p>