Need help computing gradient of the output with respect to the input

malfonsoarquimea: <p>Hi! I have a trained model and now I would like to compute the gradient of the output with respect to the inputs. Based on this post <a href="https://stackoverflow.com/questions/64988010/getting-the-outputs-grad-with-respect-to-the-input" class="inline-onebox" rel="noopener nofollow ugc">python - Getting the output's grad with respect to the input - Stack Overflow</a> I am doing it like this:</p>
<p><code>inputs.require_grad_()</code></p>
<p><code>outputs=model(preprocessing(inputs))</code></p>
<p><code>gradients= [torch.autograd.grad(inputs=inputs, outputs=outputs[i], allow_unused=True, retain_graph=True) for i in range(len(outputs))]</code></p>
<p>But it is extremely slow and I think I may be doing something wrong for it to be that slow.<br>
Is there something I am missing?<br>
Thanks in advance</p>

malfonsoarquimea: <p>I explained it really poorly. The slow part is not the grad computation itself, but the fact that, as I only get the gradient of one output per function call, I need to call the function one time per output. That is what makes it slow.<br>
In the end, I saw that I could simply pass a grad_outputs tensor as follows</p>
<pre><code class="lang-auto">torch.autograd.grad(outputs=output,inputs=input,grad_outputs=torch.ones_like(output),retain_graph=True)
</code></pre>
<p>This computes the gradient of all the outputs (with respect to their respective input) or so I think. Am I right?</p>

KFrank: <p>Hi Malfonsoarquimea!</p>
<aside class="quote no-group" data-username="malfonsoarquimea" data-post="2" data-topic="150950" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malfonsoarquimea/48/49326_2.png" class="avatar"> malfonsoarquimea:</div>
<blockquote>
<p>This computes the gradient of all the outputs (with respect to their respective input) or so I think. Am I right?</p>
</blockquote>
</aside>
<p>No, by using <code>grad_outputs = torch.ones_like (output)</code> you end up<br>
computing the inner product (dot product) of the gradients of <code>output</code> with<br>
the <code>torch.ones_like()</code> vector.  This does not preserve the gradient of<br>
each output as a separate entity.</p>
<p>At issue is the fact that, as currently implemented, autograd computes the<br>
gradient of a single scalar result (in any individual backward pass).  To get<br>
the gradients of the elements of <code>output</code> separately, you have to run multiple<br>
backward passes in a loop, such as you did in your original post.</p>
<p>And running such a loop over the elements of <code>output</code> does take time for<br>
each iteration of the loop.</p>
<p>This example illustrates these points:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; lin = torch.nn.Linear (1, 1, bias = False)   # a trivial "model"
&gt;&gt;&gt; lin.weight
Parameter containing:
tensor([[-0.2084]], requires_grad=True)
&gt;&gt;&gt; t = (torch.arange (4.) + 1).unsqueeze (-1)
&gt;&gt;&gt; t
tensor([[1.],
        [2.],
        [3.],
        [4.]])
&gt;&gt;&gt;
&gt;&gt;&gt; lin (t)   # output consists of four separate scalars
tensor([[-0.2084],
        [-0.4168],
        [-0.6252],
        [-0.8336]], grad_fn=&lt;MmBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; # use loop to compute gradient with respect to each scalar output
... # (for simplicity, this example unnecessarily runs four forward passes)
... for  i in range (4):
...     lin.weight.grad = None
...     lin (t)[i].backward()
...     print ('i:', i, ' gradient:', lin.weight.grad)
...
i: 0  gradient: tensor([[1.]])
i: 1  gradient: tensor([[2.]])
i: 2  gradient: tensor([[3.]])
i: 3  gradient: tensor([[4.]])
&gt;&gt;&gt; # gradient of a single scalar result
... lin.weight.grad = None
&gt;&gt;&gt; lin (t).sum().backward()   # sum over outputs to get a single scalar result
&gt;&gt;&gt; lin.weight.grad
tensor([[10.]])
&gt;&gt;&gt;
&gt;&gt;&gt; # using gradient = torch.ones() still just gives a single gradient
... lin.weight.grad = None
&gt;&gt;&gt; lin (t).backward (gradient = torch.ones (4, 1))
&gt;&gt;&gt; lin.weight.grad
tensor([[10.]])
</code></pre>
<p>(This example is packaged to illustrate computing gradients with respect to<br>
the weights of a “layer”, rather than with respect to “inputs,” as this is the<br>
more common use case, but there is no conceptual difference.)</p>
<p>Best.</p>
<p>K. Frank</p>

malfonsoarquimea: <aside class="quote no-group" data-username="KFrank" data-post="3" data-topic="150950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>ver the elements of <code>output</code> does take time for<br>
each iteration of the loop.</p>
<p>This example illustrates these points:</p>
</blockquote>
</aside>
<p>Hi! thanks for your answer which is very well packaged!<br>
Probably I am wrong but, as far as I have tested, using the code on my original post I would get a tensor of the size of the input where all the items except the one for which I provide the output are 0</p>
<pre><code class="lang-auto">&gt;&gt;&gt;inputs
tensor([[-1.0971,  1.1935,  1.2421],
        [-1.0270,  1.1890,  1.2345],
        [-0.9428,  1.1836,  1.2254],
        ...,
        [ 1.6314, -2.7406,  0.6462],
        [ 1.6619, -2.7843,  0.6395],
        [ 1.7008, -2.8399,  0.6311]], device='cuda:0', grad_fn=&lt;ViewBackward0&gt;)
&gt;&gt;&gt;outputs
tensor([[-10.9158],
        [ -2.2752],
        [ -5.6269],
        ...,
        [  2.9010],
        [ 22.6586],
        [ 25.9725]], device='cuda:0', grad_fn=&lt;AddmmBackward0&gt;)
&gt;&gt;&gt;gradients=torch.autograd.grad(inputs=inputs,outputs=outputs[0],allow_unused=True,retain_graph=True)
&gt;&gt;&gt;gradients
(tensor([[  534.2734,...='cuda:0'),)
&gt;&gt;&gt;gradients[0]
tensor([[  534.2734, -2844.8074,   709.5488],
        [    0.0000,     0.0000,     0.0000],
        [    0.0000,     0.0000,     0.0000],
        ...,
        [    0.0000,     0.0000,     0.0000],
        [    0.0000,     0.0000,     0.0000],
        [    0.0000,     0.0000,     0.0000]], device='cuda:0')
</code></pre>
<p>However, when doing it using passing the torch.oneslike() as grad_output I get one tensor of the size of the input. I assumed it contained the gradient of each output which respect to their respective input but maybe I am wrong</p>
<pre><code class="lang-auto">&gt;&gt;&gt;gradients=torch.autograd.grad(outputs=outputs,inputs=inputs,grad_outputs=torch.ones_like(outputs),retain_graph=True)
&gt;&gt;&gt;gradients
(tensor([[  534.2734,...='cuda:0'),)
&gt;&gt;&gt;gradients[0]
tensor([[  534.2734, -2844.8074,   709.5488],
        [ -247.5381,    85.9924,   537.6161],
        [  533.3944,   128.2310,    29.9466],
        ...,
        [ -621.1329,  -340.0153,   701.6570],
        [  620.2058,  1783.1162,  -927.3855],
        [  387.5083,   450.1457,   -69.2568]], device='cuda:0')
</code></pre>
<p>Here I don’t see all the gradients summed as one scalar output, and their values seem to be the same I would obtain by individually computing them following the code on my original post.</p>
<p>As I said, I am not experienced in this matter and probably I am not understanding it correctly, but could you point if there is some misconception I am assuming?<br>
thanks in advance</p>

KFrank: <p>Hi Malfonsoarquimea!</p>
<aside class="quote no-group" data-username="malfonsoarquimea" data-post="4" data-topic="150950" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/malfonsoarquimea/48/49326_2.png" class="avatar"> malfonsoarquimea:</div>
<blockquote>
<p>Here I don’t see all the gradients summed as one scalar output, and their values seem to be the same I would obtain by individually computing them following the code on my original post.</p>
</blockquote>
</aside>
<p>Here is what is probably going on:</p>
<p>If your model:</p>
<pre data-code-wrap="python"><code class="lang-python">outputs=model(preprocessing(inputs))
</code></pre>
<p>maps a batch of <code>inputs</code>, say of shape <code>[nBatch, N]</code>, to a batch of scalar<br>
<code>outputs</code> of shape <code>[nBatch]</code> (or similar), <em>and</em> each batch element of<br>
<code>outputs</code> depends <em>only</em> on the corresponding element of <code>inputs</code> (that is,<br>
<code>outputs[i]</code> depends only on <code>inputs[i]</code>, and not on <code>inputs[j != i]</code>),<br>
then the gradient of the scalar value, <code>outputs.sum()</code> with respect to<br>
<code>inputs[i]</code> will, in fact, be the gradient of <code>outputs[i]</code> with respect to<br>
<code>inputs[i]</code>.</p>
<p>All of the zeros you get when you compute the gradient of <code>outputs[i]</code><br>
with respect to all of the elements of <code>inputs</code> are to be expected.<br>
<code>inputs.grad[i]</code> is non-zero, as <code>outputs[i]</code> depends on <code>inputs[i]</code>,<br>
while all of the other <code>inputs.grad[j != i]</code> are zero because (by my<br>
assumption) <code>outputs[i]</code> does not depend on <code>inputs[j != i]</code> – no<br>
dependence, so zero gradient.</p>
<p>The last step, reiterating what I said in my previous post, is that:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.autograd.grad (outputs = outputs, inputs = inputs, grad_outputs = torch.ones_like (outputs))
</code></pre>
<p>is equal to:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.autograd.grad (outputs = outputs.sum(), inputs = inputs, grad_outputs = None)
</code></pre>
<p>In short, if your use case only requires that you compute the gradients<br>
of a batch of scalar outputs with respect to a batch of inputs where each<br>
output batch element only depends on the corresponding input batch<br>
element, then you do only need one call to <code>torch.autograd.grad()</code> (or,<br>
if you prefer, <code>outputs.sum().backward()</code>), and you only backpropagate<br>
through the computation graph once.</p>
<p>Best.</p>
<p>K. Frank</p>

malfonsoarquimea: <aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="150950">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>In short, if your use case only requires that you compute the gradients<br>
of a batch of scalar outputs with respect to a batch of inputs where each<br>
output batch element only depends on the corresponding input batch<br>
element, then you do only need one call to <code>torch.autograd.grad()</code> (or,<br>
if you prefer, <code>outputs.sum().backward()</code> ), and you only backpropagate<br>
through the computation graph once</p>
</blockquote>
</aside>
<p>Thanks very much for your elaborated answer K. Frank, it is really helpful!<br>
In fact, I am using a NeRF-like network where inputs are XYZ coordinates and the output is the density of the space at that XYZ coordinates. I would like to compute the density (output) gradient at each point in my coordinates (input) batch. So, as far as I understand, it is as you said because the gradient of the density at one point only depends on the coordinates of that point.<br>
As said, thanks very much for your time. Do you have any post you can recommend to learn about how pytorch autograd works in detail?<br>
Kind Regards!</p>

malfonsoarquimea: <p>I am revisiting this and still find your explanation super illustrative. Thanks again!</p>