Backward operation on an output tensor w.r.t batch input tensor

username: jarvico
cooked: <p>Hi, I need to calculate backward derivative of output tensor with respect to a batch of input tensor.</p>
<p>Here is the details:</p>
<p>Input shape is 64x1x28x28 (batch of mnist images) output shape is 64x1.Output is calculated based on some logic using the outputs of feedforward operation.</p>
<p>So actually, for each image of shape 1x1x28x28,I have a scalar value in output.</p>
<p>I can do it when using only one image tensor but batch operations are more efficient to handle and faster so I need to implement it.</p>
<p>The point is backward derivative of each element’s in the output should correspond to the related image tensor in the input tensor.</p>
<p>I know that backward operation is done based ona scalar value. So when I apply backward method on output tensor , I got error. And, I couldn’t handle this in Pytorch.</p>
<p>Can you please help me on this?</p>
<p>BR,<br>
jarvico</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jarvico" data-post="1" data-topic="110925">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>I know that backward operation is done based ona scalar value. So when I apply backward method on output tensor , I got error. And, I couldn’t handle this in Pytorch.</p>
</blockquote>
</aside>
<p>The <code>backward</code> operation accepts a <code>gradient</code> argument, so you could pass e.g. <code>torch.ones_like(output)</code> to it.<br>
Would this help in your case?</p>

username: jarvico
cooked: <p>Hi,</p>
<p>I might not explain my problem clearly. So please let me state again:</p>
<p>You may assume that I take a batch of mnist image as input x (64x1x28x28) and again you may assume that my model outputs a tensor (Out) of shape 64 x 1 which contains 64 scalar value that is obtained from each 1x28x28 image.</p>
<p>So what I need to implement is for example compute the backward derivative first output of my model with respect to first input in the batch. Ofcourse this is valid for each element in the batch.</p>
<p>And I would expect to get a tensor (Batch_Backward) of shape (64x1x28x28) after this batch backward derivative operation.  The important thing here Batch_Backward[0] should be calculated based on the backward derivative Out[0] which is actually the output for X[0].</p>

username: jarvico
cooked: <p>Something like this in the code:</p>
<pre><code class="lang-auto">def backward_batch(model, image):
    # Image is of shape 64x1x28x28


    output = model(image)
    output = softmax(output)

    Out = some_func(output)  #Out is of shape 64x1
    Out.backward()  # This does not work!

    return image.grad  # should be 64x1x28x28 as explained in the previous post
</code></pre>

username: jarvico
cooked: <p>Hi ptrblck,</p>
<p>Can you please check my additional explanations and comment whether it is possible or not?</p>
<p>Thanks in advance…</p>

username: ptrblck
cooked: <p>I’m still unsure, how to interpret this use case and what exactly is not working.<br>
In your previous post you’ve mentioned, that <code>Out.backward()</code> is not working. Could you explain, if you are seeing an error or if this method would break your use case somehow?</p>

username: jarvico
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Just forget the <code>Out.backward()</code> command and instead understand my actual need please.</p>
<p>Let me try to explain again in two parts.</p>
<p>Assume that I am using a batch of image from MNIST test loader and batch size is 64.</p>
<p>For this batch, I feed this batch image tensor( size 64x1x28x28) to my CNN model and get an output of size 64x10.</p>
<p>And I also apply some operation on my output tensor and get a final output tensor of shape 64x1.</p>
<p>That means for every input image tensor of shape 1x1x28x28 I get a scalar output of shape 1.</p>
<p>Is it clear until this part? If yes I continue with the second part.</p>
<p>Now, the thing I need to implement is I want to apply a backward derivative operation on my final output tensor. Actually what I want is to find the derivative of first output scalar in the final output tensor with respect to first input image in the input batch tensor. That is the case for all 64 output scalars and 64 input images.</p>
<p>In the example piece of code I provide, it is just used as a demonstration, please don’t be stuck with that. The backward operation there should be applied to every element of final output tensor one by one because for example first output is calculated based on the first input image and x.</p>
<p>and all the final derivatives should be stored in a tensor of shape 64x1x28x28.</p>
<p>(It is composed of sth like: d Out_1/d x_1 , d Out_2/d x_2 , …,  d Out_64/d x_64)</p>
<p>If the batch size was 1 I could do it without a problem. But when the batch size is 64 for example, I can’t do it. Can you please help me implement in Pytorch?</p>
<p>Is my problem clear for you now?</p>
<p>May be I should use 64 different ‘x’ tensors of shape 1x28x28 because the backward operation will be based on each one of these tensors… I don’t know…</p>
<pre><code class="lang-auto">def backward_batch(model, image):
    # Image is of shape 64x1x28x28

    x = torch.zeros_like(image, requires_grad=True)   # x is of shape 64x1x28x28

    output = model(image + x)
    output = softmax(output)

    Out = some_func(output)  #Final output is of shape 64x1
    Out.backward()  # How to handle here?

    return delta.grad  # should be 64x1x28x28 ```</code></pre>

username: ptrblck
cooked: <aside class="quote no-group" data-username="jarvico" data-post="7" data-topic="110925">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>Now, the thing I need to implement is I want to apply a backward derivative operation on my final output tensor. Actually what I want is to find the derivative of first output scalar in the final output tensor with respect to first input image in the input batch tensor. That is the case for all 64 output scalars and 64 input images.</p>
</blockquote>
</aside>
<p>That would be the usual use case, wouldn’t it? The samples in a batch are independent from each other (besides batch-dependent layers such as batchnorm layers).</p>
<aside class="quote no-group" data-username="jarvico" data-post="7" data-topic="110925">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jarvico/48/28509_2.png" class="avatar"> jarvico:</div>
<blockquote>
<p>The backward operation there should be applied to every element of final output tensor one by one because for example first output is calculated based on the first input image and x.</p>
<p>and all the final derivatives should be stored in a tensor of shape 64x1x28x28.</p>
<p>(It is composed of sth like: d Out_1/d x_1 , d Out_2/d x_2 , …, d Out_64/d x_64)</p>
<p>If the batch size was 1 I could do it without a problem. But when the batch size is 64 for example, I can’t do it. Can you please help me implement in Pytorch?</p>
</blockquote>
</aside>
<p>If I understand it correctly, your single sample approach would look like this:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
x = torch.randn(64, 2, requires_grad=True)
model = nn.Sequential(
    nn.Linear(2, 10),
    nn.ReLU(),
    nn.Linear(10, 10)
)
    

# single sample approach
for idx, x_ in enumerate(x):
    out = model(x_.unsqueeze(0))
    out.mean(1).backward()
    print(x.grad)
grad_ref = x.grad.clone()
</code></pre>
<p>I.e. in each iteration a single sample from the batch would be used and the gradient would be accumulated to the input tensor in the corresponding index. Note how the <code>x.grad</code> attribute is filled during the loop execution.</p>
<p>If the previous approach is what you are targeting, my previous suggestion of using <code>backward(torch.ones_like(out))</code> should still work:</p>
<pre data-code-wrap="python"><code class="lang-python"># clear gradients
x.grad = None

# pass complete batch to model
out = model(x)
out = out.mean(1)
out.backward(torch.ones_like(out))

# compare
print(torch.allclose(grad_ref, x.grad))
&gt; True
</code></pre>
<p>As you can see, both approaches yield the same gradients.<br>
If my code snippet is still not mathcing your use case, you would have to provide a code snippet to show what exactly is creating the issues, as I might not understand the terminology clearly.</p>

username: jarvico
cooked: <p>Many thanks for your support!</p>