SelectBackward0 vs AddmmBackward0

I_M: <p>Hello,</p>
<p>When I pass inputs <code>o = model(x)</code> and print <code>o.grad_fn</code> I get an <code>AddmmBackward0</code>.<br>
However, when I try to just take a single input, for example <code>o[1].grad_fn</code> I get a <code>SelectBackward0.</code><br>
Why is this?</p>
<p>When I use a DataLoader with <code>batch_size=1</code>, I get <code>AddmmBackward0</code>.</p>
<p>Anyways, down the line I have this issue:</p>
<pre><code class="lang-python">&gt;&gt;&gt;o[i] # got this from calling 
tensor([-2.0692,  2.0274], grad_fn=&lt;SelectBackward0&gt;)

# got this from data loader
#for i, data in enumerate(unknown_dataloader):
#  inputs, labels = data
#  outputs = model(inputs)
outputs
tensor([[-2.0692,  2.0274]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>if I call <code>self.criterion(o[i], labels)</code> I get an error: RuntimeError: size mismatch (got input: [2], target: [1])</p>
<p>How would I fix this for all <code>o</code>? I don’t want to use a dataloader to run the entire inputs in batch sizes of 1.</p>

ptrblck: <aside class="quote no-group" data-username="I_M" data-post="1" data-topic="157634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_m/48/35728_2.png" class="avatar"> I_M:</div>
<blockquote>
<p>However, when I try to just take a single input, for example <code>o[1].grad_fn</code> I get a <code>SelectBackward0.</code><br>
Why is this?</p>
</blockquote>
</aside>
<p>You are seeing <code>SelectBackward0</code> because you are indexing/selecting the output via <code>o[0]</code> which is a differentiable operation and are then checking the <code>.grad_fn</code> attribute of this indexed tensor.</p>
<aside class="quote no-group" data-username="I_M" data-post="1" data-topic="157634">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/i_m/48/35728_2.png" class="avatar"> I_M:</div>
<blockquote>
<p>if I call <code>self.criterion(o[i], labels)</code> I get an error: RuntimeError: size mismatch (got input: [2], target: [1])</p>
</blockquote>
</aside>
<p>You would need to explain the use case a bit more, i.e. which criterion is used, what the output and target shapes are expected to be, why you are indexing the output etc.</p>

I_M: <p>I’m using SoftMax &amp; CrossEntropy, this is a classification problem.</p>
<p>I’m implementing a novel approach that uses the values of backpropagation on a <em>trained</em> model, without optimizing. In essence, I want to save the backpropagation gradient values for every input given to me for this task.</p>
<pre><code class="lang-python">       for i, data in enumerate(some_dataloader): # dataloader batch size is 1
            inputs, labels = data
            model.zero_grad()
            outputs = model(inputs)
            loss = criterion(outputs, labels) # torch.nn.CrossEntropyLoss()
            # BP
            loss.backward()

            # get the gradient of some layer
            grad = some_layer.bias.grad # I use this later
</code></pre>
<p>Before this loop occurs I have already passed all the data I need through the model, <code>o = model(data)</code><br>
and so I want to use this output <code>o[i]</code> instead of recalculating it in the loop all over again.</p>
<p>Edit:<br>
If already on topic, is there a way for this to work on every sample with a larger batch size? For example, <code>loss= criterion(outputs, labels)</code> with batch size 32 would be 32 gradients, rather than 1?</p>

ptrblck: <p>For a per-sample gradient you might want to check <code>functorch</code> and <a href="https://pytorch.org/functorch/stable/notebooks/per_sample_grads.html">this tutorial</a> which explains the naive and the optimized approach.</p>

I_M: <p>Thanks.</p>
<p>Would you know of a solution to the first part of my problem? Even if I calculate the gradients efficiently, I would still need to pass the entire test data twice.</p>

ptrblck: <p>Calling <code>backward</code> on slices should work as seen here:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn

model = nn.Linear(10, 10)
x = torch.arange(5).unsqueeze(1).repeat(1, 10).float()

# one-by-one
grads1 = []
for x_ in x:
    x_ = x_.unsqueeze(0)
    out = model(x_)
    out.mean().backward()

    g = model.weight.grad.clone()
    grads1.append(g)

grads1 = torch.stack(grads1)

# batched forward, sample backward
model.zero_grad()
out = model(x)

grads2 = []
for o in out[:-1]:
    o.mean().backward(retain_graph=True)
    g = model.weight.grad.clone()
    grads2.append(g)
out[-1].mean().backward() # clear computation graph
g = model.weight.grad.clone()
grads2.append(g)
grads2 = torch.stack(grads2)

print((grads1 - grads2).abs().max())
# tensor(0.)
</code></pre>
<p>and I think it depends on your use case which approach would be the most efficient and most convenient one.</p>