Can I find grad of all parameters before batch avg

username: MattyMatt
cooked: <p>Hi,</p>
<p>I can access gradients of the parameters of my network like so:</p>
<pre><code class="lang-auto">my_param = next(model.parameters())
my_grad = my_param.grad
</code></pre>
<p>Those gradients are, if I understand correctly, averaged over all the inputs, i.e. if we have 10 images in our input batch, those gradients are averaged across those 10 input images. Would it be possible to get the gradients for each input separately? For example, if my layer has 6x3x5x5 parameters, normally <code>grad</code> would have the shape of 6x3x5x5. However, if my batchsize is 4, would it be possible to obtain gradients of size 4x6x3x5x5, i.e. gradients before averaging?</p>
<p>A poor solution might be reducing batchsize to 1 and calculating the gradients for each single input separately, but I assume there is a proper way of doing it.</p>
<p>EDIT:<br>
I think its all about the loss I am using, because the loss is averaging everything, not the grad…But then if I do:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(reduce=False)
loss = criterion(outputs, labels)
loss.backward()
</code></pre>
<p>then I get an error:</p>
<blockquote>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
</blockquote>
<p>How do I get the gradients for each input separately then?</p>

username: KFrank
cooked: <p>Hi Matt!</p>
<aside class="quote no-group" data-username="MattyMatt" data-post="1" data-topic="154810" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/c4cdca/48.png" class="avatar"> MattyMatt:</div>
<blockquote>
<p>A poor solution might be reducing batchsize to 1 and calculating the gradients for each single input separately</p>
</blockquote>
</aside>
<p>This is, in essence, the right solution.  Autograd performs “vector-Jacobian<br>
products” to link the steps in the chain rule for differentiation together,<br>
which is to say that, in essence, it is computing the gradient for a single<br>
scalar in a given backward pass.</p>
<p>So if you want the gradient of the loss for each batch element separately,<br>
you have to perform a separate backward pass for each batch element.</p>
<aside class="quote no-group">
<blockquote>
<blockquote>
<p>RuntimeError: grad can be implicitly created only for scalar outputs</p>
</blockquote>
</blockquote>
</aside>
<p>This error message is autograd telling you that it will only compute the<br>
gradient for a single scalar at a time (or, more precisely, compute the<br>
vector-Jacobian product for a single specific vector at a time).</p>
<p>Best.</p>
<p>K. Frank</p>

username: ABD-01
cooked: <p>I believe you are looking for per sample gradients of the Model.<br>
Unfortunately, torch.autograd does not enable this behavior.<br>
You will have to use <a href="https://pytorch.org/functorch/stable/" rel="noopener nofollow ugc">functorch</a></p>
<blockquote>
<p>functorch is <a href="https://github.com/google/jax" rel="noopener nofollow ugc">JAX-like</a> composable function transforms for PyTorch.</p>
</blockquote>
<p>I guess you will find <a href="https://pytorch.org/functorch/stable/notebooks/per_sample_grads.html" rel="noopener nofollow ugc">this</a> helpful.<br>
I too am working on something similar. You can also try using JAX for <a href="https://jax.readthedocs.io/en/latest/jax-101/04-advanced-autodiff.html#per-example-gradients" rel="noopener nofollow ugc">per-sample-gradients</a> and found this to be faster than functorch.</p>

username: MattyMatt
cooked: <p>Thanks a lot guys, I will use the naive approach and switch to functorch  or JAX when I have speed issues.</p>
<p>Cheers!</p>