How does autograd average across a minibatch?

miturian: <p>I apologize if this is adequately explained in the documention. If so, please just give me a link…</p>
<p>I have only found</p><aside class="quote" data-post="1" data-topic="13191">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shenkev/48/2105_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/does-pytorch-average-or-sum-gradients-over-a-minibatch/13191">Does PyTorch average or sum gradients over a minibatch?</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    For example when you retrieve the gradients like so: 
loss = F.nll_loss(output, target)
loss.backward()
for key, value in model.named_parameters():
    mygrad = value.grad

is mygrad the sum of gradients over the minibatch or the average?
  </blockquote>
</aside>
<p><a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html</a></p>
<p>neither of which seem explain this.</p>
<p>However, question is:<br>
The gradient in a given portion of the computational graph depends not only on the loss and the parameter values, but also on the input. If this is true, then even though the loss is summed over the minibatch outputs before backpropagation, how does the optimizer calculate the average gradient? Are all batch_size number of activations for each neuron saved during the forward step, or how?</p>
<p>I hope the question makes sense.</p>

KFrank: <p>Hi Miturian!</p>
<aside class="quote no-group" data-username="miturian" data-post="1" data-topic="131703" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miturian/48/34819_2.png" class="avatar"> miturian:</div>
<blockquote>
<p>how does the optimizer calculate the average gradient?</p>
</blockquote>
</aside>
<p>It depends <em>entirely</em> on how the “loss” on which you call <code>loss.backward()</code><br>
is calculated.</p>
<p>Autograd simply computes the gradient of on whatever scalar you called<br>
<code>.backward()</code>.</p>
<p>Most pytorch loss functions calculate the average across the minibatch<br>
of the per-sample losses, but they often given you the option to<br>
compute the <code>'sum'</code> or apply sample weightings.</p>
<p>You could, for example, have a minibatch of ten samples, but only<br>
compute the loss for the first sample, ignoring the other nine.  If you<br>
did that, you would only get the gradient of that first sample.</p>
<p>And, just to be clear, the <em>optimizer</em> does not calculate the gradients,<br>
averaged or otherwise.  By the time you call <code>opt.step()</code> the gradients<br>
have already been calculated (when you called <code>loss.backward()</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

miturian: <p>thanks for replying <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>ok, so, assume that my loss is the average over the individual losses in the minibatch. How does .backward() turn that into an average gradient? If my network was simply the operation “1/x”, with 10 different values of x in my minibatch, then presumably the gradient would have 10 different values, even though I was backpropagating an average loss?</p>

KFrank: <p>Hi Miturian!</p>
<p>First some contextual comments:</p>
<p>Pytorch’s autograd operates on tensor computations that produce<br>
a scalar.  (Autograd can manage things slightly more general than<br>
just a scalar result, but let’s leave that aside for this discussion.)</p>
<p><em>Leaf</em> tensors are those that are inputs to the overall computation,<br>
and can have their <code>requires_grad</code> property set to <code>True</code> or <code>False</code>.<br>
Autograd computes the gradient of the final scalar result with respect<br>
to (the elements of) the <code>requires_grad = True</code> leaf tensors.</p>
<p>Autograd neither knows nor cares that your computation might be<br>
implemented as a pytorch <code>Module</code> that we conceptually understand<br>
as a model, nor that one of its <code>requires_grad = False</code> leaf tensors<br>
is, in the typical use case, the input to your model and represents a<br>
batch of samples.  It neither knows nor cares that the <code>weight</code>s (and<br>
other <code>Parameters</code>s) of your model are its <code>requires_grad = True</code><br>
leaf tensors with respect to which it will calculate gradients.</p>
<p>This structure of applying a model with weights to a batch of samples<br>
and then computing a scalar loss function for the output of the model<br>
is higher-level structure that is distinct from autograd’s lower-level<br>
functionality of computing gradients of tensor operations that may<br>
or may not fit that higher-level structure.</p>
<p>(Autograd also neither knows nor cares that the gradients it computes<br>
will typically be used by an optimizer to train the weights of a model.)</p>
<p>Having gotten all of that out of the way …</p>
<aside class="quote no-group" data-username="miturian" data-post="3" data-topic="131703" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miturian/48/34819_2.png" class="avatar"> miturian:</div>
<blockquote>
<p>How does .backward() turn that into an average gradient?</p>
</blockquote>
</aside>
<p>At a mathematical level, the average of the gradients is the gradient<br>
of the average.  That is, taking the gradient is a linear operation.</p>
<p>But the more computationally mechanistic answer is that the average<br>
is computed by summing over a number of items and then multiplying<br>
that sum by one over the number of items.  Autograd tracks the<br>
“computation graph” of these operations and applies the chain rule to<br>
them numerically.  The gradient (in this case, the simple derivative) of<br>
constant times x (with respect to x) is just the constant.  Upstream of<br>
that in the computation graph is that the gradient of sum of x_i (with<br>
respect to the x_i) is just one for each of the x_i.</p>
<aside class="quote no-group">
<blockquote>
<p>If my network was simply the operation “1/x”, with 10 different values of x in my minibatch,</p>
</blockquote>
</aside>
<p>This example doesn’t really have the structure of a the model with<br>
weights applied to a batch of input samples.  (In your case, it sounds<br>
like you want the gradient with respect to the “input minibatch.”  That’s<br>
fine, but its not the typical use case.)</p>
<p>So let me phrase it a bit differently:</p>
<p>I have a 1d tensor, <code>T</code>, of shape <code>[10]</code> that consists of ten different values<br>
of <code>x</code> (that I will consider a minibatch).  I have a tensor operation that<br>
computes the (scalar) average of <code>1 / x</code> for that set of ten <code>x</code>s.  We won’t<br>
call this a “model” because it doesn’t really fit the typical model use case.<br>
This scalar-valued tensor operation has a single leaf tensor, <code>T</code>, and it<br>
has <code>requires_grad = True</code>.  Let’s use autograd to compute the gradient<br>
of the scalar result with respect to <code>T</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; T = torch.arange (10.) + 1.
&gt;&gt;&gt; T.requires_grad = True
&gt;&gt;&gt; T
tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10.], requires_grad=True)
&gt;&gt;&gt; scalar_result = (1 / T).mean()
&gt;&gt;&gt; scalar_result
tensor(0.2929, grad_fn=&lt;MeanBackward0&gt;)
&gt;&gt;&gt; scalar_result.backward()
&gt;&gt;&gt; T.grad
tensor([-0.1000, -0.0250, -0.0111, -0.0063, -0.0040, -0.0028, -0.0020, -0.0016,
        -0.0012, -0.0010])
</code></pre>
<p>The “magic” happens because the pytorch developers have implemented<br>
a companion <code>.backward()</code> function for each (differentiable) pytorch<br>
tensor operation that computes its gradient.  Autograd tracks how<br>
a series of such tensor operations are composed together in its<br>
“computation graph” and uses the chain rule to link together the<br>
individual <code>.backward()</code> functions to compute the gradient of the<br>
final scalar result with respect to the <code>requires_grad = True</code> leaf<br>
tensors.</p>
<p>Best.</p>
<p>K. Frank</p>

miturian: <p>Dear KFrank</p>
<p>Thank you <em>very</em> much for your detailed answers. I think I am getting close to understanding it, but will make one more question, to which the answer is hopefully ‘yes, that’s it’. If so, I’ll update this thread with a description of my own misunderstanding, so perhaps others making the same mistake can get some extra value from this thread.</p>
<p>Anyway:</p>
<p>I agree that my ‘1/x’ example was poorly chosen. I was simply grasping for a function with a non-constant gradient.</p>
<p>Let’s assume a very simple network:</p>
<p>y=x*w</p>
<p>with the loss being avr(y)</p>
<p>Let’s assume w=2, x=[1,2] (we have a batchSize of 2, input is a scalar)</p>
<p>In the forward pass, we get y=[2,4], L=.5*(2+4)=3</p>
<p>By the chain rule:</p>
<p><code>dL/dw=dL/dy0*dy0/dw+dL/dy1*dy1/dw</code></p>
<p>We see that there is a “batchSize” number of terms in this sum. As I understand you, all these factors are stored in the ‘graph’ until .backward() is called, is that correct? I was assuming that there would be averaging or something else going on as each layer was called in the network, but that was incorrect? Since autograd doesn’t know anything about mini batches, does that mean that x=1 and x=2 create two almost equal ‘arms’ in the graph, with a common root at L? If that picture makes sense.</p>
<p>So, the graph contains ALL computations that took place, even if a lot of them are linked to the same variables.</p>
<p>As in, I was expecting the graph to somehow make a rewriting like:</p>
<p><code>dL/dy0*dy0/dw+dL/dy1*dy1/dw=dL/dy0*(dy0/dw+dy1/dw)</code></p>
<p>to take advantage of symmetries, but from your ‘neither knows nor cares’ phrasing, I assume that autograd does no such thing, and instead keeps track of everything?</p>
<p>Is there a way to inspect this graph somehow? Or at least see its size? I am supposed to be including some of this in a lecture, and I think some way of visualizing how much is actually being saved in the graph would be instructive.</p>
<p>Regards</p>

KFrank: <p>Hi Miturian!</p>
<aside class="quote no-group" data-username="miturian" data-post="5" data-topic="131703" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/miturian/48/34819_2.png" class="avatar"> miturian:</div>
<blockquote>
<p>Let’s assume w=2, x=[1,2] (we have a batchSize of 2, input is a scalar)</p>
</blockquote>
</aside>
<p>I <em>think</em> you’re trying to say something slightly different, so let me<br>
rephrase it:</p>
<p><code>x</code> is a batchSize-2 “input” to the “model”; <code>w</code> is the (scalar) weight<br>
that defines this very simple “model”.</p>
<aside class="quote no-group">
<blockquote>
<p>In the forward pass, we get y=[2,4], L=.5*(2+4)=3</p>
</blockquote>
</aside>
<p>Yes.</p>
<aside class="quote no-group">
<blockquote>
<p>By the chain rule:</p>
<p><code>dL/dw=dL/dy0*dy0/dw+dL/dy1*dy1/dw</code></p>
</blockquote>
</aside>
<p>Yes, but I think it’s more helpful to think of this as a “tensor” chain<br>
rule, rather than a chain rule broken down into scalar pieces.</p>
<p>Thus:</p>
<p><code>dL / dw = sum_i { dL / dy_i * dy_i / dw }</code>,</p>
<p>where <code>sum_i {... * ...}</code> is a tensor <em>contraction.</em></p>
<p>That is:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt;
&gt;&gt;&gt; w = torch.tensor ([2.], requires_grad = True)  # "requires_grad = True" leaf tensor
&gt;&gt;&gt; x = torch.tensor ([1., 2.])                    # "requires_grad = False" leaf tensor
&gt;&gt;&gt; y = torch.mul (x, w)                           # non-leaf tensor (same as x * w)
&gt;&gt;&gt; L = y.mean()                                   # root (non-leaf) final result
&gt;&gt;&gt;
&gt;&gt;&gt; y.retain_grad()                                # so we can examine y.grad
&gt;&gt;&gt; L.retain_grad()                                # so we can examine L.grad
&gt;&gt;&gt;
&gt;&gt;&gt; w
tensor([2.], requires_grad=True)
&gt;&gt;&gt; x
tensor([1., 2.])
&gt;&gt;&gt; y
tensor([2., 4.], grad_fn=&lt;MulBackward0&gt;)
&gt;&gt;&gt; L
tensor(3., grad_fn=&lt;MeanBackward0&gt;)
&gt;&gt;&gt;
&gt;&gt;&gt; L.backward (retain_graph = True)
&gt;&gt;&gt;
&gt;&gt;&gt; L.grad   # unit "seed" to start chain rule
tensor(1.)
&gt;&gt;&gt; y.grad   # [dL / dy_0, dL / dy_1]
tensor([0.5000, 0.5000])
&gt;&gt;&gt; x.grad   # None, because x is a leaf with requires_grad = False
&gt;&gt;&gt; w.grad   # [dL / dw] = sum_i {dL / dy_i * dy_i / dw}
tensor([1.5000])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>all these factors are stored in the ‘graph’ until .backward() is called, is that correct?</p>
</blockquote>
</aside>
<p>If I understand you correctly that “these factors” are things like <code>dy_i / dw</code>,<br>
then, no, they’re are not stored in the graph, waiting for <code>.backward()</code> to<br>
be called.</p>
<p>Instead, when <code>y = x * w</code> is called (in the “forward pass”), only the fact<br>
that <code>*</code> (<code>torch.mul()</code>) was called is stored – by using a <code>MulBackward0</code><br>
object – together with any <code>torch.mul()</code>-specific <em>context</em> that <code>mul()</code>'s<br>
companion <code>backward()</code> function will need to calculate <code>dy_i / dw</code> <em>at<br>
the time that the backward pass is run.</em>  To emphasize, <code>dy_i / dw</code> is<br>
neither computed nor stored during the forward pass (but information<br>
sufficient to compute it during the backward pass is stored).</p>
<aside class="quote no-group">
<blockquote>
<p>does that mean that x=1 and x=2 create two almost equal ‘arms’ in the graph, with a common root at L?</p>
</blockquote>
</aside>
<p>I prefer to say that <code>y = x * w</code> creates a single arm, but that the arm in<br>
question is a “tensor arm.”  This is not purely semantic, in that autograd<br>
stores a single “tensor arm” for this operations, rather than two separate<br>
“scalar arms.”</p>
<aside class="quote no-group">
<blockquote>
<p>So, the graph contains ALL computations that took place,</p>
</blockquote>
</aside>
<p>Just to be clear, the graph only contains those computations that<br>
lead back to a <code>requires_grad = True</code> leaf tensor.  It doesn’t<br>
waste time or storage on computations that lead back solely to<br>
<code>requires_grad = False</code> leaf tensors.</p>
<aside class="quote no-group">
<blockquote>
<p>even if a lot of them are linked to the same variables.</p>
</blockquote>
</aside>
<p>If by this you mean that a lot of scalar computations are “linked to<br>
the same variables” because those scalar computations are part<br>
of the same tensor computation, the graph, in a sense, contains<br>
all of those scalar computations, but in an efficient way, because<br>
those scalar computations are packaged together as a single tensor<br>
operation.</p>
<aside class="quote no-group">
<blockquote>
<p>to take advantage of symmetries</p>
</blockquote>
</aside>
<p>If by “symmetries” you mean that the several (in this example, two)<br>
scalar operations are related to one another because they are part<br>
of the same tensor operation, then, yes, autograd takes advantage<br>
of this structure – and does so, as described above, by storing these<br>
several scalar operations together as a single tensor operation.</p>
<aside class="quote no-group">
<blockquote>
<p>Is there a way to inspect this graph somehow?</p>
</blockquote>
</aside>
<p>I don’t know how to, but I think I’ve seen posts that talk about probing<br>
the details of the graph.  (I would have to imaging that the pytorch api<br>
supports this, even if it isn’t well-documented or a part of what we<br>
think of as the “public-facing” api, but, again, I don’t know how to do it.)</p>
<p>Best.</p>
<p>K. Frank</p>

miturian: <p>Thank you, you have done a heroic job of answering</p>