How does Pytorch Backward functions handle batched inputs?

Siddhanth_Ramani: <p>Hi, I am curious as to how Pytorch’s backward function handles multiple inputs.</p>
<p>Specifically, are the losses averaged across inputs in the final layer itself (cross entropy loss, etc.), or are the input specific loss matrixes passed on to previous layers and each layer averages/sums across these losses?</p>
<p>For ex. For a Input Layer(p neurons)-&gt;Linear(q neurons)-&gt;CrossEntropy NN, and for a batched input of size “b”, does the backward differential that reaches the Linear layer have a shape of “q” or “bxq”?</p>
<p>The reason for the question is because the implementation of the latter is simpler for the linear layer (dW is a simple outer product).</p>
<p>Thanks.</p>

KFrank: <p>Hi Siddhanth!</p>
<aside class="quote no-group" data-username="Siddhanth_Ramani" data-post="1" data-topic="214837" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddhanth_ramani/48/62072_2.png" class="avatar"> Siddhanth_Ramani:</div>
<blockquote>
<p>Specifically, are the losses averaged across inputs in the final layer itself (cross entropy loss, etc.)</p>
</blockquote>
</aside>
<p>Yes, in the simplest case the loss function is averaged over the batch dimension.</p>
<p>In a typical network architecture, the leading dimension of the input tensor is a<br>
batch dimension and this batch dimension is passed through each layer of the<br>
network so that the final output of the network has the same leading batch dimension.<br>
(It doesn’t <em>have</em> to be this way, but this is typical.)</p>
<p>The loss function then averages loss values over the output’s batch dimension (as<br>
well as possibly averaging over other things, such as pixels in a output image).  This<br>
is done by default by many built-in pytorch loss functions which are given the default<br>
setting of <code>reduction = 'mean'</code> when they are instantiated.</p>
<aside class="quote no-group">
<blockquote>
<p>or are the input specific loss matrixes passed on to previous layers and each layer averages/sums across these losses?</p>
</blockquote>
</aside>
<p>Not exactly.  Autograd passes the gradient of a single scalar loss function with<br>
respect to the output of some given intermediate layer back to that layer.  Because<br>
the output of that intermediate layer (typically) has a batch dimension, that gradient<br>
will have a batch dimension.</p>
<aside class="quote no-group">
<blockquote>
<p>For ex. For a Input Layer(p neurons)-&gt;Linear(q neurons)-&gt;CrossEntropy NN, and for a batched input of size “b”, does the backward differential that reaches the Linear layer have a shape of “q” or “bxq”?</p>
</blockquote>
</aside>
<p>Let’s say that the intermediate layer you are calling “Linear(q neurons)” is<br>
<code>Linear (in_features = p, out_features = q)</code> and its input is a tensor of<br>
shape <code>[b, p]</code> (where <code>b</code> is the batch size).  Then its output will have shape <code>[b, q]</code>.<br>
When backpropagation reaches that layer, it will pass a gradient (of the scalar loss<br>
value) of shape <code>[b, q]</code> to that layer.  Autograd then computes the jacobian-vector<br>
product of the jacobian of the transformation performed by the layer with the “vector”<br>
that is the gradient that has been passed to the layer (as computed by the previous<br>
backpropagation steps).  This resulting jacobian-vector product now is the gradient<br>
of the final loss value with respect to the <em>input</em> to this layer (and has shape <code>[b, p]</code>).<br>
(Autograd also computes the gradient of the final loss value with respect to the<br>
<code>Parameter</code>s of the layer and accumulates them into those <code>Parameter</code>s’ <code>.grad</code><br>
properties.)</p>
<p>Best.</p>
<p>K. Frank</p>

Siddhanth_Ramani: <p>Hi Frank,<br>
Thank you for your reply.</p>
<p>Now that I think about it, doesn’t it <em>have</em> to include a batch dimension? - both forward and backward.</p>
<blockquote>
<p>It doesn’t <em>have</em> to be this way, but this is typical.</p>
</blockquote>
<p>In the forward direction we need to pass the batch dimension to each layer as if we perform some aggregation, we are losing information.<br>
In the backward direction, the same requirement holds as if we consider a single neuron, we need to take vector-vector dot product (weighted average), and any aggregation will lead to incorrect gradient computations.</p>
<blockquote>
<p>Autograd then computes the jacobian-vector product</p>
</blockquote>
<p>Shouldn’t this be jacobian-matrix product as otherwise the output will be vector.<br>
Moreover, just to confirm, the jacobian is the upstream gradient right?</p>
<p>Thank you so much for your prompt and detailed response.<br>
I appreciate it :).</p>

KFrank: <p>Hi Siddhanth!</p>
<aside class="quote no-group" data-username="Siddhanth_Ramani" data-post="3" data-topic="214837" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/siddhanth_ramani/48/62072_2.png" class="avatar"> Siddhanth_Ramani:</div>
<blockquote>
<p>Now that I think about it, doesn’t it <em>have</em> to include a batch dimension? - both forward and backward.</p>
</blockquote>
</aside>
<p>Logically, it is not necessary to have a batch dimension.  There are, however, a<br>
number of built-in pytorch layers that do require a batch dimension (but such batch<br>
dimensions can always have size 1).  For example, <code>Linear</code> does not require a<br>
batch dimension, but <code>Conv2d</code> does.</p>
<aside class="quote no-group">
<blockquote>
<p>In the forward direction we need to pass the batch dimension to each layer as if we perform some aggregation, we are losing information.</p>
</blockquote>
</aside>
<p>I’m not quite sure what you mean by this, but to clarify:  Let’s say we had a batch<br>
of five samples.  If we wanted to get rid of the batch dimension, we wouldn’t do it<br>
by somehow aggregating the five samples together – instead , we would take a<br>
single sample from the batch and run it through the network, doing this five times.</p>
<aside class="quote no-group">
<blockquote>
<p>Shouldn’t this be jacobian-matrix product as otherwise the output will be vector.<br>
Moreover, just to confirm, the jacobian is the upstream gradient right?</p>
</blockquote>
</aside>
<p>Using standard terminology, if you have a scalar-valued function of a vector input,<br>
<code>f (x_j)</code>, the gradient is the vector made up of the partial derivatives, <code>d f / d x_j</code>.<br>
If you have a vector-valued function of a vector input, <code>f_i (x_j)</code>, the jacobian is<br>
the matrix of partial derivatives, <code>d f_i / d x_j</code>.  It’s just terminology, but in this sense<br>
the jacobian is essentially a gradient, but for a vector-valued function.</p>
<p>A layer in a network is a vector-valued function.  For example, a <code>Linear (3, 5)</code> maps<br>
a vector of length <code>3</code> to a vector of length <code>5</code>.  When you backpropagate through a<br>
<code>Linear</code> its jacobian is never explicitly computed (let alone materialized).  Instead,<br>
autograd implicitly computes the jacobian-vector product of the jacobian for that<br>
<code>Linear</code> layer and the vector that is the gradient of the final scalar loss value with<br>
respect to the output of that <code>Linear</code>, as computed by the previous steps in the<br>
backpropagation.  This jacobian-vector product is the gradient of the final scalar loss<br>
value with respect to the input to that <code>Linear</code> (which is then passed back on up the<br>
backpropagation chain),</p>
<p>Best.</p>
<p>K. Frank</p>

Siddhanth_Ramani: <p>Hi Frank,<br>
Thank you for your reply.<br>
I believe most of my confusion was due to my own lack of understanding.</p>
<p>Thank you for giving this detailed an explanation!<br>
It was very clear.</p>
<p>Best regards,<br>
Siddhanth Ramani</p>