Backpropagation for each dimension of output

username: nil_tdr
cooked: <p>Hello!</p>
<p>In PyTorch when we call loss.backward() it performs backpropagation for the sample (for stochastic case). Let’s consider my output is 50 dimensional. I have two loss components. First one is an array of dimension 50. How can I run loss.backward() for each dimension separately? I also have another loss component which is a scalar and I want to do normal backpropagation for this one on whole output tensor, together with my first loss on each dimension of the output? Do I have to use a custom loss function?</p>
<p>Thanks a lot for you kind help!<br>
Nil</p>

username: KFrank
cooked: <p>Hi Nil!</p>
<aside class="quote no-group" data-username="nil_tdr" data-post="1" data-topic="185360" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<p>First one is an array of dimension 50. How can I run loss.backward() for each dimension separately?</p>
</blockquote>
</aside>
<p>I’m not sure that I understand your use case.  In particular, what do you<br>
mean by “run loss.backward() for each dimension separately?”</p>
<p>But, taking your question at face value, note that <code>loss.backward()</code><br>
<em>accumulates</em> the gradient of <code>loss</code> into the various <code>Parameter.grad</code>s.</p>
<p>So, for example:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_array[0].backward()   # possibly with retain_grad = True
loss_array[1].backward()
loss_array[2].backward()
</code></pre>
<p>gives the same final result for the relevant <code>.grad</code> values as does:</p>
<pre data-code-wrap="python"><code class="lang-python">(loss_array[0] + loss_array[1] + loss_array[2]).backward()
</code></pre>
<p>and the latter will be more efficient.</p>
<p>Best.</p>
<p>K. Frank</p>

username: nil_tdr
cooked: <p>Hi K. Frank,</p>
<p>Thanks for your response and sorry for my late reply. My loss is an array of dimension 50. For PyTorch I know that loss has to be a scalar value. Since my loss is an array I can’t use default backward() function. This is what I tried.</p>
<blockquote>
<p>optim.zero_grad()<br>
for grad in gradient: <span class="hashtag-raw">#gradient</span> is a 50 dimensional array<br>
loss = grad<br>
loss.backward()<br>
optim.step()</p>
</blockquote>
<p>This is the error I got:<br>
RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.</p>
<p>I would really appreciate if you can give a direction where should I look at to make this special case work?</p>
<p>Thanks,<br>
Nil</p>

username: KFrank
cooked: <p>Hi Nil!</p>
<aside class="quote no-group quote-modified" data-username="nil_tdr" data-post="3" data-topic="185360" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<blockquote>
<p>optim.zero_grad()<br>
for grad in gradient: <span class="hashtag-raw">#gradient</span> is a 50 dimensional array<br>
loss = grad<br>
loss.backward()<br>
optim.step()</p>
</blockquote>
<p>This is the error I got:<br>
RuntimeError: Trying to backward through the graph a second time</p>
</blockquote>
</aside>
<p>Presumably elements of your <code>gradient</code> array share parts of the same<br>
<em>computation graph.</em>  That is, the computation of <code>gradient[0]</code> and, for<br>
example, <code>gradient[1]</code> partially overlap.  Calling <code>gradient[0].backward()</code><br>
deletes <code>gradient[0]</code>'s computation graph, including any parts of it that<br>
are shared by <code>gradient[1]</code>'s computation graph.  So when you then call<br>
<code>gradient[1].backward()</code>, parts of <code>gradient[1]</code>'s computation graph<br>
have been deleted, leading to the error you report.</p>
<p><code>retain_graph = True</code> tells autograd not to delete the computation graph,<br>
so you could do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">optim.zero_grad()
for  i, grad in enumerate (gradient):   # gradient is a 50 dimensional array
    loss = grad
    if  i &lt; 49:
        loss.backward (retain_graph = True)
    else:
        loss.backward()
optim.step()
</code></pre>
<p>(The final call to <code>loss.backward()</code> does not have <code>retain_graph = True</code><br>
because you do need to delete computation graph at some point, typically<br>
before calling `optim.step() and / or performing the next forward pass.)</p>
<p>This is a perfectly reasonable way to use autograd and <code>.backward()</code>.<br>
However, it’s likely to be inefficient, because you repeat (the shared part<br>
of) the backward pass fifty times.</p>
<p><code>loss.backward()</code> computes the gradient of <code>loss</code> with respect to the<br>
parameters on which <code>loss</code> depends and accumulates that gradient into<br>
those parameters’ <code>.grad</code> properties.  But computing the gradient is a linear<br>
operation (so that <code>grad_of (a + b) = grad_of (a) + grad_of (b)</code>).</p>
<p>So you are likely better off with:</p>
<pre data-code-wrap="python"><code class="lang-python">optim.zero_grad()
loss_total = 0
for  grad in gradient:
    loss_total = loss_total + grad
loss_total.backward()
optim.step()
</code></pre>
<p>This only performs a single backward pass (rather than fifty) and, up to<br>
numerical round-off error, computes the same final gradient (as stored in<br>
the various parameters’ <code>.grad</code> properties) as does the version that called<br>
<code>.backward()</code> fifty times.</p>
<p>As an aside, you will probably also achieve additional efficiency (and code<br>
cleanliness) if you can arrange your computation so that <code>gradient</code> is a<br>
single one-dimensional pytorch tensor of length fifty that is computed all at<br>
once with pytorch tensor operations rather than an array of fifty length-one<br>
pytorch tensors that is computed entry by entry.</p>
<p>Best.</p>
<p>K. Frank</p>

username: nil_tdr
cooked: <p>Thank you so much Mr. Frank. This is really helpful. I am working on it and will update what I find.<br>
I found a mistake in my problem formulation. The gradient I mentioned is a gradient of loss not the loss. Is there a way to directly use gradient (which is an array of dimension 50) using Pytorch?</p>

username: KFrank
cooked: <p>Hi Nil!</p>
<aside class="quote no-group" data-username="nil_tdr" data-post="5" data-topic="185360" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nil_tdr/48/65323_2.png" class="avatar"> nil_tdr:</div>
<blockquote>
<p>The gradient I mentioned is a gradient of loss not the loss. Is there a way to directly use gradient (which is an array of dimension 50) using Pytorch?</p>
</blockquote>
</aside>
<p>I don’t know what your use case is and I don’t follow what you are asking.</p>
<p>If you want to compute the gradient of a gradient (so, a second derivative),<br>
you want, more or less, the so-called <em>hessian.</em></p>
<p>Autograd supports computing gradients of gradients.  You might start by<br>
taking a look at <a href="https://pytorch.org/docs/2.0/generated/torch.autograd.functional.hessian.html" rel="noopener nofollow ugc">hessian().</a></p>
<p>Best.</p>
<p>K. Frank</p>