How to backprop a loss with a partial derivative on it (behind .backward())

anonymous007: <p>Hello, I want to understand how .backward() handles this. I know that I can just type loss.backward() and get the correct result by magic  but I want to understand it. This is my case:</p>
<p>In PINNs you have additional loss functions, for example one for the boundary condition and the other for the PDE… Let’s focus on the PDE one.</p>
<p>Let’s  say that my input is X=(x,t), so the output is Y after a forward propagation.<br>
The conventional approach in backprop is as follow:</p>
<pre><code>Loss = (Y -Yreal)**2
</code></pre>
<p>and then backpropagation goes as</p>
<pre><code>(dLoss/dY)(dY/dZ)(..)....(..) 
</code></pre>
<p>in PyTorch I just need Loss.backward() to achieve the same result</p>
<p>Now, let’s consider the case where we enforce a differential equation, let’s say the PDE, for simplicity, is:</p>
<pre><code>dy/dt + y**2 + c = 0
</code></pre>
<p>To define this PDE I need dy/dt, so I use my network to get this value (backward propagation focused on dy/dt, also it can easily be obtained using .backward() ).</p>
<p>Then I set my loss function as</p>
<pre><code>L = (dy/dt + y**2 + c)**2....(1)
</code></pre>
<p>and my cost is the summation over all samples.</p>
<p>Now comes the question… How do I backpropagate to update my gradients?<br>
In PyTorch i just use .backward() but what is happening in this case?<br>
As before, I would start with dLoss/dY, which is dL/dy, then considering (1) I have:</p>
<pre><code>dL/dy = d/dy( (dy/dt + y**2 + c)**2 )
      = 2(dy/dt + y**2 + c)*(d/dy(dy/dt)??? + 2y).......(2)
</code></pre>
<p>Once dL/dy is obtained, what follows is a chain rule of differentiations, just like a conventional set up, so no problem there, the problem is in (2), how do I set: “dL/dy”… or my approach is incorrect? PyTorch computes Eq. (2) ??? but there is this term with “???” that seems odd to me. What is wrong in my reasoning? Thanks for your comment</p>

tom: <aside class="quote no-group" data-username="anonymous007" data-post="1" data-topic="152160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anonymous007/48/47508_2.png" class="avatar"> anonymous007:</div>
<blockquote>
<p>To define this PDE I need dy/dt, so I use my network to get this value (backward propagation focused on dy/dt, also it can easily be obtained using .backward() ).</p>
</blockquote>
</aside>
<p>I think you would have an easier time using <code>torch.autograd.grad</code> here (with <code>create_graph</code> here). Personally, I like to think of that as the standard way to get a derivative and <code>.backward</code> as as convenience method for when you need gradients of loss functions for an optimizer.</p>
<p>Best regards</p>
<p>Thomas</p>

anonymous007: <p>Thanks for your reply. I will consider that, but my biggest concern is how to solve Eq. (2)…or at least to understand how backprop handles the first derivative  dL/dy with a PDE. Some help is appreciated…</p>
<p>I think the way it goes is as follow:</p>
<p>dL/dy = 2(dy/dt + y**2 + c)*(2y)…(2)</p>
<p>Previously I wanted to derive also dy/dt, but this is not needed because I want to compute the loss between this factor, dy/dt, taken as the real value, and the other two expressions (taken as the network output)… it is a bit odd though because dy/dt was obtained with autograd.grad and is not a real value…but anyways if this the correct way a confirmation will be appreciated</p>