Custom loss calculation without breaking the computation graph

yildirimyigit: <p>I am quite new to PyTorch and the community, so please bear with me on this.</p>
<p>I want to have a neural network propose rewards for the states in a reinforcement learning task. After running my RL algorithm using these proposed rewards, I expect to have an estimated policy.</p>
<p>I want the loss of the network to be the distance between the estimated policy and the actual policy, which I already have. (The objective here is to learn the rewards, not the policy.)</p>
<hr>
<p>I implemented my algorithm using <em>numpy</em> but now I’m trying to incorporate <em>PyTorch</em> and it looks desperate. Feeding states to obtain reward propositions was pretty straightforward, I did a simple forward pass. But upon receiving the rewards, I had to use <code>detach()</code> to get these values to calculate the loss and now, I cannot backpropagate since I have broken the computation graph when I use <code>detach()</code>. This is my problem.</p>
<p>Now, I only see two possibilities: 1) move everything to pytorch and abandon <em>numpy</em> 2) try to come up with a hybrid solution.</p>
<p>My questions are:</p>
<ul>
<li>Is option 2 doable using PyTorch?
<ul>
<li>
<strong>If so</strong>, what do I need to do to have the gradients calculated automatically?</li>
<li>
<strong>And if not</strong>, is option 1 doable using PyTorch? Do I have to move all my code to PyTorch?</li>
</ul>
</li>
</ul>
<p>Many thanks for taking your time.</p>

KFrank: <p>Hi Yigit!</p>
<aside class="quote no-group quote-modified" data-username="yildirimyigit" data-post="1" data-topic="64346" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yildirimyigit/48/18866_2.png" class="avatar"> yildirimyigit:</div>
<blockquote>
<p>I did a simple forward pass. But upon receiving the rewards, I had to use <code>detach()</code> to get these values to calculate the loss and now, I cannot backpropagate since I have broken the computation graph when I use <code>detach()</code>.</p>
<p>Now, I only see two possibilities: 1) move everything to pytorch and abandon <em>numpy</em></p>
</blockquote>
</aside>
<p>If you move your loss function from pytorch to numpy (and use<br>
differentiable pytorch tensor functions to calculate your loss),<br>
pytorch’s autograd will calculate the gradients for you when you<br>
run <code>loss.backward()</code>.  (You don’t have to move everything<br>
to pytorch, but you do have to move the entire loss function to<br>
pytorch.)</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>try to come up with a hybrid solution.</li>
</ol>
</blockquote>
</aside>
<p>If by “hybrid” you mean that you want to keep your loss function<br>
in numpy, then you will have to write an explicit <code>backward()</code><br>
function (that can be in numpy, if you choose) for your loss<br>
function that calculates its gradients.  Once you use something<br>
other than pytorch tensor functions, you won’t be able to get<br>
the gradients / <code>backward()</code> done automatically for you.</p>
<p>When using a custom loss function, I’ve generally preferred to<br>
try to implement it with tensor functions to get autograd “for free.”<br>
Plus you will get pytorch’s cpu / gpu management, and possibly<br>
useful speed-ups from running both your loss function’s<br>
<code>forward()</code> and <code>backward()</code> on the gpu, if you so choose.</p>
<p>I do believe that you will have to pay one way or another:  Either<br>
port your numpy loss to pytorch; or write <code>backward()</code> explicitly.</p>
<p>Good luck.</p>
<p>K. Frank</p>

yildirimyigit: <p>Hello again. First of all, thank you for the contribution <a class="mention" href="/u/kfrank">@KFrank</a>.</p>
<p>I didn’t reply <a class="mention" href="/u/kfrank">@KFrank</a> 's answer immediately since I knew that it would take me some time to implement all of that. I think I have finished the code and guess what? It is not working <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">  I do understand the reason but I don’t know the solution. So here I am once more.</p>
<hr>
<p>This time I’m gonna use a drawing which I hope will explain the condition far more easily. I’m trying to do Inverse RL:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/9/492a6f5f1abdd1c14bcb6e3fea78b5464c9cd49b.png" data-download-href="https://discuss.pytorch.org/uploads/default/492a6f5f1abdd1c14bcb6e3fea78b5464c9cd49b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/9/492a6f5f1abdd1c14bcb6e3fea78b5464c9cd49b.png" alt="image" data-base62-sha1="arfLRAWl6QKTw6Nm8kWjGFwudir" width="690" height="401" data-dominant-color="F5F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">705×410 28.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I don’t have the actual rewards(<strong>R</strong>). That’s why, I cannot simply use <code>mse_loss(R, R_est)</code>. I calculate the distance between π and π_est somehow and want to use this value as the error signal for the backpropagation.</p>
<p>I believe the following 2 lines sum up my issue:</p>
<pre><code class="lang-auto">loss = torch.nn.functional.mse_loss(emp_fc, esvc)
loss.backward()
</code></pre>
<p>I get this error: <code>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.DoubleTensor []], which is output 0 of SelectBackward, is at version 200; expected version 199 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!</code></p>
<p>Can anybody help me solve this problem? I don’t know how can I avoid in-place operations while I have soooo many of them. Nevertheless, I have to backpropagate that value.</p>

Mayank_Jha: <p>Hi, I have a  similar problem, can you indicate if you got the solution or link to where the solution can be inspired?<br>
thanks for the help.<br>
regards, Mayank</p>