Custom Loss with detach/re-attach

welahi: <p>Hello,</p>
<p>I need to implement some kind of a “generalized MSE” loss, where I don’t just consider <code>loss_mse(target, model)</code>, but <code>loss_mse(O(target), O(model))</code>, where <code>O</code> is a rather complicated function.</p>
<p>The problem is that <code>O</code> detaches the PyTorch-tensors as numpy, calls some different thrid-party libraries to calculate the result and returns again an array. Obviously, I can convert the resulting arrays again into tensors, but this won’t help, because I once detached them and the gradient can not be calculated anymore.</p>
<p>I am fairly new to PyTorch, so please tell me if my post needs further clarification. Does anyone know where to start? i would also be happy with a link to the right documentation page or tutorial, but I had a hard time searching myself <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p>Thanks and best</p>

KFrank: <p>Hi Welahi!</p>
<aside class="quote no-group" data-username="welahi" data-post="1" data-topic="147161" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/d6d6ee/48.png" class="avatar"> welahi:</div>
<blockquote>
<p>once detached them and the gradient can not be calculated anymore.</p>
</blockquote>
</aside>
<p>The best approach – if it’s doable – would be to rewrite your custom<br>
loss <code>O</code> using (differentiable) pytorch tensor operations.  Then you will<br>
get autograd “for free.”</p>
<p>If this is not practical – maybe the stuff in your third-party libraries is<br>
too complicated – you will have to write a <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html" rel="noopener nofollow ugc">custom autograd Function</a>.</p>
<p>The <code>forward()</code> method of your <code>Function</code> can just be your current<br>
<code>O</code> (returning pytorch tensors, of course).  But for the <code>backward()</code><br>
method, you will have to work out the derivative (gradient) of <code>O</code> (or<br>
at least a good-enough approximation to it) and implement it.  Such<br>
an implementation can use numpy and your third-party libraries, if<br>
that helps (but has to be packaged to accept and return pytorch<br>
tensors, of course).</p>
<p>This is all conceptually straightforward, but can become quite difficult<br>
and nuanced, depending on how complicated <code>O</code> and its derivative are.</p>
<p>Good luck.</p>
<p>K. Frank</p>

welahi: <p>Hi K. Frank,</p>
<p>thanks for your answer and the link! I’ve played around with the example and implemented another toy problem and I think I grasped now how it works.</p>
<p>Unfortunatelly, it is near-impossible for my actual to find any kind of “closed expression” for the derivative, so I’m probably going to perform a numerical derivation using the forward-operation itself (basically <code>(O(x+h)-O(x-h))/(2h)</code> for some small <code>h</code>). It’s far from perfect but I don’t see any other way <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
<p>Best</p>

KFrank: <p>Hi Welahi!</p>
<aside class="quote no-group quote-modified" data-username="welahi" data-post="3" data-topic="147161" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/d6d6ee/48.png" class="avatar"> welahi:</div>
<blockquote>
<p>I’m probably going to perform a numerical derivation using the forward-operation itself (basically <code>(O(x+h)-O(x-h))/(2h)</code> for some small <code>h</code>).]</p>
</blockquote>
</aside>
<p>Calculating the gradient numerically is a perfectly reasonable thing<br>
to do, although it can become expensive in the multi-dimensional<br>
case.</p>
<p>Note that the <code>forward()</code> method of your custom <code>Function</code> gets passed<br>
a <em>context</em> object, <code>ctx</code>, that it can use to stash away useful information<br>
that can be helpful to your custom <code>Function</code>’s <code>backward()</code> method.<br>
Pytorch’s autograd machinery hands <code>ctx</code> back to your <code>backward()</code><br>
method during the backward pass.</p>
<p>It may be the case (or maybe not) that some (or maybe all) of the<br>
heavy lifting needed to perform the numerical differentiation can be<br>
carried out more cheaply or conveniently during the forward pass.</p>
<p>You should look for opportunities to reuse parts of the computations<br>
performed during the forward pass if they show up again in the<br>
backward pass and use the <code>ctx</code> object as the mechanism to do this<br>
(at some cost in memory).</p>
<p>Best.</p>
<p>K. Frank</p>

welahi: <p>Hi K. Frank!</p>
<p>Ah yes, but unfortunately I don’t see any possibility to make use of <code>ctx</code> in my case here. Except if I would use the forward difference <code>(O(x+h)-O(x))/h</code> instead of the central difference, since I could then re-use <code>O(x)</code>, but I’m willing to sacrifice the time gain for better accuracy. At least for now <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>Anyway, it seems to run now. Although it was really a pain to figure out if PyTorch expects which of <code>[]</code>-, <code>[1]</code>- or <code>[1,1]</code>-tensor under what circumstances…</p>
<p>Best</p>