Creating a custom loss function with the c++ api

username: vogec
cooked: <p>Hello all.<br>
I am in particular new to the c++ API interface.<br>
Did anybody manage to create his own, custom, loss function with the API?<br>
Are there predefined regularizers (l2 or similar, better Lipshitz) available? How are those used ?<br>
I could not find any examples on that via google – maybe I overlooked something.</p>
<p>I am happy for any advice,<br>
thanks guys.</p>

username: goksug
cooked: <p>Hi there,</p>
<p>Have you found out a source for that?</p>

username: mhubii
cooked: <p>As far as my understanding goes, a loss function basically behaves just as a layer does. Therefore, a custom loss function should be implementable by following Peter’s guide on how to write a custom layer <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a></p>
<p>It is necessary that you define the forward as well as the backward pass of that loss function, which means that you need to find the gradient of your custom loss function for the backward pass. However, I am not entirely sure if that is true. Can someone copy that?</p>

username: tom
cooked: <p>So there are two cases regarding the autograd graph:</p>
<ul>
<li>If your operations to compute the loss are torch functions, the autograd engine can compute the backward just like in Python.</li>
<li>If, on the other hand, you implement custom computation kernels (as Peter does in the tutorial), you need to provide a backward. The old-style thing (shown in the tutorial) to do was to export both forward and backward and use an autograd.Function. These days, you would probably create a autograd::Function in C++, <a href="https://github.com/pytorch/vision/blob/b4cb5765dcd93cf09a05e26fbebb344013d642f1/torchvision/csrc/ROIPool.h#L74" rel="nofollow noopener">similar to what we do in TorchVision</a>.</li>
</ul>
<p>Another thing to keep in mind is that if you want JIT compatibility, you need to register operators (<a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="nofollow noopener">as in the custom op tutorial</a>). Unless you need funny data types to be passed to from Python, there isn’t really a downside to using these.</p>
<p>Best regards</p>
<p>Thomas</p>

username: mhubii
cooked: <p>thanks Thomas for these valuable insights. Just as a general question, detectron2 got implemented as Peter explained in his tutorial on cpp extensions, <a href="https://github.com/facebookresearch/detectron2/tree/master/detectron2/layers" rel="nofollow noopener">https://github.com/facebookresearch/detectron2/tree/master/detectron2/layers</a></p>
<p>Is there any advantage of the old way over the autograd::Function implementation? Implementing it via the autograd::Function in cpp is JIT compatible as well, right?</p>

username: tom
cooked: <p>So the new way is just very new (we added this to TorchVision in September). They probably branched off the maskrcnn-benchmark or TorchVision earlier than that and didn’t get the update.<br>
Previously, you had the problem that “JIT means CustomOp means implemented in C++” and that meant no autograd with <a href="https://lernapparat.de/pytorch-traceable-differentiable/" rel="nofollow noopener">jumping through lots of hoops</a>. In August or so, we gained C++ autograd::Function, so now you can easily have CustomOps that record a custom backward.</p>
<p>This still isn’t the “ultimate JIT-friendly thing”, which would be <a href="https://github.com/pytorch/pytorch/pull/22582" rel="nofollow noopener">more like graph-to-graph differentiation</a>, that sadly I didn’t get to finish up yet (I’ll see if <a href="https://github.com/sponsors/t-vi" rel="nofollow noopener">adding a voting mechanism for prioritizing my PyTorch work</a> helps…).</p>
<p>Best regards</p>
<p>Thomas</p>