How to ensure BP in custom loss function

Michael_Hsi: <p>Hello all, thanks in advance for your time.</p>
<p>I am writing a custom loss function. There are 8 methods that I use in the loss function. My network is for image segmentation, and outputs logits.</p>
<p>the custom loss function uses:</p>
<p>torch.nn.Softmax()<br>
torch.max()<br>
torch.eq()<br>
torch.logical_or()<br>
torch_tensor.bool().int()<br>
torch_tensor.sum()<br>
torch_scaler.abs()</p>
<p>However when I do loss.backward(), the network says: ‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’</p>
<p>I understand that torch.eq() will break the gradient, hence question 1 is ‘how can I circumvent this issue’</p>
<p>I need to count the occurrences of certain class in the torch tensor, hence Im not sure how I would avoid using torch.eq()</p>
<p>and secondly, does torch_tensor.bool().int() break gradient as well? If so, would switching to torch.gt(torch_logits, 1).sum() resolve the broken gradient?</p>

ptrblck: <aside class="quote no-group" data-username="Michael_Hsi" data-post="1" data-topic="130904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_hsi/48/30126_2.png" class="avatar"> Michael_Hsi:</div>
<blockquote>
<p>I understand that torch.eq() will break the gradient, hence question 1 is ‘how can I circumvent this issue’</p>
</blockquote>
</aside>
<p>Yes, <code>torch.eq</code> will return a <code>BoolTensor</code> and will thus detach the output from the computation graph. Unfortunately, I don’t know how to avoid this besides trying to com up with a custom backward function, i.e. you would have to define gradients for the equality/inequality of the values.</p>
<aside class="quote no-group" data-username="Michael_Hsi" data-post="1" data-topic="130904">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_hsi/48/30126_2.png" class="avatar"> Michael_Hsi:</div>
<blockquote>
<p>and secondly, does torch_tensor.bool().int() break gradient as well?</p>
</blockquote>
</aside>
<p>Yes, as gradients are defined for floating point types.<br>
Generally, you could check if an operation detached the result by printing the <code>.grad_fn</code> of the inputs and outputs of a function. If the <code>.grad_fn</code> shows <code>None</code>, the tensor is detached.</p>

Michael_Hsi: <p>If I were to custom define gradient for eq and not eq, along with logical_or(), logical_gt(), how would I do that? can you point me to some example/tutorial for this? Im fairly new to pytorch.</p>
<p>I am thinking just to pass along the gradient if equal, and reduce gradient to 0 if not equal.<br>
same with gt, if value is greater than cmpared_value, pass along the gradient; otherwise reduce to 0<br>
and for logical_or(), I’ll add the gradient together, e.g. True + True</p>
<p>Thanks.</p>

ptrblck: <p>Sure, <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">this tutorial</a> shows how to implement custom <code>autograd.Function</code>s.</p>