Custom loss function, what's legal in forward pass

username: Aoo_Lmay
cooked: <p>I’m new to Pytorch, i’m having trouble properly implementing custom loss function that will be compatible with backward() pass. For some trivial experiments it’s ok, for others i’m getting a “element 0 of tensors does not require grad and does not have a grad_fn” error.<br>
I did check requires_grad on arguments for following examples, x True, y False, p False.</p>
<p>class cu_loss(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(cu_loss,self).<strong>init</strong>()<br>
<span class="hashtag">#1</span>  def forward(self,x,y):<br>
da = y - torch.argmax(x, dim=1)<br>
result = {…}<br>
return result<br>
<span class="hashtag">#2</span>  def forward(self,x,y,p):<br>
diff = torch.sum(y-p)<br>
return diff</p>
<p><span class="hashtag">#1</span> forward fails simply after addition of torch.argmax function<br>
<span class="hashtag">#2</span> forward i have no idea, am i not allowed to simply substract tensors with minus sign? This seems to be a common problem.</p>
<p>I read from similar topics that i can’t use functions outside of pytorch, because some data can escape the internal state of the graph? Is there a resource with general pointers how to properly do that?</p>

username: Aoo_Lmay
cooked: <p>After further research i became aware that only FloatTensors can become part of the gradient. Now after type conversion i can perform any operations in forward.<br>
Even though i jumped this hurdle, i welcome any further insights.</p>

username: ptrblck
cooked: <p>You can use any PyTorch operations in the forward pass, which have a valid backward function.<br>
While the majority of the operations meet this condition, e.g. <code>torch.argmax</code> will detach the output from the computation graph, as this operation is not differentiable.</p>

username: Aoo_Lmay
cooked: <p>How do i determine if a function is differentiable at a glance? Is there an argmax analog that could be employed? Since i figured out TensorFloat is necessary as a result in forward and started experimenting, different experiments produce different anomalies and i’m yet to learn the pytorch ways.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Aoo_Lmay" data-post="4" data-topic="99817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aoo_lmay/48/29984_2.png" class="avatar"> Aoo_Lmay:</div>
<blockquote>
<p>How do i determine if a function is differentiable at a glance?</p>
</blockquote>
</aside>
<p>You can pass an input tensor with <code>requires_grad=True</code> to a particular operation and check, if the output tensor has a valid <code>.grad_fn</code> as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">
x = torch.randn(10, 10, requires_grad=True)
out = torch.argmax(x, dim=1)
print(out.grad_fn)
&gt; None

val, idx = torch.max(x, dim=1)
print(val.grad_fn)
&gt; &lt;MaxBackward0 object at ...&gt;
print(idx.grad_fn)
&gt; None
</code></pre>
<p>Also, it’s often helpful to “draw” the function and think how the gradient could look like.</p>
<aside class="quote no-group" data-username="Aoo_Lmay" data-post="4" data-topic="99817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aoo_lmay/48/29984_2.png" class="avatar"> Aoo_Lmay:</div>
<blockquote>
<p>Is there an argmax analog that could be employed?</p>
</blockquote>
</aside>
<p>Yes, <code>softmax</code> (sometimes also called softargmax) is the smooth version of <code>argmax</code>, which provides valid gradients.</p>

username: Aoo_Lmay
cooked: <p>Thank you for the tips. Plotting the values certainly provides some insight. Enabling custom losses already allowed my model to reach ~20% accuracy, that’s over ~6% of what vanilla losses offered.<br>
Cheers!</p>

username: KFrank
cooked: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> (and Aoo)!</p>
<aside class="quote no-group quote-modified" data-username="ptrblck" data-post="5" data-topic="99817" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<aside class="quote no-group" data-username="Aoo_Lmay" data-post="4" data-topic="99817">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aoo_lmay/48/29984_2.png" class="avatar"> Aoo_Lmay:</div>
<blockquote>
<p>Is there an argmax analog that could be employed?</p>
</blockquote>
</aside>
<p>Yes, <code>softmax</code> (sometimes also called softargmax) is the smooth version of <code>argmax</code>, which provides valid gradients.</p>
</blockquote>
</aside>
<p>As a minor technical clarification, <code>softmax()</code> is a smooth version of<br>
the <em><code>one_hot()</code> encoding</em> of the result of <code>argmax()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>