How can custom loss function be backpropagated

Hwarang_Kim: <p>I built my loss function using conditional statement like:</p>
<pre><code class="lang-auto">def myloss(data):
    if blah blah:
        loss = blah
    if blah blah:
        loss = blah
    return loss
loss = myloss(output)
loss.backward()
</code></pre>
<p>I worried it won’t work but it worked.<br>
but how can my loss function be backpropagated?<br>
Is my loss function differentiable?</p>
<p>Thank you for helping me in advance.</p>

KFrank: <p>Hello Hwarang!</p>
<aside class="quote no-group quote-modified" data-username="Hwarang_Kim" data-post="1" data-topic="90501" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hwarang_kim/48/12593_2.png" class="avatar"> Hwarang_Kim:</div>
<blockquote>
<pre><code class="lang-auto">def myloss(data):
    if blah blah:
        loss = blah
    if blah blah:
        loss = blah
    return loss
loss = myloss(output)
loss.backward()
</code></pre>
<p>but how can my loss function be backpropagated?<br>
Is my loss function differentiable?</p>
</blockquote>
</aside>
<p>In short, the conditional statement doesn’t break anything.</p>
<p>If <code>loss</code> inside of your <code>myloss()</code> function is calculated with pytorch<br>
tensor operations (that have <code>backward()</code> implemented and are<br>
differentiable), backpropagation through <code>myloss()</code> will work just fine.</p>
<p>So, to be concrete, let:</p>
<pre data-code-wrap="python"><code class="lang-python">def myloss (data):
    if data[0][0] &gt; 5.0:
        loss = 1.0 * (data**2).sum()
    else:
        loss = 2.0 * (data**3).sum()
    return loss
</code></pre>
<p>Mathematically speaking, <code>myloss()</code> will be differentiable everywhere<br>
except at <code>data[0][0] = 5.0</code>, which is good enough.</p>
<p>In practice, if <code>data[0][0] = 5.0</code>, <code>myloss()</code> will take the second<br>
branch and <code>loss.backward()</code> will calculate the gradient that<br>
corresponds to <code>loss = 2.0 * (data**3).sum()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

Hwarang_Kim: <p>I`m sorry but i think that i asked wrong question which is not my intention.<br>
Actually, my loss function is constructed like:</p>
<pre><code class="lang-auto">def myloss(data):
    tmp = 0
    for i in range(len(data)):
        if data[i] &gt; 0.5:
            tmp += 1
        else:
            tmp += 2
    loss = math.log10(tmp)
    return loss
loss = myloss(output)
loss.backward()
</code></pre>
<p>In this case, does it work too?</p>

KFrank: <p>Hello Hwarang!</p>
<aside class="quote no-group" data-username="Hwarang_Kim" data-post="4" data-topic="90501">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hwarang_kim/48/12593_2.png" class="avatar"> Hwarang_Kim:</div>
<blockquote>
<p>In this case, does it work too?</p>
</blockquote>
</aside>
<p>No, this won’t work.  The problem is that this version of <code>myloss()</code> isn’t<br>
<em>usefully</em> differentiable.  It is constant almost everywhere, so the gradient<br>
will always be zero.</p>
<p>Mathematically, <code>myloss()</code> is differentiable (with zero gradient) except<br>
when any of the <code>data[i] = 0.5</code>, at which values <code>myloss()</code> jumps<br>
discontinuously and the derivative is not defined.</p>
<p>Numerically with pytorch you will always get zero gradient, even when<br>
some <code>data[i] = 0.5</code>, because whatever branch of the conditional<br>
you go through, a constant function (constant for that branch) is being<br>
calculated.</p>
<p><code>myloss()</code> and backpropagation will “work” in the sense that calling<br>
<code>loss.backward()</code> will give you a well-defined gradient, but it doesn’t<br>
actually do you any good because the gradient is always zero.</p>
<p>In practical terms, let’s say that <code>data[3] = 0.5001</code>, and you get some<br>
value of the loss function.  Let’s also say that at <code>data[3] = 0.5</code> the<br>
loss function jumps to a lower, more favorable value so that you would<br>
like your optimizer step to update <code>data[3]</code> to <code>0.4999</code>.  The problem is<br>
that the optimizer only knows about the gradient, which is zero, and<br>
doesn’t know that very nearby at <code>0.4999</code> you get a lower loss.  With zero<br>
gradient the optimizer doesn’t (and can’t) know in which direction to vary<br>
<code>data[3]</code>, that is, whether to increase, decrease, or leave unchanged<br>
<code>data[3]</code>, to get to a lower loss.</p>
<p>This is how gradient-descent optimization methods (which are the core<br>
of pytorch’s backpropagation) work, and it’s an inherent limitation they<br>
have.</p>
<p>Best.</p>
<p>K. Frank</p>

Hwarang_Kim: <p>Thanks! I totally understand for your kind answer! thanks again!</p>