Use max operation in loss function

netaglazer: <p>hi, im very confused.<br>
from all i know, mathematicly, max operation is not differential<br>
i read here a lot of contrasting answers -<br>
some people are saying that it is not differential, and therefor cant be used in a loss function<br>
and some are saying that it is differential “enough” for backpropagation</p>
<p>does someone have an answer and a <strong>good</strong> explanation about that issue?</p>

KFrank: <p>Hello Neta!</p>
<aside class="quote no-group" data-username="netaglazer" data-post="1" data-topic="67501" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/f1d935/48.png" class="avatar"> netaglazer:</div>
<blockquote>
<p>from all i know, mathematicly, max operation is not differential<br>
i read here a lot of contrasting answers -<br>
some people are saying that it is not differential, and therefor cant be used in a loss function<br>
and some are saying that it is differential “enough” for backpropagation</p>
</blockquote>
</aside>
<p>Consider this one-dimensional (single-variable) function that<br>
uses <code>max</code>:</p>
<p><code>f (x) = max (x, 0)</code></p>
<p>This function is differentiable for all values of <code>x</code> except when<br>
<code>x = 0</code>.  It is not differentiable exactly at <code>x = 0</code>, but the function<br>
isn’t crazy.  You could choose to <em>define</em> (not mathematically<br>
correctly, though) to be <code>0</code> or <code>1</code> or <code>1/2</code> when <code>x = 0</code>, and for<br>
practical purposes, for example, for back-propagation, do a<br>
perfectly reasonable job. Most of the time you won’t be<br>
back-propagating exactly through <code>x = 0</code>, and even if you do,<br>
you probably won’t do so again on the next iteration.</p>
<p>Similarly, consider this function of two variables:</p>
<p><code>f (x, y) = max (x, y)</code></p>
<p>This function is differentiable (that is the gradients – the partial<br>
derivatives with respect to <code>x</code> and <code>y</code>– are well defined) for all<br>
values of <code>x</code> and <code>y</code> except along the line where <code>x = y</code>.</p>
<p>The same reasoning applies here.  You usually won’t try to<br>
back-propagate through a point where <code>x = y</code>, and even if<br>
you do, using <code>0</code> or <code>1</code> or whatever for the partial derivative in<br>
question will be good enough.</p>
<p>In practice we know – and lots of experience proves – that this<br>
works.</p>
<p>(Now if autograd returned NaN or 10,000,000 or something<br>
when you hit one of the rare points where <code>max</code> is not technically<br>
differentiable, your training would likely break.  But autograd<br>
uses some reasonable value like <code>0</code> or <code>1</code> or <code>1/2</code>, and everything<br>
works fine.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

Minqi_Jiang: <p>Actually the max (x,0) is like the relu activation function, which is differential.</p>

ado_sar: <p>Does it uses <code>1/2</code>?</p>
<p>With the following snippet:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([2., 1., 2., 0.], requires_grad=True)
out = torch.max(x)
out.backward()
</code></pre>
<p>I get:</p>
<pre data-code-wrap="python"><code class="lang-python">x.grad
tensor([0.5000, 0.0000, 0.5000, 0.0000])
</code></pre>