Why are gradients not zero at global minimum?

jjhh: <p>I made a function to compute the IoU loss of 2 spans</p>
<pre><code class="lang-auto">def iou_loss(pred_start, pred_end, target_start, target_end):
    """
    pred_start, pred_end, target_start, target_end: b, *
    """

    num_common = (
        torch.minimum(pred_end, target_end)
        - torch.maximum(pred_start, target_start) + 1
    ).clamp_min_(0)

    num_pred = (pred_end - pred_start + 1).clamp_min_(0)
    num_target = target_end - target_start + 1

    iou = num_common / (num_pred + num_target - num_common)

    return 1 - iou
</code></pre>
<p>However the gradients are not zero when the loss is at its global minimum.</p>
<pre><code class="lang-auto">pred_start = torch.tensor([5, 2], dtype=torch.float, requires_grad=True)
pred_end = torch.tensor([5, 4], dtype=torch.float, requires_grad=True)
target_start = torch.tensor([5, 2])
target_end = torch.tensor([5, 4])

l = iou_loss(pred_start, pred_end, target_start, target_end)
print(l)
# tensor([0., 0.], grad_fn=&lt;RsubBackward1&gt;)

l = l.mean()
l.backward()

print(pred_start.grad)
# tensor([-0.5000, -0.1667])

print(pred_end.grad)
# tensor([0.5000, 0.1667])
</code></pre>
<p>Can anyone help me to understand why? Any help would be appreciated.</p>

KFrank: <p>Hi jjhh!</p>
<aside class="quote no-group quote-modified" data-username="jjhh" data-post="1" data-topic="109844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jjhh/48/13058_2.png" class="avatar"> jjhh:</div>
<blockquote>
<pre><code class="lang-auto">    num_common = (
        torch.minimum(pred_end, target_end)
        - torch.maximum(pred_start, target_start) + 1
    ).clamp_min_(0)
</code></pre>
<p>However the gradients are not zero when the loss is at its global minimum.</p>
</blockquote>
</aside>
<p>The short story is that <code>torch.minimum()</code> and <code>torch.maximum()</code> are not<br>
differentiable at the special points where the arguments are equal.</p>
<p>The function <code>abs (x)</code> offers a simpler example.  When <code>x &gt; 0</code>, the derivative<br>
is <code>+1</code>, while when <code>x &lt; 0</code>, it’s <code>-1</code>.  When <code>x = 0</code>, mathematically speaking,<br>
the derivative is not defined.  (One could try to play some game where you<br>
define the derivative to be <code>0</code> when <code>x = 0</code> – and pytorch does do this for the<br>
<code>abs()</code> function – but doing so is problematic, and  not really worth the bother.)</p>
<p>Here is an example script that probes your loss function with values where<br>
the arguments are close, but not exactly equal:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

def iou_loss(pred_start, pred_end, target_start, target_end):
    """
    pred_start, pred_end, target_start, target_end: b, *
    """

    num_common = (
        torch.minimum(pred_end, target_end)
        - torch.maximum(pred_start, target_start) + 1
    ).clamp_min_(0)

    num_pred = (pred_end - pred_start + 1).clamp_min_(0)
    num_target = target_end - target_start + 1

    iou = num_common / (num_pred + num_target - num_common)

    return 1 - iou


pred_start = torch.tensor([5, 2], dtype=torch.float, requires_grad=True)
pred_end = torch.tensor([5, 4], dtype=torch.float, requires_grad=True)
target_start = torch.tensor([5, 2])
target_end = torch.tensor([5, 4])


delta = torch.tensor ([1.e-5, 0.0])

iou_loss(pred_start, pred_end, target_start, target_end).mean().backward()
print ('no delta:    pred_start.grad =', pred_start.grad)

pred_start.grad *= 0.0
pred_end.grad *= 0.0
iou_loss(pred_start + delta, pred_end, target_start, target_end).mean().backward()
print ('plus delta:  pred_start.grad =', pred_start.grad)

pred_start.grad *= 0.0
pred_end.grad *= 0.0
iou_loss(pred_start - delta, pred_end, target_start, target_end).mean().backward()
print ('minus delta: pred_start.grad =', pred_start.grad)
</code></pre>
<p>Here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
no delta:    pred_start.grad = tensor([-0.5000, -0.1667])
plus delta:  pred_start.grad = tensor([ 0.5000, -0.1667])
minus delta: pred_start.grad = tensor([-0.5000, -0.1667])
</code></pre>
<p>You can see the gradient jump as you cross the equality boundary (just<br>
like with the <code>abs (x)</code> example).</p>
<p>Best.</p>
<p>K. Frank</p>

jjhh: <p>Hi K.Frank,</p>
<p>Thank you for the explanation.</p>
<pre><code class="lang-auto">def iou_loss(pred_start, pred_end, target_start, target_end):
    """
    pred_start, pred_end, target_start, target_end: b, *
    """

    num_common = (pred_end - pred_start + 1).clamp_min_(0)

    num_pred = (pred_end - pred_start + 1).clamp_min_(0)
    num_target = target_end - target_start + 1

    iou = num_common / (num_pred + num_target - num_common)

    return 1 - iou
</code></pre>
<p>I tested the function without <code>minimum</code> and <code>maximum</code>, but i still got the same gradients.</p>

KFrank: <p>Hi jjhh!</p>
<aside class="quote no-group" data-username="jjhh" data-post="3" data-topic="109844" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jjhh/48/13058_2.png" class="avatar"> jjhh:</div>
<blockquote>
<p>I tested the function without <code>minimum</code> and <code>maximum</code>, but i still got the same gradients.</p>
</blockquote>
</aside>
<p>First, I do not get the same gradients using your new version of <code>iou_loss()</code>.</p>
<p>Second, while your new <code>iou_loss()</code> still returns zero for the input parameters<br>
given in your first post, this is no longer the minimum as <code>iou_loss()</code> can now<br>
become negative.</p>
<p>If you believe otherwise please post a complete, runnable script, together with<br>
its output, that demonstrates your result.</p>
<p>Best.</p>
<p>K. Frank</p>

jjhh: <p>Hi K.Frank, you are right. the loss becomes <code>1 - num_pred / num_target</code> now and it can be negative. The signs of the gradients switched, too. Sorry it was a stupid question. Thank you for your help.</p>