Incorrect Smooth L1 Loss?

DKilkenny: <p>Hello,</p>
<p>I have been trying to go through all of the loss functions in PyTorch and build them from scratch to gain a better understanding of them and I’ve run into what is either an issue with my recreation, or an issue with PyTorch’s implementation.</p>
<p>According to Pytorch’s documentation for <a href="https://pytorch.org/docs/stable/generated/torch.nn.SmoothL1Loss.html#torch.nn.SmoothL1Loss" rel="noopener nofollow ugc">SmoothL1Loss</a> it simply states that if the absolute value of the prediction minus the ground truth is less than beta, we use the top equation. Otherwise, we use the bottom one. Please see documentation for the equations.</p>
<p>Below is my implementation of this in the form of a minimum test:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np

predictions = torch.randn(3, 5, requires_grad=True)
target = torch.randn(3, 5)

def l1_loss_smooth(predictions, targets, beta = 1.0):
    
    loss = 0
    for x, y in zip(predictions, targets):
        if abs(x-y).mean() &lt; beta:
            loss += (0.5*(x-y)**2 / beta).mean()
        else:
            loss += (abs(x-y) - 0.5 * beta).mean()

    loss = loss/predictions.shape[0]

output = l1_loss_smooth(predictions, target)
print(output)
</code></pre>
<p>Gives an output of:<br>
<code>tensor(0.7475, grad_fn=&lt;DivBackward0&gt;)</code></p>
<p>Now the Pytorch implementation:</p>
<pre><code class="lang-auto">loss = nn.SmoothL1Loss(beta=1.0)
output = loss(predictions, target)
</code></pre>
<p>Gives an output of:<br>
<code>tensor(0.7603, grad_fn=&lt;SmoothL1LossBackward&gt;)</code></p>
<p>I can’t figure out where the error in implementation lies.</p>
<p>Upon looking a little deeper into the <code>smooth_l1_loss</code> function in the <code>_C</code> module (file: <code>smooth_c_loss_op.cc</code>) I noticed that the doc string mentions that it’s a variation on Huber Loss but the documentation for <code>SmoothL1Loss</code> says it is huber loss.</p>
<p>So overall, just confused on how it’s implemented and whether it’s a combo of SmoothL1Loss and Huber Loss, Just Huber Loss, or something else.</p>
<p>Thanks!</p>

liangbright: <pre><code class="lang-auto">import torch
from torch.nn.functional import smooth_l1_loss, l1_loss
y_pred=torch.tensor(1.0)
y_true=torch.tensor(1.12)

loss1=smooth_l1_loss(y_pred, y_true, 1e-2, reduction = 'sum')
loss2=smooth_l1_loss(y_pred, y_true, 0, reduction = 'sum')
loss3=l1_loss(y_pred, y_true, reduction = 'sum')
print((y_pred-y_true).abs().item(), loss1.item(), loss2.item(), loss3.item())
</code></pre>
<p>Output is<br>
0.12000000476837158<br>
0.007200000341981649<br>
0.007200000341981649<br>
0.12000000476837158</p>
<p>weird</p>

ptrblck: <p>The third argument to <code>smooth_l1_loss</code> is the <code>size_average</code>, so you would have to specify this argument via <code>beta=1e-2</code> and <code>beta=0.0</code>, which will then give the same loss output as the initial custom code:</p>
<pre><code class="lang-python">y_pred=torch.tensor(1.0)
y_true=torch.tensor(1.12)

loss1=smooth_l1_loss(y_pred, y_true, beta=1e-2, reduction = 'mean')
loss2=smooth_l1_loss(y_pred, y_true, beta=0, reduction = 'mean')
loss3=l1_loss(y_pred, y_true, reduction = 'mean')
print((y_pred-y_true).abs().item(), loss1.item(), loss2.item(), loss3.item())
&gt; 0.12000000476837158 0.11500000208616257 0.12000000476837158 0.12000000476837158

def l1_loss_smooth(predictions, targets, beta = 1e-2):
    
    loss = 0
    for x, y in zip(predictions, targets):
        if abs(x-y) &lt; beta:
            loss += (0.5*(x-y)**2 / beta).mean()
        else:
            loss += (abs(x-y) - 0.5 * beta).mean()

    loss = loss/predictions.shape[0]
    return loss

output = l1_loss_smooth(y_pred.unsqueeze(0), y_true.unsqueeze(0), beta=1e-2)
print(output)
&gt; tensor(0.1150)

output = l1_loss_smooth(y_pred.unsqueeze(0), y_true.unsqueeze(0), beta=0.0)
print(output)
&gt; tensor(0.1200)
</code></pre>