Ignore_index _ also for SmoothL1Loss()?

MikeTensor: <p>Is there a way to use something like the <code>ignore_index=</code>-Function from the <code>torch.nn.CrossEntropyLoss</code> also for the <code>SmoothL1Loss()</code>?</p>
<p>I have two models (classifier and regression). For both I want to exclude a certain class/category from the loss_calculation. (For classifier I can do this in the <code>CrossEntropyLoss</code>).</p>
<p>– Does somebody see any way to do this?</p>

ptrblck: <p>No, since floating point numbers are expected for <code>SmootkL1Loss</code> as the target and ignoring one specific floating point value sounds like a bad idea.<br>
If you want to ignore specific values, you could use an unreduced loss (use <code>reduction="none"</code>), mask the loss values corresponding to your ignored values/ranges, and reduce the loss afterwards.</p>

MikeTensor: <p>okay, great – thanks.</p>
<p>How exactly does this masking work?<br>
I have a <code>target_class</code>(int) → <code>CrossEntropyLoss</code>. and a <code>target_distance</code> (float) → <code>SmoothL1Loss</code>.<br>
Both belong together to one single object.</p>
<p>So if the <code>target_class</code> is (let’s say) <code>8</code>: I don’t want to include it into the <code>CrossEntropyLoss</code> (Classification) neither into the <code>SmoothL1Loss</code> (regression).</p>
<p>Any idea how to make it easy, or where I could look for a solution?</p>

ptrblck: <p>Something like this should work:</p>
<pre><code class="lang-python">criterion = nn.SmoothL1Loss(reduction="none")

x = torch.randn(10, 1, requires_grad=True)
target = torch.randn(10, 1)
target[0, 0] = 8.
print(target)
# tensor([[ 8.0000],
#         [ 1.5230],
#         [-0.6978],
#         [ 0.6073],
#         [ 0.3451],
#         [-2.7498],
#         [ 0.5196],
#         [-0.5696],
#         [-0.4566],
#         [-0.5452]])

loss = criterion(x, target)
print(loss)
# tensor([[8.1776e+00],
#         [2.4468e-02],
#         [2.4312e-02],
#         [1.9052e-01],
#         [7.3119e-03],
#         [2.6121e+00],
#         [1.3799e+00],
#         [1.2772e+00],
#         [2.9809e-01],
#         [1.2309e-01]], grad_fn=&lt;SmoothL1LossBackward0&gt;)

mask = (~(target == 8.)).float()
print(mask)
# tensor([[0.],
#         [1.],
#         [1.],
#         [1.],
#         [1.],
#         [1.],
#         [1.],
#         [1.],
#         [1.],
#         [1.]])

loss = loss * mask
print(loss)
# tensor([[0.0000],
#         [0.0245],
#         [0.0243],
#         [0.1905],
#         [0.0073],
#         [2.6121],
#         [1.3799],
#         [1.2772],
#         [0.2981],
#         [0.1231]], grad_fn=&lt;MulBackward0&gt;)

loss.mean().backward()
print(x.grad)
# tensor([[-0.0000],
#         [-0.0221],
#         [ 0.0221],
#         [-0.0617],
#         [ 0.0121],
#         [ 0.1000],
#         [-0.1000],
#         [ 0.1000],
#         [ 0.0772],
#         [-0.0496]])
</code></pre>

MikeTensor: <p>Looks good, thanks.</p>
<p>Is there a possibility to get the <code>length</code> of your <code>loss</code>-Tensor without counting the <code>zeros</code>?</p>

ptrblck: <p>The length of a <code>loss</code> can be checked via <code>len(loss)</code> or <code>loss.size(0)</code>. In case you are interested in checking the non-zero entries, you could use <code>mask.sum()</code>.</p>