Loss for Multi-Label Segmentation

username: oezguensi
cooked: <p>Hi everybody,</p>
<p>I have following scenario. I have 4 classes (including background): “House”, “Door”, “Window”, “Background”. The two classes “Door” and “Window” obviously do not intersect. But both are in the class “House”.</p>
<p>First I subtracted the “Window” and “Door” masks from the “House” class and used a Multi-Class Segmentation approach using <code>CrossEntropyLoss</code> which uses Softmax, but I would like to change it to a Multi-Label application where the “Door” and “Window” pixels should also be labeled as “House”.</p>
<p>For that, I wanted to use <code>BCEWithLogitsLoss</code> which uses Sigmoid, but I don’t know how I can balance the classes. I have a list of weights that represent the average size of the classes. “House” having the smallest weight as it is the biggest class. E.g.: <code>{'House': 1, 'Door': 20, 'Window': 25}</code>. I passed this information in the Multi-Class scenario to the <code>weight</code> argument of <code>CrossEntropyLoss</code>.<br>
But how do I do it with <code>BCEWithLogitsLoss</code>? Do I have to pass the list <code>[1, 20, 25]</code> to <code>pos_weight</code> for that?</p>
<p>Also, is my approach even the right way to do Multi-Label Segmentation?</p>
<p>Thanks in advance.</p>

username: ptrblck
cooked: <p>The general approach of using <code>nn.BCEWithLogitsLoss</code> sounds reasonable for a multi-label segmentation, where each pixel might belong to more than a single class.</p>
<p>You could pass the <code>pos_weight</code> argument, but note that it’s defined as <code>nb_negative/nb_positive</code>. I’m not sure, if your current weights are calculates in this way.</p>

username: oezguensi
cooked: <p>Thank you for the reply. Unfortunately as I can’t check the documentation of <code>torch.binary_cross_entropy_with_logits</code>, I couldn’t quite understand the necessary shape and form of <code>pos_weight</code>. I tried following, but I quite don’t know if that is the correct approach:</p>
<pre><code class="lang-auto">def bce_with_logits_loss(logit, target):
    criterion = nn.BCEWithLogitsLoss(reduction='none')
    loss = criterion(logit, target)
    loss = torch.mul(loss, pos_weight[None, :, None, None])
    loss = torch.mean(loss)
    return loss
</code></pre>

username: ptrblck
cooked: <p>You can see the docs <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.BCEWithLogitsLoss">here</a>:</p>
<blockquote>
<p><strong>pos_weight</strong>  (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor"> <em>Tensor</em> </a> <em>,</em>  <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.</p>
</blockquote>
<p>You can pass <code>pos_weight</code> as an argument to the creation of <code>nn.BCEWithLogitsLoss</code> instead of manually applying it.</p>

username: oezguensi
cooked: <p>Sorry, I was talking about not being able to see the actual implementation of <code>torch.binary_cross_entropy_with_logits</code>. Unfortunately, passing <code>pos_weight</code> as a list of weights for each class doesn’t work. Here is an example:</p>
<pre><code class="lang-auto"># Random input and target of shape (N, C, H, W)
# Inputs are logits and target is binary
input = torch.rand(1, 3, 2, 2)
target = torch.round(torch.rand(1, 3, 2, 2))

criterion = BCEWithLogitsLoss(pos_weight=torch.tensor([1., 1., 1.]))
loss = criterion(input, target)

"""
raises RuntimeError: The size of tensor a (3) must match the size 
of tensor b (2) at non-singleton dimension 3
"""
</code></pre>
<p>I also tried the channel_last variant with tensor shaped of (N, H, W, C). That works but outputs a different result than expected. It is hard to understand the behavior of <code>pos_weight</code> in this segmentation setting.</p>

username: ptrblck
cooked: <p>You can <code>expand</code> the weights for a segmentation use case:</p>
<pre><code class="lang-python">input = torch.rand(1, 3, 2, 2)
target = torch.round(torch.rand(1, 3, 2, 2))

weight = torch.tensor([1., 1., 1.]).view(1, 3, 1, 1).expand(-1, -1, 2, 2)

criterion = nn.BCEWithLogitsLoss(pos_weight=weight)
loss = criterion(input, target)
</code></pre>
<p>The implementation of <code>binary_cross_entropy_with_logits</code> can be found <a href="https://github.com/pytorch/pytorch/blob/35cdb785228b8abea1d3bdb844aa5980e6642f8d/aten/src/ATen/native/Loss.cpp#L202-L218">here</a>.</p>

username: oezguensi
cooked: <p>Thank you, for the link to the implementation.<br>
The solution you provided unfortunately doesn’t work as expected.<br>
For example setting the last weight to 1000 like so:</p>
<pre><code class="lang-auto">input = torch.rand(1, 3, 2, 2)
target = torch.round(torch.rand(1, 3, 2, 2))

weight = torch.tensor([1., 1., 1000.]).view(1, 3, 1, 1).expand(-1, -1, 2, 2)

criterion = nn.BCEWithLogitsLoss(pos_weight=weight)
loss = criterion(input, target)
</code></pre>
<p>doesn’t change the loss at all.</p>

username: ptrblck
cooked: <p>It seems to work for me:</p>
<pre><code class="lang-python">input = torch.rand(1, 3, 2, 2)
target = torch.round(torch.rand(1, 3, 2, 2))

weight = torch.tensor([1., 1., 1000.]).view(1, 3, 1, 1).expand(-1, -1, 2, 2)

weighted_criterion = nn.BCEWithLogitsLoss(pos_weight=weight)
criterion = nn.BCEWithLogitsLoss()

weighted_loss = weighted_criterion(input, target)
loss = criterion(input, target)

print(weighted_loss)
&gt; tensor(75.8810)
print(loss)
&gt; tensor(0.7734)
</code></pre>

username: oezguensi
cooked: <p>Interesting. If I set <code>torch.manual_seed(1)</code> it doesn’t have an effect. If I leave it out it does something, but not what I would expect:</p>
<pre><code class="lang-auto">input = torch.rand(1, 3, 2, 2)
target = torch.round(torch.rand(1, 3, 2, 2))

weight = torch.tensor([1., 1., 1000.]).view(1, 3, 1, 1).expand(-1, -1, 2, 2)

weighted_criterion = nn.BCEWithLogitsLoss(reduction='none', pos_weight=weight)
criterion = nn.BCEWithLogitsLoss(reduction='none')

loss = criterion(input, target)
weighted_loss = weighted_criterion(input, target)

print(loss)
&gt; tensor([[[[0.9330, 0.8368],
          [1.2952, 0.6551]],
         [[0.3611, 0.3950],
          [1.0079, 0.7523]],
         [[0.6751, 0.3964],
          [0.9588, 0.7661]]]])
print(weighted_loss)
&gt; tensor([[[[0.9330, 0.8368],
          [1.2952, 0.6551]],
         [[0.3611, 0.3950],
          [1.0079, 0.7523]],
         [[675.08, 396.43],
          [0.95880, 0.76611]]]])
</code></pre>
<p>It only changed the upper two pixels of the last channel.</p>