Understanding the weight parameter of nn.NLLLoss()

stark: <p>Hi everyone,</p>
<p>I want to use <code>nn.NLLLoss()</code> to implement the Focal Loss function for a binary segmentation problem. The logits obtained from a network and binary targets (i.e., 0 for -ve class and 1 for +ve class) are used. Following cases summarizes my failed attempts (T_T)</p>
<pre><code class="lang-auto"># code for reproducibility
import torch
from torch import nn

log_prob = torch.rand((8, 1, 128, 128)).view(8*128*128, 1)      # (N,C,d1,d2, ..., dK) --&gt; (N*d1*...*dK,C)
target = (torch.rand((8, 1, 128, 128)) &gt; 0.5).long().view(-1)   # (N,d1,d2,...,dK) --&gt; (N*d1*...*dK,)
</code></pre>
<p><strong>Case 1: No weight OR weight for the positive class only.</strong><br>
I have no idea what caused this error since the target values are within [0, 1].</p>
<pre><code class="lang-auto">&gt;&gt; nn.NLLLoss()(log_prob, target)
</code></pre>
<p>OR</p>
<pre><code class="lang-auto">&gt;&gt; weight = torch.tensor([44.0])
&gt;&gt; nn.NLLLoss(weight)(log_prob, target)

"IndexError: Target 1 is out of bounds."
</code></pre>
<p><strong>Case 2: Weights for the negative and positive classes.</strong><br>
This time tensors for the input data were kept the same and weights for both classes were passed. What I understood from the following error is <code>nn.NLLLoss()</code> requires weight for each pixel index (i.e., <code>Case 3.</code>)</p>
<pre><code class="lang-auto">&gt;&gt; weight = torch.tensor([1.0, 44.0])    # (-ve, +ve) class weights
&gt;&gt; nn.NLLLoss(weight=weight)(log_prob, target)

"RuntimeError: weight tensor should be defined either for all 1 classes or no classes but got weight tensor of shape: [2]"
</code></pre>
<p><strong>Case 3: Weight per pixel.</strong><br>
This time the loss function again had problem with the weights.</p>
<pre><code class="lang-auto">&gt;&gt; weight = target * 44.0
&gt;&gt; nn.NLLLoss(weight=weight)(log_prob, target)

"RuntimeError: weight tensor should be defined either for all 1 classes or no classes but got weight tensor of shape: [131072]"
</code></pre>
<p>I will highly appreciate if you could explain how this <code>weight</code> parameter is designed to work. A reference to the <code>weight</code> or <code>pos_weight</code> parameter in <code>nn.BCEWithLogitsLoss()</code> would be a huge plus.</p>
<h1>
<a name="h-1" class="anchor" href="#h-1"></a>-----------------------------------------------------------------------</h1>
<p>P.S. To randomly generate the binary target tensor, I first used <code>target = torch.rand((8, 1, 128, 128), dtype=torch.bool)</code> which caused an error <code>RuntimeError: "check_uniform_bounds" not implemented for 'Bool'</code> Isn’t the random generation of a bool type tensor supported?</p>

matifali: <p>to create random bools, use</p>
<pre><code class="lang-auto">shape = (10, 10)
random_bools = torch.rand(shape) &gt; 0.5
</code></pre>

ptrblck: <p><code>nn.NLLLoss</code> expects a model output containing log probabilities in the shape <code>[batch_size, nb_classes, *]</code>. If you are using a model output of <code>[batch_size, 1]</code> only a single class is valid (class index 0) and your use case is invalid since your model will only output a single class. Fix this and passing a weight value for each class should work.</p>

stark: <p>That’s what I called <strong><code>Case 1: No weight OR weight for the positive class only</code></strong> and, strangely enough, it’s causing the error <code>IndexError: Target 1 is out of bounds</code> given a binary <code>target</code> tensor. You may use the provided reproducible code to get an insight into this.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="163213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>nn.NLLLoss</code> expects a model output containing log probabilities in the shape <code>[batch_size, nb_classes, *]</code>.</p>
</blockquote>
</aside>
<p>Thanks for mentioning the <em>log probabilities</em> as I forgot to use <code>nn.LogSigmoid()</code> in my implementation. I hope preferring <code>LogSigmoid</code> is a better choice over <code>LogSoftmax</code> for a binary segmentation problem <img src="https://discuss.pytorch.org/images/emoji/apple/interrobang.png?v=12" title=":interrobang:" class="emoji" alt=":interrobang:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="163213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>If you are using a model output of <code>[batch_size, 1]</code> only a single class is valid (class index 0)</p>
</blockquote>
</aside>
<p>Shouldn’t it be the <code>class index 1</code> since the model output logits are converted into the probabilities for being a foreground pixel (denoted as class 1)?</p>

ptrblck: <aside class="quote no-group" data-username="stark" data-post="4" data-topic="163213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>strangely enough, it’s causing the error <code>IndexError: Target 1 is out of bounds</code> given a binary <code>target</code> tensor.</p>
</blockquote>
</aside>
<p>That’s expected as explained in my previous post. If you want to use <code>nn.NLLLoss</code> for a binary classification the model output should have the shape <code>[batch_size, 2]</code>.</p>
<aside class="quote no-group" data-username="stark" data-post="4" data-topic="163213">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>I hope preferring <code>LogSigmoid</code> is a better choice over <code>LogSoftmax</code> for a binary segmentation problem</p>
</blockquote>
</aside>
<p>No, since <code>nn.NLLLoss</code> is used for multi-class classification use cases. A better choice could be <code>nn.BCEWithLogitsLoss</code> as this criterion is used for binary or multi-label use cases.</p>