BCELoss with class weights

Rafa: <p>I’m doing an image segmentation task. It’s a binary case. That is, the target pixels are either 0 (not of the class) or 1 (belong to the class).</p>
<p>I’m using BCELoss as the loss function. I’m using BCE instead of BCEWithLogits because my model already has a sigmoid at the end.</p>
<p>My dataset is quite unbalanced. Of all the pixels, only a small percentage belong to the target class. Hence, I’d like to use class weights.</p>
<p>I’ve tried to do this:</p>
<pre><code class="lang-auto">class_weights = torch.tensor([0.5, 74.0])
criterion = nn.BCELoss(weight=class_weights)

loss = criterion(predictions, label)
</code></pre>
<p>but I get an error:<br>
RuntimeError: The size of tensor a (352) must match the size of tensor b (2) at non-singleton dimension 3.</p>
<p>How can I do this? From what I’ve gathered, weights is expected to have the same dimensions as the batch. This is doing batch weighting, but what I actually want is class weighting. From what I understand, BCEWithLogits does have both weights and pos_weights, but I would like to avoid changing the criterion.</p>

ptrblck: <p>I guess there might be workarounds using <code>nn.BCELoss</code> with the sample weighting by computing the weight per batch using the class distribution, but I would rather switch to <code>nn.BCEWithLogitsLoss</code> allowing you to use <code>pos_weights</code> directly and for better numerical stability.</p>

Rafa: <p>I guess I don’t quite understand the difference between weights and pos_weights for the loss, but pos_weights are the class weights, is that it?</p>
<p>So, I should:</p>
<ol>
<li>Remove the sigmoid layer at the end of my model;</li>
<li>Apply the class weights as below;</li>
<li>Apply a sigmoid at my predictions when inferring<br>
Is this correct?</li>
</ol>
<pre><code class="lang-auto">class_weights = torch.tensor([74.0 / 0.5], dtype=torch.float32)
criterion = nn.BCEWithLogitsLoss(pos_weight=class_weights)
</code></pre>
<p>Also, if I can ask, is this the right way to calculate class weights? I have</p>
<pre><code class="lang-auto">no_class_count = 480473341       # Number of pixels with no_class
class_count = 3279159                 # Number of pixels with class
total_samples = no_class_count + class_count
no_class_weight = total_samples / (2 * no_kelp_count)    # 0.5
class_weight = total_samples / (2 * kelp_count)                # 0.74
</code></pre>
<p>Also, for the binary case I shouldn’t have what I had previously and instead have as below, is this correct?</p>
<pre><code class="lang-auto">class_weights = torch.tensor([0.5, 74.0])   # Raises error due to dimensions
class_weights = torch.tensor([74.0 / 0.5])
</code></pre>

ptrblck: <p>Yes, your approach looks generally correct.</p>
<p>A simple choice for <code>pos_weights</code> would be <code>num_negatives / num_positives</code> as it will weight the positive samples so that they have approx. the same influence on the loss.</p>

radhikachanian: <p>I had a doubt, I am using focal loss to tackle class imbalance. And I do not have a sigmoid layer at the end of my model. So I should be using BCEWithLogits correct?<br>
Also, these class weights are obtained using the inverse class frequencies:<br>
This is my focal loss implementation:<br>
class FocalLoss(nn.Module):<br>
def <strong>init</strong>(self, alpha=0.25, gamma=2.0, reduction=‘mean’):<br>
super(FocalLoss, self).<strong>init</strong>()<br>
self.alpha = alpha<br>
self.gamma = gamma<br>
self.reduction = reduction</p>
<pre><code>def forward(self, inputs, targets):
    targets = F.one_hot(targets, num_classes=2).float()
    
    inputs = torch.sigmoid(inputs)  
    inputs = inputs.view(-1)
    targets = targets.view(-1)
    class_weights = torch.tensor([1.11/10.04], dtype=torch.float32)
    BCE_loss = F.binary_cross_entropy(inputs, targets, reduction='mean', weight=class_weights)
    pt = torch.exp(-BCE_loss)
    focal_loss = self.alpha * ((1 - pt) ** self.gamma) * BCE_loss
    if self.reduction == 'mean':
        return torch.mean(focal_loss)
    elif self.reduction == 'sum':
        return torch.sum(focal_loss)
    elif self.reduction == 'none':
        return focal_loss
    else:
        raise ValueError("Invalid reduction mode. options: 'mean', 'sum', or 'none'.")
</code></pre>

alepistola: <p>You can use the loss provided by <a href="https://smp.readthedocs.io/en/latest/losses.html#segmentation_models_pytorch.losses.FocalLoss" rel="noopener nofollow ugc">Segmentation Models</a> specifing it inside the _init_ :</p>
<p><code>self.loss_fn = smp.losses.FocalLoss(mode=smp.losses.BINARY_MODE)</code></p>
<p>Then, inside your step function (train, validation, test) you can calculate the loss as follows:</p>
<pre><code class="lang-auto">logits = self.forward(image)
loss = self.loss_fn(logits, ground_truth)
</code></pre>