BCEWithLogitsLoss with weights causes runtimeError

Giuseppe_Sarno: <p>Hello,<br>
first of all let me say that I have seen different issues related to this problem however I cannot get it to work for my use case.</p>
<p>I am using a resnet network with a single output and therefore trying to use BCEWithLogitsLoss for binary classification (1 or 0). This is some code snippets:</p>
<p>THE LOSS FUNCTION</p>
<pre><code class="lang-auto">...
loss_fn = nn.BCEWithLogitsLoss(weight=weight)
...
</code></pre>
<p>WEIGHT TENSOR</p>
<pre><code class="lang-auto">tensor([7.2039, 0.5373], dtype=torch.float64)
weight size torch.Size([2])
</code></pre>
<p>MODEL</p>
<pre><code class="lang-auto">class Net2(nn.Module):

    def __init__(self):
        super().__init__()
        self.network = models.resnet152(pretrained=False)
        num_ftrs = self.network.fc.in_features
        self.network.fc = nn.Linear(num_ftrs, 1)
    def forward(self, xb):
        return self.network(xb)
</code></pre>
<p>TRAINING SNIPPET</p>
<pre><code class="lang-auto">        prediction_logist = model(input).squeeze()
        predicted_probability = torch.sigmoid(prediction_logist) 
        prediction = torch.round(predicted_probability)
        target = target.type_as(prediction_logist)        
        loss = loss_fn(prediction_logist, target)

</code></pre>
<p>ERROR</p>
<pre><code class="lang-auto">return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
RuntimeError: The size of tensor a (3) must match the size of tensor b (2) at non-singleton dimension 0
</code></pre>
<p>BATCH SIZE = 3</p>
<p>OUTPUT and TARGET/LABELS SIZE</p>
<pre><code class="lang-auto">torch.Size([3]) torch.Size([3])
</code></pre>
<p>the issue is obviously related to the the weights being a tensor of size 2 while target a tensor of size 3 = to the batchsize… things work for batchsize = 2 or no weights.</p>
<p>What do I need to do to work with a generic batchsize ?</p>
<p>Thanks you.<br>
Giuseppe.</p>

ptrblck: <p><code>unsqueeze</code> the model output and target to have a shape of <code>[batch_size, 1]</code> and rerun your code.</p>

Giuseppe_Sarno: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="209328">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>the model output and target to have a shape o</p>
</blockquote>
</aside>
<p>Hello, thank you for your reply.<br>
I added</p>
<pre><code class="lang-auto">        prediction_logist=prediction_logist.unsqueeze(1)
        target=target.unsqueeze(1)
</code></pre>
<p>before the loss function and the Size of output is now torch.Size([3, 1]) torch.Size([3, 1]</p>
<p>however now the issue moved to a different dimension and I now get</p>
<pre><code class="lang-auto">RuntimeError: output with shape [3, 1] doesn't match the broadcast shape [3, 2]
</code></pre>
<p>I assume the broadcast share has a 2 because of the weights dimension,</p>
<p>Thanks.</p>

ptrblck: <p>You are right and I misread the <code>weight</code> argument as <code>pos_weight</code>.<br>
Based on the <a href="https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html#bcewithlogitsloss">docs</a> the <code>weight</code> will rescale the loss of each batch element and should thus have the same shape as the batch size.<br>
Unsure, but you might want to use <code>pos_weight</code> instead which is used to balance e.g. an class imbalance in the dataset and can be defined as <code>pos_weight = nb_negative_examples/nb_positive_examples</code>.</p>

Giuseppe_Sarno: <p>Hello,<br>
Thanks for your update and took sometime to validate things as I am currently training the model.<br>
In a nutshell this is what I have done:</p>
<pre><code class="lang-auto">#pos_weight = nb_negative_examples/nb_positive_examples
pos_weight=torch.tensor(posweight)
pos_weight = pos_weight.to(device)
loss_fn = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
</code></pre>
<p>For the training loop I had to do a lot of squeeze and unsqueeze but things worked in the end.</p>
<pre><code class="lang-auto">modeloutput = model(input)
prediction_logist=modeloutput.squeeze()
predicted_probability = torch.sigmoid(prediction_logist) # calculate probabilities
predicted = torch.round(predicted_probability)
target = target.type_as(prediction_logist)

if datasize==1:
	prediction_logist = torch.tensor([prediction_logist])
	predicted= torch.tensor([predicted])

prediction_logist=prediction_logist.unsqueeze(1)
target=target.unsqueeze(1)

loss = loss_fn(modeloutput, target)
</code></pre>
<p>2 Things to highlight:</p>
<ol>
<li>I had to change logic when the batch size is == 1 as the dimension was not correct.</li>
</ol>
<pre><code class="lang-auto">if datasize==1:
        prediction_logist = torch.tensor([prediction_logist])
        predicted= torch.tensor([predicted])

</code></pre>
<ol start="2">
<li>I am impressed with the performace of this model compared when I used cross entropy and 2 multilabels.<br>
It took 200 EPOCHs before and not quite happy with the result while now with BCE modification I am getting good results even after 30 EPOCHs. Why is that ?</li>
</ol>
<p>Thanks</p>