BCELoss for MultiClass problem

a-parida12: <p>Is it a possibility to calculate the Multiclass crossentropy loss by successively using the <code>nn.BCELoss()</code> implementation</p>
<p>This is what I have tried.</p>
<pre><code class="lang-auto"># Implementation of the Multiclass Cross Entropy classification
def SegLossFn(predictions,targets):
    _, c, _, _ = predictions.size()
    loss=0
    m=nn.Sigmoid()
    loss_fn=nn.BCELoss()
    #BCE-&gt; MCE by adding for each of the classes BCE
    for i in range(c):
        loss+=loss_fn(m(predictions[0][i]),Variable(targets[i][0]).cuda())
            
    return loss

</code></pre>

ptrblck: <p>For multi-class classification you would usually just use <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="nofollow noopener">nn.CrossEntropyLoss</a>, and I don’t think you’ll end up with the same result, as you are calling <code>torch.sigmoid</code> on each prediction.</p>
<p>For multi-label classification, you might use <code>nn.BCELoss</code> with hot-encoded targets and won’t need a for loop.</p>
<p>Could you explain your use case a bit as I’m currently not sure to understand it properly?</p>

a-parida12: <p>My question is about the the addition operation and how it is handled. What happens the <code>grad_fn</code>? how do they get added?</p>

ptrblck: <p>The losses will get accumulated and your <code>loss</code> tensor will get <code>grad_fn=&lt;AddBackward0&gt;</code> as its <code>grad_fn</code>.</p>

a-parida12: <p>The problem for not using <code>nn.CrossEntropyLoss</code>was my predicted output is of size [N,C,H,W] and the target is of [N,C,H,W]. [H,W] in the C channel of the target is 0 or 1 based on presence or absence of class C. Is their a work around?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects a <code>torch.LongTensor</code> containing the class indices without the channel dimension. In your case, you could simply use:</p>
<pre><code class="lang-python">targets = torch.argmax(targets, 1)
</code></pre>
<p>to create your target tensor.</p>

lugiavn: <p>I’m not sure if there is so called “Multiclass Cross Entropy”.<br>
But doing binary classification for each class makes sense, so I think what you have tried is correct.</p>

rasbt: <aside class="quote no-group" data-username="lugiavn" data-post="7" data-topic="36675">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lugiavn/48/1799_2.png" class="avatar"> lugiavn:</div>
<blockquote>
<p>Multiclass Cross Entropy</p>
</blockquote>
</aside>
<p>It’s usually called multi-category cross entropy but yeah, the CrossEntropyLoss is essentially that. Just be careful, the CrossEntropyLoss takes the logits as inputs (before softmax) and the BCELoss takes the probabilities as input (after logistic sigmoid)</p>

lugiavn: <p>Thanks, I think I confuse multi-class with multi-label, where they do multiple BCE like that</p>

rasbt: <p>I don’t think you confused anything, because both multi-label cross entropy and binary cross entropy work for dealing with multi-class problems. The difference though is that in binary cross entropy, mathematically, you assume that the classes are independent.</p>

a-parida12: <p>i think this particular implementation has some problem beause when I do <code>loss.backward()</code> improvement is seen only in the last class. The other previous channels just have complement of the last class.</p>
<p>For example:<br>
<strong>Expected Output</strong></p>
<pre><code class="lang-auto">[[1, 0],             [[0, 1],           [[0, 0],
 [0, 0]]              [0, 0]]            [1, 1]] 
</code></pre>
<p><strong>Got Output</strong></p>
<pre><code class="lang-auto">[[1, 1],             [[1, 1],           [[0, 0],
 [0, 0]]              [0, 0]]            [1, 1]] 
</code></pre>

netaglazer: <p>how can i use MSEloss for multi class?</p>

ptrblck: <p>You could try to apply a <code>softmax</code> as the last activation function in your model and use <code>nn.MSELoss</code> directly on these outputs and the one-hot encoded targets.<br>
Alternatively, you could also use a single output unit, which could be used to predict the class “index” as a floating point number and calculate the loss using the targets containing class indices.</p>
<p>However, note that <code>nn.MSELoss</code> is not the usual loss function you would use for a classification use case.</p>

netaglazer: <p>tnx<br>
if I do soft-max and one-hot, the final loss is a vector with length equal to the classes count<br>
and in this case I cant do backwards because it works on single value:</p>
<pre><code class="lang-auto">def weighted_mse_loss(input,target,weights):
    out = (input-target)**2
    out = out * weights.expand_as(out)
    loss = out.sum(0)
    return loss
</code></pre>

ptrblck: <p>You could either pass the gradients in the same shape as the output (<code>torch.ones_like(loss)</code> would be the default) or reduce the loss to a scalar.</p>
<p>By default the loss functions will calculate the loss average and will return a scalar.</p>