Same output when using Multilabel BCE

Michael_Lempart: <p>Hi all,</p>
<p>Iam working on a segmentation project, where the input are CT images and the output consists of segmentations with a total of 3 classes.<br>
My first model was trained by transforming the segmentation map into a one-hot encoded vector and applying a softmax after the final layer. (so actually 4 class labels when counting the background)<br>
That works quite ok, but I have noticed that labels can overlap in certain cases.</p>
<p>My idea was therefore to stack the 3 different segmentation maps into 3 seperate image channels, where each segmentation is binary (background = 0, foreground = 1).</p>
<p>To train the model I used the BCEwithLogits loss and during training the result looks good.</p>
<p>Nevertheless, when using the model to predict new segmentations for a given CT image, the model outputs the same segmentation in all three channels and I can’t figure out whats wrong…</p>
<p>(Just to be sure that Iam not providing wrong target segmentations,  I already checked those…)</p>
<p>Any ideas or hints would be awesome!</p>
<p>Thanks in advance,</p>
<p>Cheers,</p>
<p>Michael</p>

KFrank: <p>Hi Michael!</p>
<p>What you describe is a standard approach.  You should be able to<br>
make it work, assuming you have enough good training data, and<br>
your network is appropriate for the task.</p>
<p>I have some comments in line, below.</p>
<aside class="quote no-group" data-username="Michael_Lempart" data-post="1" data-topic="104158" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_lempart/48/12375_2.png" class="avatar"> Michael_Lempart:</div>
<blockquote>
<p>My first model was trained by transforming the segmentation map into a one-hot encoded vector</p>
</blockquote>
</aside>
<p>Because you can transform your segmentation map into one-hot<br>
encoding, it sounds like you are working on a <em>single-label,</em> multi-class<br>
classification (segmentation) problem.</p>
<aside class="quote no-group">
<blockquote>
<p>and applying a softmax after the final layer.</p>
</blockquote>
</aside>
<p>Applying <code>softmax()</code> also suggests that you have a single-label<br>
problem.  (In such case, however, you would use <code>CrossEntropyLoss</code><br>
without the <code>softmax()</code> as <code>CrossEntropyLoss</code> has, in effect, the<br>
<code>softmax()</code> built in.  You would also not use one-hot labels with<br>
<code>CrossEntropyLoss</code>.)</p>
<aside class="quote no-group">
<blockquote>
<p>I have noticed that labels can overlap in certain cases.</p>
</blockquote>
</aside>
<p>“Overlapping” labels sounds like a <em>multi-label,</em> multi-class problem.</p>
<p>I assume that you mean you sometimes have overlapping <em>target</em><br>
labels – your ground-truth annotations that you use to train – rather<br>
than overlapping predicted labels.</p>
<p>But if you do have overlapping labels, how can you convert them<br>
to one-hot encoding?</p>
<aside class="quote no-group">
<blockquote>
<p>My idea was therefore to stack the 3 different segmentation maps into 3 seperate image channels, where each segmentation is binary (background = 0, foreground = 1).</p>
</blockquote>
</aside>
<p>This is appropriate for multi-label image segmentation.</p>
<aside class="quote no-group">
<blockquote>
<p>To train the model I used the BCEwithLogits loss</p>
</blockquote>
</aside>
<p>This is also appropriate.  But, just to be sure, you should be feeding<br>
the output of your final <code>Linear</code> layer as your logit predictions to<br>
your <code>BCEWithLogitsLoss </code> loss function, without passing it through<br>
<code>softmax()</code> or <code>sigmoid()</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>and during training the result looks good.</p>
</blockquote>
</aside>
<p>I assume that this means that your training-set loss goes down and<br>
your training-set accuracy goes up a sensible way.</p>
<aside class="quote no-group">
<blockquote>
<p>Nevertheless, when using the model to predict new segmentations for a given CT image, the model outputs the same segmentation in all three channels and I can’t figure out whats wrong…</p>
</blockquote>
</aside>
<p>This, of course, is not what you want.  Also, this should show up<br>
quantitatively in that your test-set loss and accuracy should be<br>
much worse that that for your training set.</p>
<p>One thing to check for:  Do you have an unbalanced dataset where<br>
some classes (including your background class) appear much more<br>
frequently than others?  If so, consider using <code>BCEWithLogitsLoss</code>’s<br>
<code>pos_weight</code> argument to help account for this.</p>
<p>Good luck.</p>
<p>K. Frank</p>

Michael_Lempart: <p>Hi and thanks for your great and detailed answer!</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>assume that you mean you sometimes have overlapping <em>target</em><br>
labels – your ground-truth annotations that you use to train – rather<br>
than overlapping predicted labels.</p>
</blockquote>
</aside>
<p>Here I need to clarify some things about my data:<br>
My dataset consists of one large bone structure. This bone structure has been segmented as a whole but it has also been split and segmented as three single structures.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>(In such case, however, you would use <code>CrossEntropyLoss</code></p>
</blockquote>
</aside>
<p>I actually use a combination of CE and Soft-dice which seems to work quite well.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>But if you do have overlapping labels, how can you convert them<br>
to one-hot encoding?</p>
</blockquote>
</aside>
<p>Good question! Some further clarification:<br>
The  network that uses one hot encoded labels was only trained in order to predict the three single structures.<br>
What I noticed is that the network struggles somewhat when one structure ends and The next begins. This is just because they are basically belonging to the same bone structure. Nevertheless I want my model to be able to differentiate between them.</p>
<p>In addition I want my model to be able to predict the three single structures as well as the whole structure.<br>
This is what I meant with ‘overlapping (target) labels’ and where I had the idea of stacking the segmentations into different image channels.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Do you have an unbalanced dataset</p>
</blockquote>
</aside>
<p>That is probably the case as the whole structure has much more foreground pixels than the single structures on their own.</p>
<p>I made a quick check and removed the channel with the whole structure and that seems to work.</p>
<p>What I don’t understand though is why the predictions in the different channels become the same when I include the segmentation of the whole structure.<br>
I would have expected that the channel wise Dice coeffient (which is used as an evaluation metric), would be good for 1 channel but would get worse for the other three as the whole segmentation takes over the loss function…</p>
<p>Best regards,</p>
<p>Michael</p>

Michael_Lempart: <p>One question that came to my mind:</p>
<p>Do the images have to been one-hot-encoded for BCEwithLogits?<br>
Right know Iam just tacking the different segmentation masks into 4 different channels:<br>
ch1: whole structure<br>
ch2: sub_structure 1<br>
ch3: sub_structure 2<br>
ch4: sub-structure 3</p>
<p>All images are binary (0 background, 1 foreground).</p>
<p>Cheers,</p>
<p>Michael</p>

KFrank: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="Michael_Lempart" data-post="3" data-topic="104158" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_lempart/48/12375_2.png" class="avatar"> Michael_Lempart:</div>
<blockquote>
<p>Here I need to clarify some things about my data:<br>
My dataset consists of one large bone structure. This bone structure has been segmented as a whole but it has also been split and segmented as three single structures.</p>
</blockquote>
</aside>
<p>Does this mean that:</p>
<ol>
<li>
<p>The large structure is – either in the real world and/or in your<br>
target data – <em>exactly</em> the union of the three substructures?</p>
</li>
<li>
<p>The three substructures have <em>exactly</em> zero intersection with<br>
one another?</p>
</li>
</ol>
<p>If so, would it suffice to simply segment the three substructures, with<br>
the large structure being determined by post-processing that takes<br>
the union of the substructures, rather than directly by the model itself?</p>
<p>You could argue that getting the large structure even more correct<br>
is more important that having the large structure fully consistent<br>
with the substructures, and that having the model segment the<br>
large structure (in addition to the substructures) lets you do this.<br>
But there is something to be said for at least trying the simpler<br>
approach.</p>
<aside class="quote no-group">
<blockquote>
<p>I actually use a combination of CE and Soft-dice which seems to work quite well.</p>
</blockquote>
</aside>
<p>I’ll mention <code>CrossEntropyLoss</code> again, below, but for now:  It’s perfectly<br>
reasonable to add Dice loss to <code>CrossEntropyLoss</code>, but my general<br>
advice would be to start with <code>CrossEntropyLoss</code> as the default, and<br>
only add Dice loss if it gives you a clear improvement.</p>
<aside class="quote no-group">
<blockquote>
<p>I want my model to be able to predict the three single structures as well as the whole structure.</p>
</blockquote>
</aside>
<p>If you don’t need to explicitly predict the large structure (because it<br>
can be adequately recovered as the union of the three substructures),<br>
you will be performing a single-label, four-class (background,<br>
substructure-1, substructure-2, substructure-3) segmentation problem.<br>
For this you will want <code>CrossEntropyLoss</code> (adding Dice loss, if that<br>
clearly helps).</p>
<p>Speaking in terms of a 2d, grayscale image, the input to your model<br>
should have shape <code>[nBatch, width, height]</code>, and the output<br>
(which will become the <code>input</code> to <code>CrossEntropyLoss</code>) should have<br>
shape <code>[nBatch, nClass = 4, width, height]</code>.  From the masks<br>
you have, you should build a single mask (per image) that consists<br>
of integer class labels (with values 0 through 3) and has shape<br>
<code>[nBatch, width, height]</code>.  (Note that there is no <code>nClass</code> dimension.)<br>
The integer-class-label mask will be the <code>target</code> for <code>CrossEntropyLoss</code>.</p>
<p>If you want to also explicitly predict your large structure, either because<br>
it is different than the union of the substructures, or because predicting<br>
it separately is (usefully) more accurate than taking the union, then I<br>
would recommend performing the two predictions is a single network<br>
as follows:</p>
<p>Understand the large-structure prediction as a single-label, binary<br>
segmentation problem, and the substructure predictions as a<br>
single-label, four-class segmentation problem (as described above).</p>
<p>Make your last <code>Linear</code> layer have <em>five</em> outputs, so that the model<br>
output has shape <code>[nBatch, 5, width, height]</code>.  Understand the<br>
first of your five outputs to be your large-structure prediction, and<br>
the remaining four to be your substructure predictions. Make a binary<br>
(0-1) mask for your large structure, and make a second four-class<br>
(as above) mask for your substructures.  Peel off the first of your<br>
five outputs and feed it into <code>BCEWithLogitsLoss</code> together with<br>
your large-structure mask, and peel off the last four outputs, feeding<br>
them into <code>CrossEntropyLoss</code> together with the class-label mask.<br>
Add the two losses together (probably with some relative weight<br>
that you tune by hand) to get a combined loss (that you call<br>
<code>loss.backward()</code> on).</p>
<p>Your large structure and substructures are closely related, so it<br>
makes sense to train a single model jointly on these two segmentation<br>
tasks.  Most of the upstream processing and features are shared by<br>
the two tasks, and only the final <code>Linear</code> layer “learns” how to predict<br>
the large structure and the substructures separately from the upstream<br>
features.</p>
<p>Based on my assumptions about your use case, this approach seems<br>
to do what you most naturally want.  You have both a single-label,<br>
binary segmentation task, and a single-label, four-class segmentation<br>
task, and this approach uses the (generally) most applicable methods<br>
to train these two tasks.</p>
<p>Lastly, to clarify:</p>
<aside class="quote no-group" data-username="Michael_Lempart" data-post="4" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_lempart/48/12375_2.png" class="avatar"> Michael_Lempart:</div>
<blockquote>
<p>Do the images have to been one-hot-encoded for BCEwithLogits?</p>
</blockquote>
</aside>
<p>You would typically use <code>BCEWithLogitsLoss</code> for a <em>multi-label</em>,<br>
multi-class problem, in which case your target image (mask) should<br>
be, if you will, <em>multi-hot</em> encoded.  That is, there can be a <code>1</code> in the<br>
position for any number of your classes, including none and all of<br>
them.  (In this case, you would not have a background class – the<br>
background “class” would be indicated by all <code>0</code>’s, that is, <code>0</code>’s for all<br>
of your foreground classes.)</p>
<p>If you can, in fact, one-hot encoded your target – exactly one of your<br>
classes (including the background class) is active at a time – then<br>
you have a *single-label, multi-class problem, and, in general,<br>
<code>CrossEntropyLoss</code> will be the better choice (because, in essence,<br>
your network “knows” that it’s being trained for a single-label task).</p>
<p>Good luck.</p>
<p>K. Frank</p>

Michael_Lempart: <p>Hi and thanks again for a more than helpful answer!</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If so, would it suffice to simply segment the three substructures, with<br>
the large structure being determined by post-processing</p>
</blockquote>
</aside>
<p>Yes, that would be an option. Nevertheless there is a slight difference between the large structure and single structure.</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>but my general<br>
advice would be to start with <code>CrossEntropyLoss</code> as the default, and<br>
only add Dice loss if it gives you a clear improvement</p>
</blockquote>
</aside>
<p>Thanks! I will make a comparison…</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Peel off the first of your<br>
five outputs and feed it into <code>BCEWithLogitsLoss</code> together with<br>
your large-structure mask, and peel off the last four outputs, feeding<br>
them into <code>CrossEntropyLoss</code> together with the class-label mask.</p>
</blockquote>
</aside>
<p>Thats I great idea!</p>
<aside class="quote no-group" data-username="KFrank" data-post="5" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you can, in fact, one-hot encoded your target – exactly one of your<br>
classes (including the background class) is active at a time – then<br>
you have a *single-label, multi-class problem, and, in general,<br>
<code>CrossEntropyLoss</code> will be the better choice (because, in essence,<br>
your network “knows” that it’s being trained for a single-label task).</p>
</blockquote>
</aside>
<p>Just to be sure:</p>
<p>If I use BCEwithLogits, i do not have to use a one hot encoded label, which In case of the large structure would give me a 2 channel image (ch 0 :background, ch1:foreground). In that case a simple binary image with 1 channel would be enough? (using a Sigmoid activation function)</p>
<p>Otherwise I could one hot encode my image and use a  Crossentropy loss instead? (using a Softmax function)?</p>
<p>Cheers,</p>
<p>Michael</p>

KFrank: <p>Hi Michael!</p>
<p>First, to clear up some terminology:</p>
<p>An <em>integer class label</em> is a single integer that specifies the class.<br>
These values run from <code>0</code> to <code>nClass - 1</code>, inclusive.  Pytorch’s<br>
<code>CrossEntropyLoss</code> takes integer class labels.</p>
<p><em>One-hot encoding</em> encodes exactly the same information as an<br>
integer class label.  (You can convert back and forth between the<br>
two without loss of information.)  A one-hot-encoded value is a<br>
vector of <code>nClass</code> <code>0</code>’s and <code>1</code>’s, where there is exactly one <code>1</code> and<br>
<code>nClass - 1</code> <code>0</code>’s.  The index of the location of the <code>1</code> is equal to the<br>
integer class label.</p>
<p><code>CrossEntropyLoss</code> does not take one-hot-encoded targets, but<br>
instead takes integer class labels, which, to reiterate, contain exactly<br>
the same information.</p>
<p><code>BCEWithLogitsLoss</code> (and <code>BCELoss</code>) also do not take one-hot<br>
encoded targets.  This is because they are used for somewhat<br>
different classification problems, and one-hot targets (or, equivalently,<br>
integer class labels) don’t contain the correct information for these<br>
problems.</p>
<aside class="quote no-group" data-username="Michael_Lempart" data-post="6" data-topic="104158" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michael_lempart/48/12375_2.png" class="avatar"> Michael_Lempart:</div>
<blockquote>
<p>If I use BCEwithLogits, i do not have to use a one hot encoded label,</p>
</blockquote>
</aside>
<p>Correct.</p>
<p>And, as outlined above, you would never want to use one-hot-encoded<br>
targets with <code>BCEWithLogitsLoss</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>In that case a simple binary image with 1 channel would be enough?</p>
</blockquote>
</aside>
<p>Correct.</p>
<aside class="quote no-group">
<blockquote>
<p>(using a Sigmoid activation function)</p>
</blockquote>
</aside>
<p>But you shouldn’t use <code>sigmoid()</code> with <code>BCEWithLogitsLoss</code>, as<br>
<code>BCEWithLogitsLoss</code>  has, in effect, the <code>sigmoid()</code> built in.  Just<br>
pass the raw-score <em>logits</em> that are the output of your final <code>Linear</code><br>
layer as the <code>input</code> to <code>BCEWithLogitsLoss</code>.  (You can use an explicit<br>
<code>sigmoid()</code> to feed <code>BCELoss</code>, but, although mathematically equivalent<br>
to feeding logits to <code>BCEWithLogitsLoss</code>, using <code>BCELoss</code> is<br>
numerically less stable.)</p>
<aside class="quote no-group">
<blockquote>
<p>Otherwise I could one hot encode my image and use a  Crossentropy loss instead?</p>
</blockquote>
</aside>
<p>No.  As outlined above, <code>CrossEntropyLoss</code> takes integer class labels<br>
rather than one-hot-encoded labels.</p>
<aside class="quote no-group">
<blockquote>
<p>(using a Softmax function)?</p>
</blockquote>
</aside>
<p>Again, no.  <code>CrossEntropyLoss</code> has, in effect, <code>softmax()</code> built in,<br>
so you should feed the output of your final <code>Linear</code> layer directly to<br>
<code>CrossEntropyLoss</code> as its <code>input</code> (without passing it through a<br>
<code>softmax()</code>).</p>
<p>Best.</p>
<p>K. Frank</p>

moreshud: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a></p>
<p>Firstly, I would like to appreciate your explanations and contributions. However, I would be glad for the affirmation of gained knowledge from the thread and likewise some clarifications.</p>
<ul>
<li>The difference between single label and multi-label multi-class segmentation is the presence overlapping - a case where particular corresponding pixel value(s) for each target classes is/are annotated</li>
<li>The go loss function for single label multi-class is <code>CrossEntropyLoss</code> while for multi-label multi-class is <code>BCEWithLoistLoss</code>. In any of the tasks, the logits (the final layer of the network’s output) are fed to the loss function with the stacked_target both of dimension <code>[nBatch, nClassess, width, height]</code>. A combination with dice loss can also be experimented.</li>
</ul>
<aside class="quote no-group quote-modified" data-username="KFrank" data-post="5" data-topic="104158">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>You would typically use <code>BCEWithLogitsLoss</code> for a <em>multi-label</em>,<br>
multi-class problem, in which case your target image (mask) should<br>
be, if you will, <em>multi-hot</em> encoded. That is, there can be a <code>1</code> in the<br>
position for any number of your classes, including none and all of<br>
them. (In this case, you would not have a background class – the<br>
background “class” would be indicated by all <code>0</code>’s, that is, <code>0</code>’s for all<br>
of your foreground classes.)</p>
</blockquote>
</aside>
<p>A typical instance of this</p>
<pre><code class="lang-auto">torch.tensor([[[1., 0., 0.],
         [1., 0., 0.],
         [0., 0., 0.]],

        [[0., 1., 1.],
         [0., 1., 0.],
         [1., 1., 1.]],

        [[0., 0., 1.],
         [0., 1., 1.],
         [1., 0., 1.]]])
</code></pre>
<p>Is this what you referred to as multi-hot encoded?</p>
<p>In addressing class imbalance, my approach is to take the summed average of the 1’s in each class as the class weight to be passed into my loss function i.e <code>[2, 6, 5]</code> in the case of <code>stacked_one_hot_encoded_targets</code> above.</p>

moreshud: <p><a class="mention" href="/u/kfrank">@KFrank</a> Your input will be appreciated</p>