Logit normalization and loss functions to perform instance segmentation

stark: <p>The goal is to perform <strong>instance segmentation</strong> with input RGB images and corresponding ground truth labels.</p>
<p>The ground truth label is multi-channel i.e. each class has a separate channel and there are different instances in each channel denoted by unique random numbers. Each label consists of 5 non-background classes (no background class info). The following figure will give a better insight into the labels (please note that the actual dimension is <code>128x128</code> not <code>7x7</code>).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03f469b72cf1d66d9938b749ecc0bebe68d741b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/03f469b72cf1d66d9938b749ecc0bebe68d741b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/3/03f469b72cf1d66d9938b749ecc0bebe68d741b1.png" alt="image" data-base62-sha1="yZ5fKY2W3LWQUBHxObBX9bQqpb" width="690" height="146" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">798×170 2.78 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<em>A label may have multiple instances of the same class objects.</em></p>
<p>Logit predictions were obtained on a batch data <code>[64, 5, 128, 128]</code> i.e. <code>[B, C, W, H]</code> from a typical U-Net (in a forward pass) and then passed through a sigmoid layer to get the normalized predictions in the interval <code>[0,1]</code>. There were so many problems encountered right before performing the backward pass and some of those are:</p>
<ol>
<li>
<p><code>nn.BCELoss()</code>: reported loss for the batches was either too low i.e. {0.0310, 0.0470, 0.1696,…} or negative i.e. {-0.2363, -0.0790, -0.1972,…}. I can’t think of a reason other than normalization of logits BUT sigmoid layer had already normalized the target values in interval [0,1]. (?)</p>
</li>
<li>
<p><code>nn.CrossEntropyLoss()</code>: This loss gave an error (as reported below). My understanding is to construct a single channel i.e. [64, 1, 128, 128] to use this loss function but I haven’t tried this idea.</p>
</li>
</ol>
<blockquote>
<p>ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [64, 5, 128, 128]</p>
</blockquote>
<p>I do understand that Dice, IOU, Jaccard,… are the popular loss functions for segmentation problems in addition to the cross-entropy based losses.</p>
<p>Would you please tell me where did I make the mistakes above? Is it the formulation of target labels?</p>

KFrank: <p>Hi Stark!</p>
<p>It’s not entirely clear to me what you are doing.</p>
<p>But to answer two technical questions:</p>
<aside class="quote no-group" data-username="stark" data-post="1" data-topic="97009" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<ol>
<li><code>nn.BCELoss()</code> : reported loss for the batches was either too low i.e. {0.0310, 0.0470, 0.1696,…} or negative i.e. {-0.2363, -0.0790, -0.1972,…}.</li>
</ol>
</blockquote>
</aside>
<p>If I take at face value the images of numbers you posted in your <em>image<br>
of textual information</em> by assuming that these are your actual <code>target</code><br>
values, then they are outside the <code>[0.0, 1.0]</code> range required by<br>
<code>BCELoss</code> (and <code>BCEWithLogitsLoss</code>).  <code>BCELoss</code> requires its <code>target</code><br>
values to be probabilities between 0 and 1.  Hence your unreasonable<br>
loss values.</p>
<aside class="quote no-group">
<blockquote>
<p>BUT sigmoid layer had already normalized the target values in interval [0,1]. (?)</p>
</blockquote>
</aside>
<p>Do you really mean that you have passed your <code>target</code> values through<br>
<code>Sigmoid</code>?  Or do you mean that your <code>input</code> to <code>BCELoss</code> (the output<br>
of your model) has been passed through <code>Sigmoid</code>?</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li><code>nn.CrossEntropyLoss()</code>: This loss gave an error (as reported below).</li>
</ol>
</blockquote>
</aside>
<p><code>CrossEntropyLoss</code> requires an <code>input</code> and <code>target</code> with different shapes,<br>
where <code>input</code> has an <code>nClass</code> dimension, and <code>target</code> does not.  For<br>
example, if your <code>input</code> is of shape <code>[nBatch, nClass, width, height]</code>,<br>
your target should have shape <code>[nBatch, nClass, width, height]</code>.</p>
<p>It appears that your <code>input</code> and <code>target</code> don’t satisfy this relationship. Hence<br>
your error.</p>
<p>Now some comments:</p>
<aside class="quote no-group">
<blockquote>
<p>The goal is to perform <strong>instance segmentation</strong> with input RGB images</p>
</blockquote>
</aside>
<p>Are your really applying your model to RGB images?</p>
<aside class="quote no-group">
<blockquote>
<p>Logit predictions were obtained on a batch data <code>[64, 5, 128, 128]</code></p>
</blockquote>
</aside>
<p>If you were using RGB images, I would expect your model to take a batch<br>
of data with shape <code>[64, nColor = 3, nChannel = 5, 128, 128]</code> (or<br>
something).</p>
<p>Are you somehow folding your RGB channels into your larger set of<br>
five channels?</p>
<aside class="quote no-group">
<blockquote>
<p>The ground truth label is multi-channel …  Each label consists of 5 non-background classes.</p>
</blockquote>
</aside>
<p>Does the number – 5 – of “ground truth” channels have anything to do<br>
with the apparent number – also 5 – of input channels, or is this just a<br>
coincidence?</p>
<aside class="quote no-group">
<blockquote>
<p>there are different instances in each channel denoted by unique random numbers.</p>
</blockquote>
</aside>
<p>What does this labelling scheme mean?  The first image of a number in<br>
your <em>image of textual information</em> is 0.  The second (distinct) number is<br>
32.  What does 32 mean?  Is it really random?  Are you expecting your<br>
network to learn the sequence of random numbers generated by your<br>
random number generator?</p>
<p>Does a “32” in “ch <span class="hashtag-raw">#0</span>” mean the same thing as a “32” in, say, “ch <span class="hashtag-raw">#2</span>”?<br>
Does a “32” in sample 3 of a batch mean the same thing as a “32” in,<br>
say, sample 12 of that batch?</p>
<p>Good luck.</p>
<p>K. Frank</p>

stark: <p>Thanks <a class="mention" href="/u/kfrank">@KFrank</a> for your time.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>It’s not entirely clear to me what you are doing.</p>
</blockquote>
</aside>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/2/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080_2_690x276.jpeg" alt="image" data-base62-sha1="i41B34du4ktimBzgvrtYItTYXK" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080_2_690x276.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/2/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080_2_1035x414.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/2/020ac4aab8d5e97ecfd8e07c52eefc8f25ad6080.jpeg 2x" data-dominant-color="6B675D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1126×452 200 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want a U-Net to learn the ground truth labels to perform the instance segmenttaion task. In semantic segmentation, all objects of the same type are marked using one class label while <strong>in instance segmentation similar objects get their own separate labels</strong>.</p>
<p>The problem is I cannot figure out a suitable encoding scheme that can work with the loss functions implemented in the PyTorch.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Do you really mean that you have passed your <code>target</code> values through <code>Sigmoid</code> ? Or do you mean that your <code>input</code> to <code>BCELoss</code> (the output of your model) has been passed through <code>Sigmoid</code> ?</p>
</blockquote>
</aside>
<p>The input to <code>BCELoss</code> is the model output after passing through <code>sigmoid</code>. Overall, I followed this sequence:<br>
RGB images + 5D labels (input) —&gt; model —&gt; 5D logit output —&gt; <code>sigmoid()</code> —&gt; <code>BCELoss()</code>/<code>CrossEntropyLoss()</code></p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>if your <code>input</code> is of shape <code>[nBatch, nClass, width, height]</code>, your target should have shape <code>[nBatch, nClass, width, height]</code>. It appears that your <code>input</code> and <code>target</code> don’t satisfy this relationship.</p>
</blockquote>
</aside>
<p><strong>The number of input channels to a model can be different from the number of output channels.</strong> Please correct me if I’m wrong. I’ve seen so many segmentation models designed this way while using the <code>CrossEntropyLoss()</code>.  My best guess is some problem with the way target labels are encoded.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are your really applying your model to RGB images?</p>
</blockquote>
</aside>
<p>Yes, my model takes a batch of [<strong>3</strong>, 128, 128] images and [<strong>5</strong>, 128, 128] of target labels as input.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>If you were using RGB images, I would expect your model to take a batch of data with shape <code>[64, nColor = 3, nChannel = 5, 128, 128]</code> (or something).</p>
</blockquote>
</aside>
<p>I would like to read some relevant research work using such a formulation. It’s quite interesting that you used <code>nColor</code> and <code>nChannel</code> differently as opposed to the conventional wisdom that says <code>nColor = nChannel</code>. For instance, <code>nColor = nChannel = 3</code> in an RGB image of size <code>3x128x128</code>.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are you somehow folding your RGB channels into your larger set of five channels?</p>
</blockquote>
</aside>
<p>Can you please elaborate on this?</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Does the number – 5 – of “ground truth” channels have anything to do with the apparent number – also 5 – of input channels, or is this just a coincidence?</p>
</blockquote>
</aside>
<p>A single label is of size [5, 128, 128] where <code>nChannel = 5</code> is indicating the <code>nClass = 5</code> which means foreground pixels in each input image may belong to any of the five classes.</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are you expecting your network to learn the sequence of random numbers generated by your random number generator?</p>
</blockquote>
</aside>
<p>Referring to the figure in the initial post, <strong>0 means a background pixel</strong>, any <strong>non-zero value means a foreground pixel</strong>. Random numbers were assigned just to distinguish multiple foreground instances belonging to the same class (Note: each class is a separate channel). By the way, <strong>assigning 1 to every foreground pixels in target labels (as in the figure below) let <code>BCELoss()</code> to produce reasonable values</strong>. However, I’m not sure how the multiple occurrences of foreground objects from the same class can be identified in the model output (since the model cannot infer this from the labels).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a43983c690b842c54dffea6a3c350b4f1f4220c.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a43983c690b842c54dffea6a3c350b4f1f4220c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a43983c690b842c54dffea6a3c350b4f1f4220c.png" alt="image" data-base62-sha1="61SSefBxdqYcPGBguHQlvDZTtla" width="690" height="392" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">796×453 5.83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Does a “32” in “ch <span class="hashtag-raw">#0</span>” mean the same thing as a “32” in, say, “ch <span class="hashtag-raw">#2</span>”? Does a “32” in sample 3 of a batch mean the same thing as a “32” in, say, sample 12 of that batch?</p>
</blockquote>
</aside>
<p>Not at all since the foreground objects have heterogeneous shapes and can be found anywhere in an image. It’s nothing more than a way to tell the model that there are different instances of the class-X.</p>

KFrank: <p>Hi Stark!</p>
<p>Okay, now I think I understand your use case.</p>
<p>Unfortunately, I don’t think I have anything useful to say about “instance<br>
segmentation.”  Perhaps some experts will chime in.</p>
<p>I do stand by my technical comments about the immediate causes of the<br>
errors your reported.  Some clarifying comment appear in line, below.</p>
<aside class="quote no-group" data-username="stark" data-post="3" data-topic="97009" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stark/48/29013_2.png" class="avatar"> stark:</div>
<blockquote>
<p>The problem is I cannot figure out a suitable encoding scheme that can work with the loss functions implemented in the PyTorch.</p>
</blockquote>
</aside>
<p>I’m not aware of any built-in pytorch loss functions that are <em>directly</em><br>
applicable to instance segmentation, but, like I said, I’m not knowledgeable<br>
about this use case.</p>
<aside class="quote no-group">
<blockquote>
<p>Overall, I followed this sequence:<br>
RGB images + 5D labels (input) —&gt; model —&gt; 5D logit output —&gt; <code>sigmoid()</code> —&gt; <code>BCELoss()</code>/<code>CrossEntropyLoss()</code></p>
</blockquote>
</aside>
<p>When you say “5D labels (input)” the only thing I think you can be referring<br>
to are the ground-truth labels (“target”) that you use for training.  Do you<br>
<em>really</em> feed in the ground-truth labels as input into your model?</p>
<p>Also, you <em>can’t</em> use the output of the same model as <code>input</code> to both<br>
<code>BCELoss</code> and <code>CrossEntropyLoss</code> (if they are using the same <code>target</code>).<br>
Wrong shapes.</p>
<aside class="quote no-group">
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>if your <code>input</code> is of shape <code>[nBatch, nClass, width, height]</code> , your target should have shape <code>[nBatch, nClass, width, height]</code> . It appears that your <code>input</code> and <code>target</code> don’t satisfy this relationship.</p>
</blockquote>
</aside>
<p><strong>The number of input channels to a model can be different from the number of output channels.</strong> Please correct me if I’m wrong. I’ve seen so many segmentation models designed this way while using the <code>CrossEntropyLoss()</code>.</p>
</blockquote>
</aside>
<p>As I mentioned in my previous post in the context of <code>BCELoss</code>, I am using<br>
<code>input</code> to refer to what pytorch calls the <code>input</code> to your loss function.  This<br>
is the output of your model, not the input to your model.  (This bit of pytorch<br>
terminology can lead to confusion.)</p>
<p>Note, in the bit you quoted, I gave the wrong shape for the <code>target</code> fed to<br>
<code>CrossEntropyLoss</code>:  It should be <code>[nBatch, width, height]</code>, <em>without</em> an<br>
<code>nClass</code> dimension.</p>
<aside class="quote no-group">
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="97009">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Are your really applying your model to RGB images?</p>
</blockquote>
</aside>
<p>Yes, my model takes a batch of [<strong>3</strong>, 128, 128] images and [<strong>5</strong>, 128, 128] of target labels as input.</p>
</blockquote>
</aside>
<p>Again, does your <em>model</em> really take <em>target labels</em> as input.  Or are you<br>
referring to your overall training procedure?</p>
<p>Leaving this aside, if I understand “target labels” to be the <code>target</code><br>
you feed to <code>BCELoss</code>, then your shapes do makes sense.  To use my<br>
preferred terminology, the input to your model is a tensor of shape<br>
<code>[nBatch, nChannel = nColor = 3, 128, 128]</code>.  (And, yes, it is<br>
common to refer to the three colors as “channels.”)</p>
<p>Your model then outputs predictions (the <code>input</code> to your loss function) of<br>
shape <code>[nBatch, nClass = 5, 128, 128]</code>.  This would be appropriate for<br>
<code>BCELoss</code> and (something analogous to) a multi-label, multi-class problem.</p>
<p>The <code>target</code> you feed your loss function should, for <code>BCELoss</code>, have the<br>
<em>same</em> shape (<code>[nBatch, 5, 128, 128]</code>) (which it appears it does), while,<br>
for <code>CrossEntropyLoss</code>, it should have a shape lacking the <code>nClass</code><br>
dimension (<code>[nBatch, 128, 128]</code>).</p>
<aside class="quote no-group">
<blockquote>
<p>By the way, <strong>assigning 1 to every foreground pixels in target labels (as in the figure below) let <code>BCELoss()</code> to produce reasonable values</strong>.</p>
</blockquote>
</aside>
<p>Yes. By doing so you are (properly) training a conventional multi-label,<br>
multi-class classifier.</p>
<aside class="quote no-group">
<blockquote>
<p>However, I’m not sure how the multiple occurrences of foreground objects from the same class can be identified in the model output (since the model cannot infer this from the labels).</p>
</blockquote>
</aside>
<p>But, also yes, doing so won’t perform <em>instance</em> segmentation for you<br>
(which, again, I don’t know how to do).  That is, I agree with your analysis<br>
that, while setting the all of the foreground pixel values to <code>1</code> does fix your<br>
problem of bad <code>BCELoss</code> values, doing so throws away the information<br>
you need to perform instance segmentation.</p>
<p>Best.</p>
<p>K. Frank</p>