Nuclei segmentation with variable nuclei classes per patch/image

kountaydwivedi: <p>I need to perform nuclei segmentation. There are total 5 nuclei classes. However, some patches contain only 2, 3, or 4 nuclei classes, while some contain all 5 classes. Lets say there is a patch A with only 3 nuclei classes – class A, C, and E. In this case, how do I create labelled and binary masks? Since each colour represents a particular nuclei class, should I use a black mask for class B and D?</p>
<p>Thank you.</p>

KFrank: <p>Hi Kountay!</p>
<aside class="quote no-group" data-username="kountaydwivedi" data-post="1" data-topic="218860">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kountaydwivedi/48/39666_2.png" class="avatar"> kountaydwivedi:</div>
<blockquote>
<p>Lets say there is a patch A with only 3 nuclei classes – class A, C, and E. In this case, how do I create labelled and binary masks?</p>
</blockquote>
</aside>
<p>Let me assume that you want to perform <em>semantic segmentation,</em> that is, you want your<br>
model to classify each pixel in an image as belonging to one of your classes.  Also, by<br>
“patch” I assume that you mean an entire image (that might be cropped out of  larger<br>
image and that might be processed as one sample in a batch of several samples) that<br>
is passed through your model for training / inference.</p>
<p>You will be performing a multi-class classification problem, presumably using<br>
<code>CrossEntropyLoss</code> as your loss criterion.  You will have <em>six</em> classes – your five nuclear<br>
types plus a “background” class, used to label pixels that aren’t part of any nucleus.</p>
<p>Your ground-truth mask “image” for a given patch / image will have the same spatial size<br>
as the given input image and consist of integer (<code>long</code>) class labels, in your case, the values<br>
0 through 5.  The value 0 would typically be used to label the background class (but you<br>
don’t have to do it this way), so if a pixel in the input image belongs to a nucleus of class A,<br>
the corresponding pixel in the mask would have value 1, class B would have mask value<br>
2, and so on.  A pixel in the input image that belongs to no nucleus would correspond to<br>
a mask pixel with value 0.</p>
<p>It’s perfectly fine for a mask to not have all six values in it.  If an input image happens not<br>
to contain any nuclei of class A, then its mask will not contain any pixels with value 1.</p>
<p>(It is not uncommon for pixels of one or more classes to be much more common than<br>
others, often with the background class being the most common.  If the class counts in<br>
your training data are sufficiently unbalanced, say by a factor of maybe three or more,<br>
you might consider using <code>CrossEntropyLoss</code>’s <code>weight</code> constructor argument to help compensate for the imbalance.)</p>
<p>Best.</p>
<p>K. Frank</p>

kountaydwivedi: <p>Thank you so much for your insights.</p>
<p>Actually, currently I just have the binary masks corresponding to each nucleus class. Using these binary masks, I should create a single multiclass mask having pixel values just as you mentioned (class A-E ==&gt; values 1-5; class BACKGROUND ==&gt; value 0). So, if a patch does not have class A and C, then there would not be a binary mask for these classes. In that case, how should I proceed with creating the multiclass mask?</p>
<p>In gist, what you mean by ground-truth mask is what I need to create. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thank you.</p>

KFrank: <p>Hi Kountay!</p>
<aside class="quote no-group" data-username="kountaydwivedi" data-post="3" data-topic="218860">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kountaydwivedi/48/39666_2.png" class="avatar"> kountaydwivedi:</div>
<blockquote>
<p>So, if a patch does not have class A and C, then there would not be a binary mask for these classes. In that case, how should I proceed with creating the multiclass mask?</p>
</blockquote>
</aside>
<p>Let me assume that each pixel in an image is in exactly one class (including background).<br>
Let me also assume that your existing binary masks respect this, that is, if a given pixel<br>
is <code>1</code> in the binary mask for class A, it is <code>0</code> for the binary class for class B, and so on.  For<br>
convenience, let me also assume that your binary masks are already pytorch tensors of<br>
type <code>long</code>.</p>
<p>You could then do something like the following:</p>
<pre data-code-wrap="python"><code class="lang-python">multiclassMask = torch.zeros (H, W, dtype = torch.long)
if  maskExists ('A'):  multiclassMask += 1 * maskA
if  maskExists ('B'):  multiclassMask += 2 * maskB
if  maskExists ('C'):  multiclassMask += 3 * maskC
if  maskExists ('D'):  multiclassMask += 4 * maskD
if  maskExists ('E'):  multiclassMask += 5 * maskE
</code></pre>
<p>(You wouldn’t have to follow this particular scheme – there are lots of ways to accomplish<br>
the same thing.  For example, if <code>maskA</code> doesn’t exist, you could simply create one made<br>
up of all zeros, and then add all five masks together with weights, as above.)</p>
<p>Best.</p>
<p>K. Frank</p>

kountaydwivedi: <p>Amazing! Thank you so much for your response.</p>
<p>Based on your answer, I have tried to implement the steps as follows (please let me know if it is correct):</p>
<pre><code class="lang-auto">maskA = cv.imread('maskA.png')
maskB = np.zeros_like(maskA)
maskC = cv.imread('maskC.png')
maskD = np.zeros_like(maskA)
maskE = cv.imread('maskE.png')

stackedBinaryMask= np.dstack([maskA, maskB, maskC, maskD, maskE])
multiclassMask = np.zeros_like(maskA)
for i in range(stackedBinaryMask.shape[-1]):
    multiclassMask[stackedBinaryMask[...,i] == 255.] = i+1

</code></pre>
<p>Is this approach correct?<br>
Thank you.</p>

KFrank: <p>Hi Kountay!</p>
<aside class="quote no-group" data-username="kountaydwivedi" data-post="5" data-topic="218860">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kountaydwivedi/48/39666_2.png" class="avatar"> kountaydwivedi:</div>
<blockquote>
<p>Is this approach correct?</p>
</blockquote>
</aside>
<p>I can’t speak to the details of the <code>cv</code> and <code>np</code> functions you are using, but if they behave<br>
like their pytorch cousins, yes, your approach should work.</p>
<p>Note that you will need to convert your <code>multiclassMask</code> to a pytorch tensor if you want<br>
to use it in a pytorch loss criterion such as <code>CrossEntropyLoss</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

kountaydwivedi: <p>Yes, the <code>np.zeros_like()</code> works similar to <code>torch.zeros()</code>. Sure, I will keep in mind to convert the mask into pytorch tensor. Thank you for your response.<br>
Regards<br>
Kountay Dwivedi</p>