[Image Segmentation] Preprocessing of multilabels in 2D training data before training model

ankita.buntolia: <p>Hi,</p>
<p>I have 2D  training data (Image, label) of shape 496x512. Each Image can have multi labels (in total 4 classes as 0,1,2, or 3).</p>
<p><strong>Here what I have until now:</strong><br>
From the DataSet:<br>
BEFORE TRANSFORMSTION:<br>
 → label shape from OCTDataset: (496, 512)<br>
 → img shape from OCTDataset: (496, 512)<br>
 → image dtype from OCTDataset: float32, label dtype from OCTDataset: float32</p>
<p>AFTER TRANSFORMSTION:<br>
 → shape just after tranform of the label: torch.Size([1, 512, 512])<br>
–&gt;shape just after tranform of the img: torch.Size([1, 512, 512])<br>
–&gt;image dtype after transform : torch.float32, label dtype after transform : torch.float32</p>
<p>FROM DATALOADER:<br>
image shape: torch.Size([2, 1, 512, 512]), label shape: torch.Size([2, 1, 512, 512])<br>
image dtype: torch.float32, label dtype: torch.float32</p>
<p><strong>My Question:</strong><br>
In total there can be 4 classes (including background) but how should I encode these classes? With current code, my label from dataloader is in shape of torch.Size([2, 1, 512, 512]) but I guess it should be torch.Size([2, classes=4, 512, 512]). But how Should I get it?</p>
<p>Thanks!</p>

ptrblck: <p>I assume you are working on a multi-label segmentation use case where each pixel can belong to 0, 1, or more classes.<br>
In this case the target should be multi-hot encoded where a 0 and 1 indicates if the class is inactive or active in the current pixel, respectively.<br>
Based on your current label shape it seems you are using a single channel only so are you using class indices?<br>
If so, than your use case would sound like a multi-class segmentation where each pixel belongs to one class only?</p>

ankita.buntolia: <p><strong>My Task:</strong><br>
Actually, original training data is in 3D with shape 49x496x512 for both Images and labels. And I have to create 2D images and labels out of it and then train a UNet to segment each class (3 eye fluids).</p>
<p>here is the code for converting 2D from slices of 3D data:</p>
<pre><code class="lang-auto"># give list of all oct images and references from the provided ROOT folder
def preprocess_oct_images_in_numpy(dir:str):
  references = list() #only for training data
  oct_images =list()
  for subdir, dirs, files in os.walk(dir):
    for file in files:
      filepath = subdir + os.sep + file
      if filepath.endswith("reference.mhd"):
        references.append(filepath)     
      elif filepath.endswith("oct.mhd"):
        oct_images.append(filepath)     

  references = sorted(references)
  oct_images = sorted(oct_images)
  
  updated_image_list =[]
  updated_label_list =[] #only for training data
  
  # creating 2D image (496x512) from each slice of 3D Image (49x496x512)
  # For image
  for data in range(len(oct_images)):
    image_numpy_3D = sitk.GetArrayFromImage(sitk.ReadImage(oct_images[data], sitk.sitkFloat32)) # order z,y,x
    for idx in range(image_numpy_3D.shape[0]):
      updated_image_list.append(image_numpy_3D[idx,:,:])

  # for label
  for index in range(len(references)):
    label_numpy_3D = sitk.GetArrayFromImage(sitk.ReadImage(references[index], sitk.sitkFloat32)) # order z,y,x
    for idx in range(label_numpy_3D.shape[0]):
      updated_label_list.append(label_numpy_3D[idx,:,:])

  return updated_image_list, updated_label_list
</code></pre>
<p>This is what one of my training data (image and label) before and after transform (in 2D) look like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/aef6660c8239579fcc145daa8f1200af96f3e721.png" data-download-href="https://discuss.pytorch.org/uploads/default/aef6660c8239579fcc145daa8f1200af96f3e721" title="Data"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aef6660c8239579fcc145daa8f1200af96f3e721_2_495x500.png" alt="Data" data-base62-sha1="oXMZa2VSk2cFsGj78TMWFK2FMcx" width="495" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aef6660c8239579fcc145daa8f1200af96f3e721_2_495x500.png, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aef6660c8239579fcc145daa8f1200af96f3e721.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aef6660c8239579fcc145daa8f1200af96f3e721.png 2x" data-dominant-color="231426"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Data</span><span class="informations">707×713 180 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="158537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In this case the target should be multi-hot encoded where a 0 and 1 indicates if the class is inactive or active in the current pixel, respectively.</p>
</blockquote>
</aside>
<p>Thanks so here I have created multi one hot encoding for labels within the dataset class (I have 2nd data set class as well for tranformation) and with this I get label shape in 4x496x512 and after transform (with padding) it converted to shape 4x512x512</p>
<pre><code class="lang-auto">class OCTDataset(Dataset):
  "OCT Scan data set"
  def __init__(self, images, references):
    super(OCTDataset, self).__init__()
    self.images = images
    self.references = references

  def __len__(self):
    return len(self.images)

  def __getitem__(self, idx):
    img = self.images[idx]
    label = self.references[idx] 

    # manual multi one hot encoding
    new_label = np.zeros((4, 496, 512),dtype=np.float32) # 4 here is 4 classes including background and 3 eye fluids

    for iRow in range(label.shape[0]):
      for iCol in range(label.shape[1]):
        if label[iRow,iCol] == 1:
          new_label[iRow,iCol,1] = 1
        if label[iRow,iCol] == 2:
          new_label[iRow,iCol,2] = 2
        if label[iRow,iCol] == 3:
          new_label[iRow,iCol,3] = 3

    return img, new_label
</code></pre>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="158537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Based on your current label shape it seems you are using a single channel only so are you using class indices?<br>
If so, than your use case would sound like a multi-class segmentation where each pixel belongs to one class only?</p>
</blockquote>
</aside>
<p>How can I check if its not multi class segmentation and not multilabel segmentation?<br>
May be in 2D it seems like multi class but when one have to rebuild from 2D to 3D again in that case may be its multi label segmentation (because of overlapping of identified fluid regions), is it correct?</p>
<p>Many thanks!</p>

ptrblck: <aside class="quote no-group" data-username="ankita.buntolia" data-post="3" data-topic="158537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bc79bd/48.png" class="avatar"> ankita.buntolia:</div>
<blockquote>
<p>How can I check if its not multi class segmentation and not multilabel segmentation?<br>
May be in 2D it seems like multi class but when one have to rebuild from 2D to 3D again in that case may be its multi label segmentation (because of overlapping of identified fluid regions), is it correct?</p>
</blockquote>
</aside>
<p>As previously described the main difference would be: does each “element” (pixel, voxel, sample) belong to only one class (multi-class segmentation/classification) or can it belong to zero, one, or multiple classes (multi-label segmentation/classification).</p>
<p>I don’t quite understand your label creation, as you seem to index the spatial dimensions with the class index?</p>
<pre data-code-wrap="python"><code class="lang-python">    # manual multi one hot encoding
    new_label = np.zeros((4, 496, 512),dtype=np.float32) # 4 here is 4 classes including background and 3 eye fluids

    for iRow in range(label.shape[0]):
      for iCol in range(label.shape[1]):
        if label[iRow,iCol] == 1:
          new_label[iRow,iCol,1] = 1
        if label[iRow,iCol] == 2:
          new_label[iRow,iCol,2] = 2
        if label[iRow,iCol] == 3:
          new_label[iRow,iCol,3] = 3
</code></pre>
<p>It seems <code>dim1</code> and <code>dim2</code> are the spatial dimensions, while you are indexing <code>dim0</code> as the spatial dim.</p>

ankita.buntolia: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="158537">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It seems <code>dim1</code> and <code>dim2</code> are the spatial dimensions, while you are indexing <code>dim0</code> as the spatial dim.</p>
</blockquote>
</aside>
<p>you mean like this?</p>
<pre><code class="lang-auto">class OCTDataset(Dataset):
  "OCT Scan data set"
  def __init__(self, images, references):
    super(OCTDataset, self).__init__()
    self.images = images
    self.references = references

  def __len__(self):
    return len(self.images)

  def __getitem__(self, idx):
    img = self.images[idx]
    label = self.references[idx] 

    # # manual multi one hot encoding
    new_label = np.zeros((4, label.shape[0], label.shape[1]),dtype=np.float32) # 4 here is 4 classes including background and 3 eye fluids

    for iRow in range(label.shape[0]):
      for iCol in range(label.shape[1]):
        if label[iRow,iCol] == 1:
          new_label[1,iRow,iCol] = 1
        if label[iRow,iCol] == 2:
          new_label[2, iRow,iCol] = 1
        if label[iRow,iCol] == 3:
          new_label[3, iRow,iCol] = 1
    return img, new_label
</code></pre>