Data augmentation in semantic segmentation

Sarmad_GTU: <p>Hi again<br>
I just want to ask about these lines in semantic segmentation data augmentation operation<br>
based on previews question discussion ptrblck said :</p><aside class="quote" data-post="6" data-topic="129993">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarmad_gtu/48/41574_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/data-augmentation-changed-the-mask-in-semantic-segmentation/129993/6">Data augmentation changed the mask in semantic segmentation</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    so, which transform operations can I apply to my train dataset?
  </blockquote>
</aside>

<p>Spatial transformations applied on the input image should also be applied on the mask tensor to make sure that the input pixel location still corresponds to the mask (e.g. rotations, resizing, etc.).<br>
Besides that, you should treat the mask as a classification target and should not change its values.</p>
<p>also, I found this statement in Albumentations data augmentation methods:<br>
Many non-spatial transformations like,CLAHE, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added. They will be applied only to the image and not the mask.</p>
<p>and  they writ this code :</p>
<p>aug = A.Compose([<br>
A.OneOf([<br>
A.RandomSizedCrop(min_max_height=(50, 101), height=original_height, width=original_width, p=0.5),<br>
A.PadIfNeeded(min_height=original_height, min_width=original_width, p=0.5)<br>
], p=1),<br>
A.VerticalFlip(p=0.5),<br>
A.RandomRotate90(p=0.5),<br>
A.OneOf([<br>
A.ElasticTransform(alpha=120, sigma=120 * 0.05, alpha_affine=120 * 0.03, p=0.5),<br>
A.GridDistortion(p=0.5),<br>
A.OpticalDistortion(distort_limit=2, shift_limit=0.5, p=1)<br>
], p=0.8),<br>
A.CLAHE(p=0.8),<br>
A.RandomBrightnessContrast(p=0.8),<br>
A.RandomGamma(p=0.8)])</p>
<p>random.seed(11)<br>
augmented = aug(image=image, mask=mask)</p>
<p>image_heavy = augmented[‘image’]<br>
mask_heavy = augmented[‘mask’]</p>
<p>visualize(image_heavy, mask_heavy, original_image=image, original_mask=mask)</p>
<p>my question is: based on the above code the augment operation applied to the image and mask why they write “they will be applied only to the image and not the mask”</p>

Megh_Bhalerao: <p>Yes, because some image transformations which involve change of the intensity values of the original image, such as contrast enhancement, randomGamma etc. need to be only applied to the image.</p>
<p>Applying intensity based transformations to the mask does not make sense since the mask is composed of discrete values 0,1,2 etc. and it makes no sense to apply intensity values to mask as it will destroy the mask values.</p>
<p>Wheres, spatial transformations such as flipping the image, or rotating the image should be applied to the mask as well, since the orientation of the image is changed which changes the orientation of the corresponding mask also.</p>
<p>Is this clear?</p>

Sarmad_GTU: <p>Yep, It is clear , Thank you<br>
but in code they apply the transformation to both image and mask , Why ?<br>
this my question?</p>

Megh_Bhalerao: <p>Which line exactly in your original post. Can you tell?</p>

Sarmad_GTU: <aside class="quote no-group" data-username="Sarmad_GTU" data-post="1" data-topic="130060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarmad_gtu/48/41574_2.png" class="avatar"> Sarmad_GTU:</div>
<blockquote>
<p>mask_heavy = augmented[‘mask’]</p>
</blockquote>
</aside>
<p>mask_heavy = augmented[‘mask’]</p>

Sarmad_GTU: <p>Here they call the augment and the mask value changed</p>

Megh_Bhalerao: <p>I think the <code>heavy()</code> transformation may be a spatial augmentation (such as <code>rotate(), flip()</code> etc.) that is why the mask and image <strong>both</strong> are being changed.</p>

Sarmad_GTU: <aside class="quote no-group" data-username="Sarmad_GTU" data-post="1" data-topic="130060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarmad_gtu/48/41574_2.png" class="avatar"> Sarmad_GTU:</div>
<blockquote>
<p>A.CLAHE(p=0.8),<br>
A.RandomBrightnessContrast(p=0.8),<br>
A.RandomGamma(p=0.8)</p>
</blockquote>
</aside>
<p>but here we change the contrast and adding some noise to mask</p>

Megh_Bhalerao: <p>I think there is some misunderstanding. Noise is <strong>never</strong> added to the mask, noise is only added to the image. Since, think about it, if you add noise to the mask, the mask values get changed and the mask will stop making sense. Did you understand?</p>

Sarmad_GTU: <p>I understand dear <a href="https://discuss.pytorch.org/u/Megh_Bhalerao">Megh_Bhalerao</a><br>
but I am asking about these statements A.CLAHE(p=0.8),<br>
A.RandomBrightnessContrast(p=0.8),<br>
A.RandomGamma(p=0.8), are these statements change the mask value or not? because changing contrast seems to be adding some value to the mask.</p>

Megh_Bhalerao: <p>No, they will not modify the mask in any way as far as my understanding goes. Since <code>Brightness</code> and <code>Gamma</code> are intensity based transformations. I am not sure why you are observing the a change in the mask values. Maybe you can double check your code.</p>

Sarmad_GTU: <aside class="quote no-group" data-username="Sarmad_GTU" data-post="1" data-topic="130060">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sarmad_gtu/48/41574_2.png" class="avatar"> Sarmad_GTU:</div>
<blockquote>
<p>non-spatial transformations like,CLAHE, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added.</p>
</blockquote>
</aside>
<p>ok, because of this statement I am confused :<br>
“Many non-spatial transformations like,CLAHE, [RandomBrightness], [RandomContrast] [RandomGamma] can be also added. <strong>They will be applied only to the image and not the mask</strong>.”<br>
they said will be only applied to the image, not the mask but they applied to both using this :<br>
image_heavy = augmented[‘image’]<br>
mask_heavy = augmented[‘mask’]<br>
this is my mistake</p>

Megh_Bhalerao: <blockquote>
<p>image_heavy = augmented[‘image’]<br>
mask_heavy = augmented[‘mask’]</p>
</blockquote>
<p>This is just representative. Some aug will be added to mask and some will not.</p>

Sarmad_GTU: <p>How can I know which one added to image and which one will added to mask if they call augment<br>
am I just call augment or i have to specify which one will be called.</p>

Megh_Bhalerao: <p>No, you do not need to specify. It will automatically know.</p>

Sarmad_GTU: <p>this means we can add any type of augmentation in the function above and during these two lines:<br>
image_heavy = augmented[‘image’]<br>
mask_heavy = augmented[‘mask’]<br>
some operations will apply to images and others to mask based on the type of augmentation.<br>
is this right?</p>

Megh_Bhalerao: <p>Yes. That is correct.</p>

Sarmad_GTU: <p>ok<br>
thank you so much<br>
I appreciate that</p>