Is it possible to train a segmentation model to identify part of a single image without having it overfit?

username: farazk86
cooked: <p>Hi all,</p>
<p>I have thousands of images that look like below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/2/b270c26e2c19308f410f3fd121b29792a9b3d7bf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b270c26e2c19308f410f3fd121b29792a9b3d7bf" title="FeatureIdx_1554"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b270c26e2c19308f410f3fd121b29792a9b3d7bf_2_506x500.jpeg" alt="FeatureIdx_1554" data-base62-sha1="psyzHwYN0SMNVFdt9rbvXmbIBKT" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b270c26e2c19308f410f3fd121b29792a9b3d7bf_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b270c26e2c19308f410f3fd121b29792a9b3d7bf_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/2/b270c26e2c19308f410f3fd121b29792a9b3d7bf_2_1012x1000.jpeg 2x" data-dominant-color="CEC9BF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_1554</span><span class="informations">1157×1143 242 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The different shapes are the growth we are interested in detecting. Every image has unique shapes in it and some images have hundreds of these objects we are interested in.</p>
<p>Another example is:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74fa960b2d523581bdd4aee42083bd3a657b6aba.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/74fa960b2d523581bdd4aee42083bd3a657b6aba" title="FeatureIdx_367-connected"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74fa960b2d523581bdd4aee42083bd3a657b6aba_2_506x500.jpeg" alt="FeatureIdx_367-connected" data-base62-sha1="gGQgq10p1TD2HPwK5SRkBoUm2Yq" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74fa960b2d523581bdd4aee42083bd3a657b6aba_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74fa960b2d523581bdd4aee42083bd3a657b6aba_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/4/74fa960b2d523581bdd4aee42083bd3a657b6aba_2_1012x1000.jpeg 2x" data-dominant-color="C7C5BE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_367-connected</span><span class="informations">1157×1143 299 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We want to extract the shapes only and extract the background.<br>
Since they are all different shapes, it is not possible to train a single model as we have 1 image per shape.</p>
<p>One solution I tried was to train a segmentation model using the input image that has been part labelled and then once it learns the shape, the same image can be used to identify the rest of the shapes.</p>
<p>For example, in labelMe I created a label file for one of the shapes:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e" title="image_2022-04-28_232321046"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e_2_512x500.jpeg" alt="image_2022-04-28_232321046" data-base62-sha1="wVspmqoAvFkzDuQZKObbpIAVnky" width="512" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e_2_512x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e_2_768x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/6/e6c44f31fc45535ca7da6e1ccab9fc83e0d0cc0e.jpeg 2x" data-dominant-color="CDC3BA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2022-04-28_232321046</span><span class="informations">1013×988 148 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Then trained a model using this image.</p>
<p>The assumption was that after training, the model would be able to identify the rest of the shapes in the image. But what actually happens is that it only identifies the training data. The same mask that was used to train it.</p>
<p>This does make sense as the model overfits to the input data, and when the test image is the same training image.</p>
<p>So my question is, given the problem I have, is it even possible to train a model like this? Learn from the input image and then use that to identify the remaining objects in the same image?</p>
<p>Or would I have to look into conventional computer vision approaches for this?</p>
<p>Thank you</p>

username: eqy
cooked: <p>Is the training example exactly as presented, with only part of the image labeled? In this case it would be confusing from the model perspective as the loss would actually penalize the model for labeling the rest of the image. You may want to try some preprocessing steps (e.g., cropping/padding images) if labeling them exhaustively is too time consuming. Additionally, some data augmentations (flips, rotations, etc.,) could help as it doesn’t seem like the task is orientation-dependent.</p>
<p>However, based on the well-defined contrast of the borders it might be worthwhile to visit traditional CV approaches to segment shapes (either to check if it works well enough or is a source of training data that can be refined).</p>

username: farazk86
cooked: <p>Thanks for the reply.</p>
<p>Thats what I thought as well. Using the same image for training and testing would not work in this case as the model would overfit to the labelled data and as you mentioned the model would penalize for labelling the rest of the image.</p>
<p>I guess, in my case then it would be best to look at traditional CV methods.</p>
<p>Thanks</p>

username: farazk86
cooked: <p>So a little update on this…</p>
<p>considering the helpful reply by <a class="mention" href="/u/eqy">@eqy</a>, I had another go at this after changing the input data a little bit.</p>
<p>I first extracted only the region of interest and labelled that in LabelMe. So that it is different from the input image and the model learns the shape of my object.</p>
<p>I am using the following input image for this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/01a204c0046fddc08b23aedee883ac84538b0074.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/01a204c0046fddc08b23aedee883ac84538b0074" title="FeatureIdx_380"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a204c0046fddc08b23aedee883ac84538b0074_2_506x500.jpeg" alt="FeatureIdx_380" data-base62-sha1="erBeNoVh8RD6p82FRq0CkX7Y0c" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a204c0046fddc08b23aedee883ac84538b0074_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a204c0046fddc08b23aedee883ac84538b0074_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/01a204c0046fddc08b23aedee883ac84538b0074_2_1012x1000.jpeg 2x" data-dominant-color="D1CEC4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_380</span><span class="informations">1157×1143 184 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the cropped out version of only a single column and the associated screenshot from LabelMe.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9dae06d0744b1227c0c130315aa25d278baf0bb9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9dae06d0744b1227c0c130315aa25d278baf0bb9" title="FeatureIdx_380-cropped"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dae06d0744b1227c0c130315aa25d278baf0bb9_2_506x500.jpeg" alt="FeatureIdx_380-cropped" data-base62-sha1="muTOpPjm1hihBqY7SiqQR3OBKd3" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dae06d0744b1227c0c130315aa25d278baf0bb9_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dae06d0744b1227c0c130315aa25d278baf0bb9_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/d/9dae06d0744b1227c0c130315aa25d278baf0bb9_2_1012x1000.jpeg 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_380-cropped</span><span class="informations">1157×1143 60.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/088fc77fafe1616aa29b0a24983aa9ddc2fffca7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/088fc77fafe1616aa29b0a24983aa9ddc2fffca7" title="Screenshot from 2022-05-09 17-13-25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/088fc77fafe1616aa29b0a24983aa9ddc2fffca7_2_690x388.jpeg" alt="Screenshot from 2022-05-09 17-13-25" data-base62-sha1="1dJRETdu8GolSbrJvBWBvaEAIHJ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/088fc77fafe1616aa29b0a24983aa9ddc2fffca7_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/088fc77fafe1616aa29b0a24983aa9ddc2fffca7_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/8/088fc77fafe1616aa29b0a24983aa9ddc2fffca7_2_1380x776.jpeg 2x" data-dominant-color="E0DDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2022-05-09 17-13-25</span><span class="informations">1920×1080 191 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I then gave the above as input to the model… and after training I gave the input image back as the test image. But the results were not  that good.</p>
<p>I then thought that maybe it is the lack of information in the image and that the majority of the image is empty, so I kept a single column as is and blurred the rest of the image, as can be seen from below:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/3/8348ae8d280dd3ad674abe2f5f88502bef6e177d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8348ae8d280dd3ad674abe2f5f88502bef6e177d" title="FeatureIdx_380"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8348ae8d280dd3ad674abe2f5f88502bef6e177d_2_506x500.jpeg" alt="FeatureIdx_380" data-base62-sha1="iJogF621URzuBPoTnT16XSRJ7D7" width="506" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8348ae8d280dd3ad674abe2f5f88502bef6e177d_2_506x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8348ae8d280dd3ad674abe2f5f88502bef6e177d_2_759x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/3/8348ae8d280dd3ad674abe2f5f88502bef6e177d_2_1012x1000.jpeg 2x" data-dominant-color="D6D2C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_380</span><span class="informations">1157×1143 46 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And I labelled the above image in LabelMe as well…</p>
<p>I then used this image to train a segmentation model and below are the results when that model is applied on the non-blurred input image:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f346db11672b6909417d273c3770da80bce246d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8f346db11672b6909417d273c3770da80bce246d" title="FeatureIdx_380"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f346db11672b6909417d273c3770da80bce246d_2_506x499.jpeg" alt="FeatureIdx_380" data-base62-sha1="kqQByCLXxvmyTL2ua6cUtQvYbqB" width="506" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f346db11672b6909417d273c3770da80bce246d_2_506x499.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f346db11672b6909417d273c3770da80bce246d_2_759x748.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f346db11672b6909417d273c3770da80bce246d.jpeg 2x" data-dominant-color="98A8B7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">FeatureIdx_380</span><span class="informations">925×914 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And the masks of the detections:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a9daf22fb867540f1c64e4337c98a6d41e26d96.png" data-download-href="https://discuss.pytorch.org/uploads/default/8a9daf22fb867540f1c64e4337c98a6d41e26d96" title="MASK-FeatureIdx_380"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/a/8a9daf22fb867540f1c64e4337c98a6d41e26d96.png" alt="MASK-FeatureIdx_380" data-base62-sha1="jMfJ6V9znYPoF0wiaFR3BM5WPY2" width="506" height="500" data-dominant-color="999999"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">MASK-FeatureIdx_380</span><span class="informations">1157×1143 9.23 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As can be seen the results are not very impressive.</p>
<p>What can I do to improve the results? Will augmentation help? If so which augmentations should I consider given the above that I want to achieve?</p>
<p>Why is augmentation even needed, as the regions I want to detect are almost exactly like the labelled region.</p>
<p>Why are the results so poor for all regions that look exactly like the input?</p>
<p>Some thoughts on this would  be helpful please.</p>
<p>Thank you</p>

username: eqy
cooked: <p>Augmentation should help but I would also consider the effect of aspect ratios/scale carefully here. If you are feeding the thin column as input during training time with a resize step, and then doing evaluation with a more “square” image with resizing, then the model will see wildly different aspect ratios at train vs. test time.<br>
So here I would design the augmentations and preprocessing such that they don’t cause a large discrepancy between the training time and testing time inputs (from the perspective of factors such as scale, resolution, etc.) An example of this could also be to break up both training and test images into smaller patches to ensure that the aspect ratios are close regardless of the total size of the input.</p>
<p>But ultimately there is no magic bullet for which augmentations; I recommend browsing through the literature and blog posts to see what is popular these days and what is applicable to your particular problem. The intro/related work section of popular vision papers should be a good starting point e.g., <a href="https://arxiv.org/pdf/1912.02781.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1912.02781.pdf</a></p>

username: KFrank
cooked: <p>Hi Faraz!</p>
<aside class="quote no-group" data-username="farazk86" data-post="4" data-topic="150361" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/farazk86/48/7071_2.png" class="avatar"> farazk86:</div>
<blockquote>
<p>What can I do to improve the results?</p>
</blockquote>
</aside>
<p>Let me not answer your specific question about training on only part<br>
of an image, but instead make the following observation:</p>
<p>I would think that if you had the resources to label (that is, to provide<br>
ground-truth segmentation masks for) entire images (rather than just a<br>
single vertical slice as in the example you posted), it could be possible<br>
to train a semantic-segmentation network (such as U-Net) to perform<br>
well on your task.</p>
<p>In addition to seemingly well-defined boundaries between “foreground”<br>
and “background,” the example images you posted have lots of structure<br>
that I imagine such a network could learn.  The images you posted have<br>
a relatively regular (within a given image) stripe-like / grid-like structure.<br>
So the idea would be that your network would learn not only local features<br>
that indicate the boundaries, but would also be looking for predicted<br>
boundaries that fit such a (learned) grid-like structure.</p>
<p>A multi-scale network such as U-Net might well be able to learn to<br>
recognize the larger-scale grid-like structure that your images have<br>
(in addition to learning the local features relevant for local boundary<br>
detection).</p>
<p>You also say that you have thousands of images, so – if you’re able to<br>
label them – I would think you could have enough training data.</p>
<p>As for augmentations, flips and rotations by 90 and 180 degrees would<br>
seem to make sense (unless there is a substantive horizontal vs. vertical<br>
difference that isn’t clear from the three example images you posted).</p>
<p>Given that the first two images you posted have different grid sizes<br>
relative to the size of the image, rescaling might also work for augmentation.</p>
<p>From the examples you posted, I can’t tell whether the boundaries of the<br>
images provide substantively useful information.  If not, then translations<br>
within the image – which is to say – cropping, could also be useful.</p>
<p>Also, mild blurring and addition of noise could be a legitimate augmentation.</p>
<p>But if you indeed have thousands of images that you could train on, I would<br>
first try training without augmentation, and see how far you get with that.</p>
<p>Best.</p>
<p>K. Frank</p>