Loss function for semantic segmentation in high resolution images, using patching/tiling strategy

jona699d: <p>Hi, i am doing binary semantic segmentation on large scale images 6000x6000. My data is highly skewed meaning there are only very few objects in the image. As the image has a high resolution I have the need of patching/tiling (of max 512x512 (memory limits)), the issue is when training, as I do random cropping, alot of the corresponding masks are empty/all black. Therefore I need some loss function which can handle alot of masks only containing 0, and every now and then, but most importantly learn from the masks wherein there is acutal content. See images below for reference.<br>
all suggestions are very welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>please note this is not the actual images but mimics my issue.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbc2de153614328143c0dd92dd09504759c6e4bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/fbc2de153614328143c0dd92dd09504759c6e4bb" title="binary_segmentation"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbc2de153614328143c0dd92dd09504759c6e4bb_2_690x388.png" alt="binary_segmentation" data-base62-sha1="zVblv67pPz5zWUBJgObY7iirKxZ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbc2de153614328143c0dd92dd09504759c6e4bb_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fbc2de153614328143c0dd92dd09504759c6e4bb_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fbc2de153614328143c0dd92dd09504759c6e4bb.png 2x" data-dominant-color="C4C4C4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">binary_segmentation</span><span class="informations">1280×720 12.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Jona!</p>
<aside class="quote no-group" data-username="jona699d" data-post="1" data-topic="167674" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jona699d/48/55446_2.png" class="avatar"> jona699d:</div>
<blockquote>
<p>i am doing binary semantic segmentation on large scale images 6000x6000.</p>
</blockquote>
</aside>
<p>Consider using a U-Net as your model (if you’'re not already).  U-Net is<br>
designed for performing semantic segmentation on large images, and<br>
generally works well.  (The sample image you posted appears to be<br>
appropriate for U-Net.)</p>
<aside class="quote no-group">
<blockquote>
<p>As the image has a high resolution I have the need of patching/tiling (of max 512x512 (memory limits)),</p>
</blockquote>
</aside>
<p>U-Net – as a fully-convolutional network – accepts images of arbitrary size,<br>
and with its “tiling strategy,” can accept images of arbitrary size, regardless<br>
of (reasonable) memory limitations.</p>
<p>It has a finite “field of view” (which can be either a good thing or a bad thing),<br>
so, for example, the data in an input image’s upper left corner has no effect<br>
on the predictions made for the lower right corner (as long as the two<br>
corners are outside of one another’s fields of view).</p>
<aside class="quote no-group">
<blockquote>
<p>Therefore I need some loss function which can handle alot of masks only containing 0, and every now and then, but most importantly learn from the masks wherein there is acutal content.</p>
</blockquote>
</aside>
<p>Pytorch’s <code>BCEWithLogitsLoss</code> is likely the most appropriate loss function<br>
for your use case.</p>
<p>You describe a situation where you have <em>imbalanced</em> training data in that<br>
you have many more background pixels than foreground pixels.  For your<br>
use case I would suggest a combination of two standard approaches to<br>
dealing with imbalanced training data: weighted sampling of the training<br>
data and class weights in the loss function.</p>
<p>Specifically, I wouldn’t crop randomly.  Crop out patches that include a<br>
foreground object or do not.  (But otherwise crop randomly.)  Crop out<br>
patches that are at least as large as your U-Net’s field of view.  From your<br>
description, you will have many more no-foreground-object patches than<br>
foreground-object patches in your cropped-patch training set.</p>
<p>Now when you randomly select training patches in your training loop, bias<br>
that selection so that you select, on average, equal numbers of object and<br>
no-object patches.  (<code>WeightedRandomSampler</code> might prove to be helpful<br>
when implementing such sampling.)</p>
<p>Depending on the character of your data, any given object patch may still,<br>
nonetheless, have significantly more background pixels than foreground<br>
pixels, so your training data could still be unbalanced in this regard.  If so,<br>
use <code>BCEWithLogitsLoss</code>’s <code>pos_weight</code> constructor argument to weight<br>
the (fewer) foreground pixels more heavily in the loss function.</p>
<p>One note:  Precise use of U-Net has some nuances.  When used with proper<br>
care, running a large image through U-Net as a bunch of separate patches<br>
using the tiling strategy yields exactly the same predictions as if you had run<br>
the whole image through U-Net all at once.  (This is a good thing.)</p>
<p>Best.</p>
<p>K. Frank</p>