Line segmentation of a rugby field with multi class

MarkW: <p>I have a task to segment the different lines of a rugby field:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/faad6722c87deb4482020fb5d22e3cbd767c310d.png" data-download-href="https://discuss.pytorch.org/uploads/default/faad6722c87deb4482020fb5d22e3cbd767c310d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faad6722c87deb4482020fb5d22e3cbd767c310d_2_647x500.png" alt="image" data-base62-sha1="zLASEHPqOMV9gwndcDRAKqfKEDX" width="647" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faad6722c87deb4482020fb5d22e3cbd767c310d_2_647x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/faad6722c87deb4482020fb5d22e3cbd767c310d_2_970x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/a/faad6722c87deb4482020fb5d22e3cbd767c310d.png 2x" data-dominant-color="4DA167"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1024×791 69.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Each line is labeled with a different class (11 classes + background).</p>
<ol>
<li>Which model would you use for such task? I thought about U-Net, is there anything more suitable?</li>
<li>In each frame of the tv stream the different classes takes only 2-4% of the pixels compared to the background. Is there a smart way to handle that?</li>
<li>What loss would you chose? Is there anything optimized for such case?</li>
</ol>

KFrank: <p>Hi Mark!</p>
<aside class="quote no-group" data-username="MarkW" data-post="1" data-topic="184682" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/53a042/48.png" class="avatar"> MarkW:</div>
<blockquote>
<ol>
<li>Which model would you use for such task? I thought about U-Net, is there anything more suitable?</li>
</ol>
</blockquote>
</aside>
<p>U-Net is a sound model for semantic segmentation. You should certainly<br>
give it a try.  (It would probably be my first choice.)</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>In each frame of the tv stream the different classes takes only 2-4% of the pixels compared to the background. Is there a smart way to handle that?</li>
</ol>
</blockquote>
</aside>
<p>Use <code>CrossEntropyLoss</code>’s <code>weight</code> constructor argument.  Use class<br>
weights that are (approximately) inversely proportional to the frequencies<br>
with which pixels of specific classes appear in your training data.</p>
<aside class="quote no-group">
<blockquote>
<ol start="3">
<li>What loss would you chose?</li>
</ol>
</blockquote>
</aside>
<p>Use <code>CrossEntropyLoss</code>.  It’s the go-to loss function for multi-class<br>
classification (of which multi-class semantic segmentation is a type).<br>
There are other losses, but, if I were to use them at all, I would probably<br>
use them as adjuncts to <code>CrossEntropyLoss</code> rather than instead of<br>
<code>CrossEntropyLoss</code>.</p>
<p>Do you have annotated ground-truth training data?  (The more, the<br>
better.)  Note that adjacent frames in the tv stream are likely to be<br>
very similar to one another, so they won’t really count as independent<br>
data samples.</p>
<p>Best.</p>
<p>K. Frank</p>

MarkW: <p><a class="mention" href="/u/kfrank">@KFrank</a> , Thanks for the advises.<br>
I have ~100 frames. Pretty different.</p>
<p>So all I have is to set the weight of the 11 classes to be high vs. the 1 background class?</p>

KFrank: <p>Hi Mark!</p>
<aside class="quote no-group" data-username="MarkW" data-post="3" data-topic="184682" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/53a042/48.png" class="avatar"> MarkW:</div>
<blockquote>
<p>I have ~100 frames.</p>
</blockquote>
</aside>
<p>That doesn’t seem like a lot – training might be a bit tricky.</p>
<p>Maybe split into 80 frames for training and 20 frames for validation.  Track<br>
your loss and some performance metrics, e.g., accuracy and per-class<br>
intersection-over-union, for both your training and validation sets as you<br>
train.  If your model keeps working better on your training set, but starts<br>
working worse on your validation set, you’ve started to <em>overfit,</em> a common<br>
problem when you don’t have a lot of training data.</p>
<p>If you do start to overfit, you could try data augmentation, but it’s not a<br>
magic bullet.  (The magic bullet is more data.)</p>
<aside class="quote no-group">
<blockquote>
<p>So all I have is to set the weight of the 11 classes to be high vs. the 1 background class?</p>
</blockquote>
</aside>
<p>Basically, yes.</p>
<p>Let’s say 10% of your pixels were foreground, with each foreground class<br>
accounting for about 1% of the pixels.  Then you would use a weight of<br>
about <code>1.1</code> (<code>100 / 90</code>) for your background class and weights of about<br>
<code>100.0</code> for each of your foreground classes.  Your foreground classes are<br>
probably not equally balanced, so, for example, if one of your foreground<br>
classes accounted for only 0.5% of your pixels, you would use for it a weight<br>
of about <code>200.0</code>.</p>
<p>Note, the weights need not be particularly precise – they should just<br>
roughly compensate for the class  imbalance.  If your results depend on<br>
the <em>exact</em> values of your weights, that would be a sign of some other<br>
problem such as unstable training or not enough data.</p>
<p>Best.</p>
<p>K. Frank</p>