UNET trained on DAVIS_2017 dataset struggles after 10 epochs

alessandro_mondin: <p>Hello all,<br>
I am trying to train the UNET on the DAVIS 2017 dataset.<br>
I am facing some issues because after 10 epochs the model is still struggling to output correct masks.<br>
In particular these are some of the outputs:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/9738193fc0714f06c2c69f1a157b0d93686c6085.png" data-download-href="https://discuss.pytorch.org/uploads/default/9738193fc0714f06c2c69f1a157b0d93686c6085" title="image_4"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9738193fc0714f06c2c69f1a157b0d93686c6085_2_690x482.png" alt="image_4" data-base62-sha1="lzKhU8dlVXhJDxaDxE9asC3DU5T" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/7/9738193fc0714f06c2c69f1a157b0d93686c6085_2_690x482.png, https://discuss.pytorch.org/uploads/default/original/3X/9/7/9738193fc0714f06c2c69f1a157b0d93686c6085.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/7/9738193fc0714f06c2c69f1a157b0d93686c6085.png 2x" data-dominant-color="D7D4D5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_4</span><span class="informations">1000×700 214 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe.png" data-download-href="https://discuss.pytorch.org/uploads/default/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe" title="image_3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe_2_690x482.png" alt="image_3" data-base62-sha1="hFs7KC1DUm8UNe9XrhfIOyfqo9g" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe_2_690x482.png, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7bd48dfb4e57c8f4f5bd5134477cfa9dc7994bfe.png 2x" data-dominant-color="C3BAB2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_3</span><span class="informations">1000×700 228 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0c04215c089025dad57a046e120cf01af49f47f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0c04215c089025dad57a046e120cf01af49f47f" title="image_2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0c04215c089025dad57a046e120cf01af49f47f_2_690x482.png" alt="image_2" data-base62-sha1="tMHdX0uE9GzRcCVtvPs2PKeis7d" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0c04215c089025dad57a046e120cf01af49f47f_2_690x482.png, https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0c04215c089025dad57a046e120cf01af49f47f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0c04215c089025dad57a046e120cf01af49f47f.png 2x" data-dominant-color="C9C9CB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_2</span><span class="informations">1000×700 159 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7.png" data-download-href="https://discuss.pytorch.org/uploads/default/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7" title="image_1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7_2_690x482.png" alt="image_1" data-base62-sha1="p7pQ8lhkpVI6ohSLPmXjaFpO1RZ" width="690" height="482" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7_2_690x482.png, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/0/b00cfa7808f870f2c02cf6bcf36a820c2e6164f7.png 2x" data-dominant-color="D2D9DD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image_1</span><span class="informations">1000×700 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am truly struggling to identify the issue. Could the problem be that even if the dataset is composed by many classes I am considering them as just one class and hence I am expecting it to output “1” for each target mask without considering classes? Could it be the root cause or should I further investigate in the training loop etc? I am trying to train it for other 10 epochs but I have the feeling that under-fitting it is not the problem here.<br>
I know that this question could be quite vague but I am really struggling to identify the issue.<br>
Here below the <a href="https://github.com/AlessandroMondin/computer_vision/tree/main/U-NET" rel="noopener nofollow ugc">link</a> to my project on Github.<br>
Thanks a lot!<br>
Alessandro</p>

KFrank: <p>Hi Alessandro!</p>
<aside class="quote no-group" data-username="alessandro_mondin" data-post="1" data-topic="157085" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alessandro_mondin/48/46669_2.png" class="avatar"> alessandro_mondin:</div>
<blockquote>
<p>I am facing some issues because after 10 epochs the model is still struggling to output correct masks.</p>
</blockquote>
</aside>
<p>10 epochs of training does not sound like a lot to me.  (As always, this<br>
will depend on the details of your use case.)</p>
<p>Networks for reasonably complicated image-processing tasks are often<br>
trained on hundreds or thousands (or more!) epochs.</p>
<p>I would suggest training on a training dataset, and then tracking the<br>
performance on a separate validation dataset.  You would certainly<br>
want to track the loss on both datasets, and you could also look at<br>
other performance metrics (such as accuracy).</p>
<p>People most commonly compute the loss (and other metrics) for both<br>
datasets after each epoch.</p>
<p>As long as your validation-dataset loss is still going down, your network<br>
is still learning features, etc., relevant to your real problem.  This is true<br>
even if your validation-dataset loss is significantly higher than your<br>
training-dataset loss – what matters is that the validation-dataset loss<br>
is still going down.</p>
<p>(If your validation-dataset loss starts going up, even though your<br>
training-dataset loss is still going down, you have likely started to<br>
“overfit” and further training, without changing something else about<br>
what you are doing, won’t help – and will likely degrade – your<br>
real-world performance.)</p>
<aside class="quote no-group">
<blockquote>
<p>Could the problem be that even if the dataset is composed by many classes I am considering them as just one class and hence I am expecting it to output “1” for each target mask without considering classes?</p>
</blockquote>
</aside>
<p>You could argue this either way:</p>
<p>My intuition tells me that if all you want to do is distinguish “foreground”<br>
from “background” (i.e., all non-background classes are collapsed into<br>
a single foreground class), then it will be easier to train with one single<br>
foreground class, and you will get better performance (on this simplified<br>
problem).  The idea is that this problem easier and that your network<br>
won’t be “wasting effort” on details that aren’t relevant (to your simplified<br>
performance metric).</p>
<p>On the other hand, if you collapse all of your non-background classes<br>
into a single foreground class,  you would be “hiding” information from<br>
the network.  Perhaps the information in differences between those classes<br>
helps the network better  learn the most important features, <em>even for<br>
merely distinguishing foreground from background.</em>  So maybe the network<br>
will train better if you keep the multiple foreground classes separate.</p>
<aside class="quote no-group">
<blockquote>
<p>or should I further investigate in the training loop etc?</p>
</blockquote>
</aside>
<p>Well, yes, a bug could always be the cause of your issues, so it’s always<br>
worthwhile double-checking  your code.  Having said that, although I can’t<br>
rule out a bug, I don’t see obvious symptoms of a bug in the results you’ve<br>
posted.</p>
<p>Best.</p>
<p>K. Frank</p>