Check lists for debugging CNN

username: banikr
cooked: <p>Based on <a href="https://towardsdatascience.com/checklist-for-debugging-neural-networks-d8b2a9434f21" rel="noopener nofollow ugc">this post</a> here I am running some tests to check if the framework has no error/bug.</p>
<p><strong>1.</strong> <br>I created a train dataset with one single data point(3D image and mask) to overfit the U-Net. The validation set is also the same data point. After training for 200 epochs I got the following curves.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45.png" data-download-href="https://discuss.pytorch.org/uploads/default/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45_2_501x500.png" alt="image" data-base62-sha1="5U2LAtXafLTvbtdbZTDPV1Mn0GN" width="501" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45_2_501x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45_2_751x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/2960b4b0b7f8e3d2cab16ba2422df50dbe285f45_2_1002x1000.png 2x" data-dominant-color="FBFAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1216×1212 184 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Both train and validation dice score seems to reach 80%(<em>I expected it to be 100%</em>). And validation loss curve follows the training curve very closely. <br></p>
<p><strong>2.</strong> <br>With the same loss, learning rate, and weight initialization, I changed the validation set with completely different image patches. The result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e48adc90ac49a22c76d077f85e65bc3e3e660e81.png" data-download-href="https://discuss.pytorch.org/uploads/default/e48adc90ac49a22c76d077f85e65bc3e3e660e81" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e48adc90ac49a22c76d077f85e65bc3e3e660e81_2_511x500.png" alt="image" data-base62-sha1="wBMn7a28z0Yfnw93eZbRWRk71tf" width="511" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e48adc90ac49a22c76d077f85e65bc3e3e660e81_2_511x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e48adc90ac49a22c76d077f85e65bc3e3e660e81_2_766x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e48adc90ac49a22c76d077f85e65bc3e3e660e81_2_1022x1000.png 2x" data-dominant-color="FBFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1212×1184 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The validation curves seem to behave in a random way(<em>as it should</em>) and for some reason the dice go beyond 1 in train samples(all samples are the same image/mask).<br>
The dice overshoot may be due to the MSDL(multi-sourced dice loss) I have used(<em>which doesn’t take background into account</em>).<br>
<br><br>
From these tests’ results, is it confirmed that the current framework works? Are there any other checks that need to be performed?<br>
If the setup is good, shouldn’t it overfit in the first case or dice metric reaching nearly 1?</p>