How to train a system on a highly imbalanced data? Why is my system not learning?

farazk86: <p>I dont know if my data is unbalanced as I would imagine most of the MRI datasets are unbalanced.</p>
<p>I have 3D volumes of size <code>[30, 512, 1024]</code>, I am targetting small spheres in this volume, I have a target tensor for each input. Since Each volume may have a small number of events (spheres) then the majority of the target tensor is black with the spheres of interest labeled as 1.</p>
<p>I assume this is how all binary segmentation volumes are labelled. Keep the background 0 and the areas we are interested as 1.</p>
<p>To iterate over each volume I have divided it into patches of size <code>[30, 128, 128]</code>, so I would get a patch having nothing in it, for example for that patch my target patch would be all 0. But I noticed that for such patches I am consistently getting a very high loss and very low accuracy.</p>
<p>But for blocks where there is data in there, for example for patches where the target patch also shows my spheres location then I would get some sort of loss and accuracy, but would be very low.</p>
<p>This is the kind of output I am getting from my system, it looks like it is not learning at all with so many patches having no value of interest for me.</p>
<p>I wrote the input and target images to disk for each iteration and for <code>Loss: 1</code> it would be the patches where there is nothing in the target patch.</p>
<p>Am I doing something wrong? Is my system learning?</p>
<pre><code class="lang-auto">idx:  0 of  312 - Training Loss:  1.0 - Training Accuracy:  3.204042239857152e-11
idx:  5 of  312 - Training Loss:  0.9876335859298706 - Training Accuracy:  0.0119545953348279
idx:  10 of  312 - Training Loss:  1.0 - Training Accuracy:  7.269467666715101e-11
idx:  15 of  312 - Training Loss:  0.7320756912231445 - Training Accuracy:  0.22638492286205292
idx:  20 of  312 - Training Loss:  0.3599294424057007 - Training Accuracy:  0.49074622988700867
idx:  25 of  312 - Training Loss:  1.0 - Training Accuracy:  1.0720428988975073e-09
idx:  30 of  312 - Training Loss:  1.0 - Training Accuracy:  1.19782361807097e-09
idx:  35 of  312 - Training Loss:  1.0 - Training Accuracy:  1.956790285362331e-09
idx:  40 of  312 - Training Loss:  1.0 - Training Accuracy:  1.6055999862985004e-09
idx:  45 of  312 - Training Loss:  1.0 - Training Accuracy:  7.580232552761856e-10
idx:  50 of  312 - Training Loss:  1.0 - Training Accuracy:  9.510597864803572e-10
idx:  55 of  312 - Training Loss:  1.0 - Training Accuracy:  1.341515676323013e-09
idx:  60 of  312 - Training Loss:  0.7165247797966003 - Training Accuracy:  0.02658153884112835
idx:  65 of  312 - Training Loss:  1.0 - Training Accuracy:  4.528208030762926e-09
idx:  70 of  312 - Training Loss:  0.3205708861351013 - Training Accuracy:  0.6673439145088196
idx:  75 of  312 - Training Loss:  0.9305377006530762 - Training Accuracy:  2.3437689378624782e-05
idx:  80 of  312 - Training Loss:  1.0 - Training Accuracy:  5.305786885401176e-07
idx:  85 of  312 - Training Loss:  1.0 - Training Accuracy:  4.0612556517771736e-07
idx:  90 of  312 - Training Loss:  0.8207412362098694 - Training Accuracy:  0.0344742126762867
idx:  95 of  312 - Training Loss:  0.7463213205337524 - Training Accuracy:  0.19459737837314606
idx:  100 of  312 - Training Loss:  1.0 - Training Accuracy:  4.863646818620282e-09
idx:  105 of  312 - Training Loss:  0.35790306329727173 - Training Accuracy:  0.608722984790802
idx:  110 of  312 - Training Loss:  1.0 - Training Accuracy:  3.3852198821904267e-09
idx:  115 of  312 - Training Loss:  1.0 - Training Accuracy:  1.5268487585373691e-09
idx:  120 of  312 - Training Loss:  1.0 - Training Accuracy:  3.46353523639209e-09
idx:  125 of  312 - Training Loss:  1.0 - Training Accuracy:  2.5878148582347826e-11
idx:  130 of  312 - Training Loss:  1.0 - Training Accuracy:  2.3601216467272756e-11
idx:  135 of  312 - Training Loss:  1.0 - Training Accuracy:  1.1504343033763575e-09
idx:  140 of  312 - Training Loss:  0.4516671299934387 - Training Accuracy:  0.13879922032356262
</code></pre>
<p>I read that for such datasets <code>DiceLoss</code> works much better and that is what I am using here. I am using a 3D implementation of the U-NET.</p>
<p>I’m very new to this and want to ask if I am on the right path?</p>
<ul>
<li>
<p>Should I use some sort of weighted loss to factor in the imbalance in my dataset?</p>
</li>
<li>
<p>Should I increase the patch size so as there is a higher probability of points of interest being included in each patch?</p>
</li>
<li>
<p>Is this running as it is supposed to? And should I just increase the number of epochs to a very high number and just let it train overnight and hope that it learns?</p>
</li>
</ul>
<p>I assume MRI scans where only small tumors are detected have similar datasets, is my understanding correct?</p>
<p>Many thanks</p>

ptrblck: <p>You could try a weighted loss using the inverse of the class frequencies, which might help if your model is overfitting.<br>
However, I’m wondering why you’re getting a high loss for patches containing just background.<br>
If your model overfits to the background class, these patches should be nearly perfectly classified.</p>

farazk86: <blockquote>
<p>However, I’m wondering why you’re getting a high loss for patches containing just background.<br>
If your model overfits to the background class, these patches should be nearly perfectly classified.</p>
</blockquote>
<p>Thanks for answering, this is what I dont understand as well hence me asking here.</p>
<p>I am using the model provided here: <a href="https://github.com/wolny/pytorch-3dunet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - wolny/pytorch-3dunet: 3D U-Net model for volumetric semantic segmentation written in pytorch</a>. I am only using their provided implementation of the U-NET model along with their implementation of DiceLoss and Dicecoefficient, I have my own dataloader class for my dataset and my own trainer.py file to train the model.</p>
<p>The system works as expected with the provided random data, but with my data I am getting the above results. I manually checked each input and label as well, and the inputs seem to be in order. The background only classes should not give such a high loss.</p>
<p>I dont know what to check for next at this stage, as I dont know what to do to fix this. =/</p>

ptrblck: <p>Yeah, your loss probably explodes, since it seems the <a href="https://github.com/wolny/pytorch-3dunet/blob/94661c4a96cd92c5e3b011a43ce7e6f9c904f509/unet3d/losses.py#L67">class_weights</a> will be really high in case you are just dealing with a background image (<code>target_sum</code> should be zero, so you are dividing by <code>epsilon</code>).</p>
<p><a href="https://github.com/pytorch/pytorch/issues/1249#issuecomment-305088398">Here</a> is a simple implementation from <a class="mention" href="/u/issamlaradji">@IssamLaradji</a> which uses a smoothing factor, which might counter this effect.</p>

farazk86: <p>Hi,<br>
Unfortunately the linked code and the smoothing factor does not help much either, as for a patch where all I have are background the <code>tflat.sum()</code> would be 0. This would make <code>intersection</code> 0 as well, thus for majority of my patches or blocks I will get a return of 1.</p>
<p>So now rather than getting a very high loss I am getting a consistent loss of 1 for almost all patches.</p>
<p>This is strange that people have solved similar situations with dice loss but its not working for me.</p>

moreshud: <p>Hi,</p>
<p>I would be glad you share your way through as I am currently facing the same challenge. Thanks</p>