Binary classification with noisy images

Fatih_Haslak: <h2><a name="quarter_1_nok_mini_kik_egzos_00001124x123upload3qyqguohl9atl6njekqrv7pbkj7png-1" class="anchor" href="#quarter_1_nok_mini_kik_egzos_00001124x123upload3qyqguohl9atl6njekqrv7pbkj7png-1"></a><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/181351a453a2d5e5e5f9ce749bf7f66e1b93a9d9.png" alt="quarter_1_NOK_Mini_Kik_Egzos_00001" data-base62-sha1="3qYQGUoHl9ATl6NJEkQrV7PBkj7" width="124" height="123"></h2>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e79cae2d4f8c6491a6f779405e46306488fa7f10.png" alt="quarter_1_OK_Mini_Kik_Egzos_00003" data-base62-sha1="x2VYSbkgn8NaRYEccqqyjEHA9Uc" width="124" height="124"></p>
<p>The ones above are two different paintings of mine that I’m trying to classify. I have 700 examples each. My classes are OK and NOK. The size of my pictures is 122x122, 140x140. When I train, I train as 144x144. I’ve been working on it for a long time. I tried many models and hyperparameters. Still I can’t get proper results. While my F1 score can reach 0.8 in the train section, it remains at 0.50 in the valid section. I cut a plate as a result of segmentation of my data. Since my data quality was low and noisy, I softened my image in this part as well.</p>

KFrank: <p>Hi Fatih!</p>
<aside class="quote no-group" data-username="Fatih_Haslak" data-post="1" data-topic="201494" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>My classes are OK and NOK.</p>
</blockquote>
</aside>
<p>My first question:  If you were to show me some samples from your data,<br>
say ten to one hundred each of OK and NOK, and give me a few words of<br>
explanation about what made them different, how well would I be able to<br>
do classifying your validation data set?</p>
<p>This is a simplistic, but very practical measure of how difficult you problem is.</p>
<aside class="quote no-group">
<blockquote>
<p>The size of my pictures is 122x122, 140x140.</p>
</blockquote>
</aside>
<p>Is there any correlation between your image sizes and whether an image<br>
is OK or NOK?  For example, are most of your OK images 122x122, while<br>
most of your NOK images are 140x140?</p>
<aside class="quote no-group">
<blockquote>
<p>When I train, I train as 144x144.</p>
</blockquote>
</aside>
<p>How do you get your images with size 144x144?  Do you upsample them?<br>
Do you pad them?</p>
<aside class="quote no-group">
<blockquote>
<p>While my F1 score can reach 0.8 in the train section, it remains at 0.50 in the valid section. I cut a plate as a result of segmentation of my data.</p>
</blockquote>
</aside>
<p>If I understand you correctly, your model performs significantly better (although<br>
not close to perfectly) on your training data set, but quite poorly on your<br>
validation data set.  This could be overfitting.</p>
<p>How big are your training and validation datasets?</p>
<p>You haven’t told us what your model looks like (or how long you’ve trained for),<br>
but, depending on the architecture of your model, you could add to it techniques<br>
like <code>Dropout</code> that are designed to reduce overfitting when training your model.</p>
<p>You could also use data augmentation as a way to semi-artificially increase the<br>
size of your training dataset which again should reduce overfitting.</p>
<p>Also, just to check, do your training and validation datasets have the same<br>
general character?  A good way to insure that they’re statistically the same is<br>
to take one large dataset and divide it randomly into a training and validation<br>
datasets.</p>
<aside class="quote no-group">
<blockquote>
<p>Since my data quality was low and noisy, I softened my image in this part as well.</p>
</blockquote>
</aside>
<p>Softening your images might make sense, but you could take the attitude that<br>
your network should itself “learn” to soften the images.  You could argue that<br>
even if softening makes the images look better to the eye, it could also obscure<br>
some information that your network could use to classify the images.</p>
<p>Best.</p>
<p>K. Frank</p>

Fatih_Haslak: <p>Hİ K.Frank</p>
<p>First of all, thank you for your valuable time.</p>
<p>For your first question: These pictures are a specific area of a metal plate. The black area between the 2 white circles may have scratches and dents. We classify them according to the size and depth of these scratches and dents. In the 1st picture, the white area in the bottom left is an example of an impact and we classify it as NOK.  Sometimes I also have a hard time classifying OK or NOK because it can be difficult to decide the depth of the scratches. 90% of the time I can classify it.</p>
<p>For your second question: Because a part has a front and a back side and I crop this area as a result of segmentation, the dimensions can be different. Some of my parts are 124x123, 123x123, 160x161, 159x158 because the part on the back side is a bit smaller. This does not change for OK and NOK, in general it varies according to the front and back side and segmentation mask.<br>
I use the function “img=cv2.resize(img,(144,144),interpolation=cv2.INTER_LINEAR)” to change the size.</p>
<p>For your third question: I have 600 OK and 600 NOK train yields. I have 60 OK and 60 NOK validation yields.</p>
<p>I used models such as resnet34, resnet50, densenet161 as models. The last layers of my model are as follows.  For example</p>
<p>" model.classifier =nn.Sequential(<br>
nn.Linear(num_features, 1, bias=True),<br>
nn.Sigmoid() ) "</p>
<p>I use Adam or RmsProb algorithm, I use 0.1 as weight_decay and 0.00001 as learning rate. Batch_size is 2.<br>
I only use gaussian noise as data augmentation.<br>
I try to train up to 200 epochs but after a while my loss graphs show that it is overfit.</p>
<p>Finally, Training and validation sets have general properties</p>

Fatih_Haslak: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a0d017029812f143365eafba1afcd19135895b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/2a0d017029812f143365eafba1afcd19135895b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2a0d017029812f143365eafba1afcd19135895b5.png" alt="image" data-base62-sha1="5ZZUVqJCilndfv9cCAYXjT1ijat" width="690" height="42" data-dominant-color="353535"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1238×77 2.56 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
A little sample my training</p>

de_Beyssac_Olivier: <p>Hello,</p>
<p>In your case, I was wondering whether making encode/decode process before entering model classifier would not bring any benefit,</p>
<p>That step of encoding/decoding would reduce noise. Output of decoder is then something cleaner that can be used as input to your model classifier,</p>
<p>Variational Autoencoders (VAEs) are type of auto encoder to consider in regards to that approach.</p>
<p>Hope it can help !</p>

Fatih_Haslak: <p>Hello,</p>
<p>I tried EsrGAN but I didn’t get a good result.</p>

KFrank: <p>Hi Fatih!</p>
<p>A quick summary:  It does sound like you are seeing overfitting.  Try more<br>
data, augmented data, <code>Dropout</code>, pre-trained models, smaller models (if<br>
they work for your use case), and <code>SGD</code> with momentum.</p>
<aside class="quote no-group" data-username="Fatih_Haslak" data-post="3" data-topic="201494" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>Sometimes I also have a hard time classifying OK or NOK because it can be difficult to decide the depth of the scratches. 90% of the time I can classify it.</p>
</blockquote>
</aside>
<p>This sounds reasonable.  It should be straightforward to build a classifier<br>
that works on your use case (for some machine-learning definition of<br>
“straightforward.”)</p>
<aside class="quote no-group">
<blockquote>
<p>For your third question: I have 600 OK and 600 NOK train yields. I have 60 OK and 60 NOK validation yields.</p>
</blockquote>
</aside>
<p>This is not an unreasonably small amount of data, but it’s not a lot.  In general,<br>
the best way to address overfitting is to get more training data if it is practical<br>
to do so.  (Of course it isn’t always.)  A factor of ten more data – say 5000 to<br>
10000 samples with ground-truth labels – could prove very helpful.</p>
<aside class="quote no-group">
<blockquote>
<p>I used models such as resnet34, resnet50, densenet161 as models.</p>
</blockquote>
</aside>
<p>Pytorch’s resnet, say <a href="https://pytorch.org/vision/0.18/models/generated/torchvision.models.resnet34.html" rel="noopener nofollow ugc">resnet34,</a> consistent with the original resnet paper, does<br>
not include <code>Dropout</code>, but you can add <code>Dropout</code> layers.  (There is some debate<br>
about whether <code>Dropout</code> works well with <code>BatchNorm</code> layers and where they<br>
should be located relative to one another, but <code>Dropout</code> is, in general, a helpful<br>
technique for reducing overfitting.)</p>
<p>Furthermore, if you’re not doing it already, you might try using a pre-trained<br>
resnet model, and only fine tune the last few layers.  Or you might first fine tune<br>
the last few layers and then fine tune the whole model a little bit.  The basic idea<br>
is that the more (trainable) parameters a model has, the more likely it is to<br>
overfit, so using the pre-trained parameter values as much as possible (which<br>
know nothing about your data, so can’t already be overfitting it) could very well<br>
help limit overfitting.</p>
<aside class="quote no-group">
<blockquote>
<p>The last layers of my model are as follows.  For example</p>
<p>" model.classifier =nn.Sequential(<br>
nn.Linear(num_features, 1, bias=True),<br>
nn.Sigmoid() ) "</p>
</blockquote>
</aside>
<p>Because of the <code>Sigmoid</code>, I assume that you are using <code>BCELoss</code>.  (If, not, that’s<br>
a problem in and of itself.)  This is unlikely to be your problem, but, for reasons<br>
of numerical stability, you should get rid of the <code>Sigmoid</code> and use as your loss<br>
criterion <code>BCEWithLogitsLoss</code> (which has <code>logsigmoid()</code> built into it).</p>
<aside class="quote no-group">
<blockquote>
<p>I use Adam or RmsProb algorithm</p>
</blockquote>
</aside>
<p>There is some credible lore that <code>Adam</code> (as well as <code>RMSprop</code>) can be more<br>
subject to overfitting than non-adaptive optimizers.  Try using <code>SGD</code> (with<br>
momentum).</p>
<aside class="quote no-group">
<blockquote>
<p>I only use gaussian noise as data augmentation.</p>
</blockquote>
</aside>
<p>Based on the images you posted, I assume that the white circular arcs in<br>
your images are always in the same place.  This would seem to limit the kinds<br>
of data augmentation you could use – for example, <code>RandomCrop</code> would cause<br>
the arcs to appear in somewhat different locations in your image, which might<br>
not work for you.</p>
<p>However, it does look to me like you could reflect the images about the diagonal<br>
that runs from the upper left to the lower right.  That would only give you a factor<br>
of two augmentation, but every little bit helps.</p>
<p>At the cost of interpolating pixel values (because pixels line up only for rotations<br>
of 90 degrees), you might try augmenting by randomly rotating your plate images<br>
around the center of the plate.  (Whether the pixel-interpolation would hurt more<br>
than the augmentation helps, I don’t know.)</p>
<p>Best.</p>
<p>K. Frank</p>

Fatih_Haslak: <p>Hi Frank,</p>
<p>(fc): Linear(in_features=512, out_features=1000, bias=True) For example with drop out:</p>
<p>Linear(512,1000)<br>
Dropout(0.5)<br>
Linear(1000,500)<br>
Dropout(0.5)</p>
<hr>
<hr>
<p>Linear(x,1)</p>
<p>Is this the right approach<br>
Should I use the SGD algorithm with lr_scheduler.StepLR</p>
<p>Best.</p>

KFrank: <p>Hi Fatih!</p>
<aside class="quote no-group" data-username="Fatih_Haslak" data-post="8" data-topic="201494" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fatih_haslak/48/66598_2.png" class="avatar"> Fatih_Haslak:</div>
<blockquote>
<p>For example with drop out:</p>
<p>Linear(512,1000)<br>
Dropout(0.5)<br>
Linear(1000,500)<br>
Dropout(0.5)</p>
<hr>
<hr>
<p>Linear(x,1)</p>
</blockquote>
</aside>
<p>Yes, but you would still want nonlinear “activations” (such as <code>ReLU</code>) between<br>
your <code>Linear</code> layers.</p>
<aside class="quote no-group">
<blockquote>
<p>Should I use the SGD algorithm with lr_scheduler.StepLR</p>
</blockquote>
</aside>
<p>That would be a reasonable thing to do.  But having a learning-rate scheduler<br>
doesn’t really have anything to do with overfitting – it just has the potential to<br>
help with your training in other ways.</p>
<p>Best.</p>
<p>K. Frank</p>

Fatih_Haslak: <p>Hi Frank,</p>
<p>Thanks for everything. I am working on it.</p>
<p>Best.</p>

mtrantalainen: <p>I would recommend trying to train for alternative output. Instead of binary out OK / NOK you should train to output rectangle which marks the biggest failure in the image. For the example images, the OK part should generate rectangle with coordinates (0,0),(0,0) and the NOK part should generate a rectangle around the segment of the image that causes the part to fail.</p>
<p>This would allow a practical way to see which kind of features your AI model finds for false positives or false negatives and checking for zeros in the output would be your final OK signal.</p>
<p>Trying to figure out why a binary model fails to output the expeted binary value is going to be really hard because you don’t have any clue about the detection quality.</p>
<p>And you could generate a lot more test cases by having slightly different coordinates for each failure case to train the model not to focus on individual pixels but overall area of the failure.</p>
<p>You could also train the model to output failure value (e.g. in range 0…1) for the failed detail (“how sure the AI model is about detecting failure”). Output 0 if the part is OK (that is , 1 if the part obviously fails and some value in between for borderline cases where even a human has hard time making the decision. That would further allow automating the output and separate parts that require further human decision but allow clear cases to be successfully marked automatically.</p>