Broader, deeper, dropout... what approach to try first to increase accuracy?

username: Sally_Goldin
cooked: <p>I am a newcomer to practical ML and PyTorch. I’m working on training an image discrimination model based mostly on textural features. My current model is very simple: 3 convolutional layers (plus ReLu/MaxPool), 3 fully connected layers. My latest validation gives me an accuracy of 61% for 5 classes, which is not sufficient for my application.</p>
<p>Are there any guidelines for modifying my model to improve this performance? To someone as naive as I am, it appears there are almost infinite dimensions for experimentation. I could increase the number of filters in each layer (make the layers “broader”). I could increase the number of layers (“deeper”). I could try using dropout; I currently have about 1300 inputs per class, so I could well be having overfit problems. Of course I could also experiment with the changing the loss function (currently Cross-Entropy Loss) or the optimizer (Stochastic Gradient Descent), or other parameters.</p>
<p>Can anyone offer suggestions on a systematic way to evaluate these many alternatives (as opposed to trial and error!) Or are there well-known rules of thumb as to what sort of modifications are likely to be most effective?</p>
<p>I realize this is a very general question, compared to most in this forum. I hope it will generate some useful discussion.</p>
<p>Thanks!</p>

username: BTNug
cooked: <p>Hi <a class="mention" href="/u/sally_goldin">@Sally_Goldin</a>, welcome to ML with Pytorch,</p>
<p>Before checking the below suggestion, please make sure that you are indeed overfitting. If yes, skip the step A.</p>
<p><strong>Step A. Make your Model overfit</strong></p>
<hr>
<p>According to my experiences, I tend to do these things:</p>
<ul>
<li>Try to make the model overfit (which is currently not optimized on your model)</li>
<li>Use pretrained model (this will <strong>HUGE</strong>-ly increase the performance and the <strong>speed</strong> of your training time. You could use <a href="https://pytorch.org/docs/stable/torchvision/models.html" rel="nofollow noopener">these available models</a> (*thanks to the community)</li>
<li>*Try to avoid the regularizations as much as you can at first, aka Dropout, L2/L1 norm, or other fancy ones on the internet.</li>
<li>In my experience, try to avoid using Batchnorm on the last layer.</li>
<li>Double check your final activation layer (in your case, you should have used “softmax” CE since your goal is image recognition/discrimination).</li>
<li>Try to add batchnorm2d on your convolution layers (in case you are using custom model).</li>
<li>Try to normalize your image (in case you haven’t normalized it, you could normalized it [0,1] or [-1,1] etc).<br>
(These 7 steps should basically solved most of your initial issue, which is to make your model very good on your training data)</li>
</ul>
<p><strong>Step B. Make your Model generalized on the unseen data</strong></p>
<ul>
<li>This is where you should do generalizations*. There are many resources on the internet for your starting point to save your time.</li>
<li>Double check your training data’s distribution, make sure that they have similar total images per-class (this is often happening).</li>
<li>If it is still not working, you might gonna need to check your image, make sure that they are recognizable (this is often happening in my prev. projects).</li>
</ul>
<p>Hope it helps, cheers~</p>

username: Sally_Goldin
cooked: <p>Maybe I wasn’t clear. The model classifies the training data more or less perfectly, but the randomly chosen 20% holdout images for validation aren’t very accurate.</p>
<p>There is no problem with balance of samples across the classes.</p>
<p>Also, the training is quite fast (a few seconds per epoch) because I am preloading the training images into the GPGPU.</p>
<p>I am trying some experiments with dropouts now. Definitely slows down the learning. Meanwhile, I welcome other suggestions. Thanks!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Sally_Goldin" data-post="3" data-topic="86502">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sally_goldin/48/18108_2.png" class="avatar"> Sally_Goldin:</div>
<blockquote>
<p>The model classifies the training data more or less perfectly, but the randomly chosen 20% holdout images for validation aren’t very accurate.</p>
</blockquote>
</aside>
<p>In this case your model is overfitting (which is a good start, as your model is at least able to perfectly predict the training set <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ).<br>
You would have to add more regularization or decrease the model capacity e.g. using</p>
<ul>
<li>dropout</li>
<li>data augmentation</li>
<li>weight decay</li>
<li>smaller model</li>
</ul>
<p>If I remember your last topics correctly, you are using image data (and are preloading to save time).<br>
Unfortunately, the preloading onto the GPU won’t allow you to use <code>torchvision.transforms</code> for the data augmentation and you could manually crop, flip, rotate, change the brightness, saturation etc. of the images.</p>

username: Sally_Goldin
cooked: <p>How would a smaller model increase regularization? (My apologies if this seems like a silly or obvious question!) By smaller do you mean fewer layers? Fewer filters per layer?</p>

username: ptrblck
cooked: <p>It wouldn’t increase the regularization, but would decrease the model capacity.<br>
Dropout can also be seen as a method to decrease the model capacity during training, as each training iteration will use a new and “smaller” model than the original one.<br>
I would nevertheless try to increase the regularization first (dropout and data augmentation), before using a smaller model.</p>

username: Sally_Goldin
cooked: <p>I’ve been experimenting with dropout over the past couple of hours. Based on tracking of minimum loss, it seems to have reduced the model’s ability to learn to near zero! BTW I <em>really</em> appreciate how helpful you and the other contributors are on this forum.</p>

username: ptrblck
cooked: <p>Do you mean that your training accuracy is now really bad after you’ve added these dropout layers?<br>
If so, did you specify the drop rate or used the default <code>0.5</code>? Also, where did you add these layers?<br>
Usually you could add dropout between linear layers (not as the last output layer).</p>

username: Sally_Goldin
cooked: <p>With the dropout, the loss was barely declining, even with thousands of epochs. I used a dropout of 0.3, but between the second and third convolutional layers. I can go back and try  a dropout between the linear layers instead.</p>
<p>Meanwhile, I’m currently doing augmentation, manually, to double the number of training samples.</p>

username: paul_mwaniki
cooked: <p>I would stick to one of the pre-trained models such as resnet50 and concentrate on hyperparameter optimization: augmentation, weight decay, batch size, learning rate etc.</p>

username: Sally_Goldin
cooked: <p>Thanks for your suggestion, but based on my admittedly limited understanding of transfer learning, I don’t think the pre-trained models are appropriate for this application. This is not object category recognition or anything similar.</p>

username: fgauthier
cooked: <p>I do not know if this will help but here is a good link to understand dropout better! <a href="https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/" rel="nofollow noopener">https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/</a></p>

username: Sally_Goldin
cooked: <p>Thank you. Excellent article.<br>
I have now added a dropout between the first and second fully connected layer. As expected, the training is much slower, but I am approaching an accuracy level that is appropriate to the task.<br>
Just a side note - the really terrible performance I was seeing earlier was not due to the dropout but to my misunderstanding of BTNug’s suggestion. I had also added a softmax, not realizing that my loss function, Cross Entropy, ALREADY does a softmax transformation. When I removed the extra softmax, the training behaved more reasonably.<br>
Thanks for your suggestions.</p>