Still overfitting, no matter how strong i regularize

milan_kalkenings: <p>Hi all,<br>
my net (a multimodal transformer for text and vision) vastly overfitted the validation set after 4 epochs. I added all kinds of combinations of different dropout strengths, weight decays, learning rate schedulings, and so on…</p>
<p>the only thing that changed is the amount of epochs until the train and eval scores converge to the same values. example:</p>
<p>without any regularization, best scores: epoch 4: train roc auc 99%, val roc auc: 64%</p>
<p>with 40% dropout, weight decay, best validation scores along all epochs: epoch  30: train roc auc 99%, val roc auc: 64%</p>
<p>This is just one example. I tried many different regularization strengths and i always end up having such bad scores on the validation data. The only thing that varies is the amount of epochs to reach that.</p>
<p>the baseline method achieves validation scores of 80% roc auc…so there is definitely space for improvement…</p>
<p>What am I doing wrong?<br>
Do you have any suggestions?</p>
<p>Kind Regards,<br>
Milan</p>

J_Johnson: <ol>
<li>How many samples are in your dataset?</li>
<li>Have you tried using torch.vision transforms to augment the image data during training(shut off during eval)?</li>
<li>During eval, are you using:</li>
</ol>
<pre><code class="lang-auto">with torch.no_grad():
    model.eval()
    ... # eval steps
model.train()
</code></pre>
<p>Sounds like an issue of too little data and no transforms.</p>

milan_kalkenings: <ol>
<li>only about 8000 training observations but the baseline model used the same data.</li>
<li>I use ViT(visual transformer) for extracting a semantic representation. it comes with its own feature extractor that normalizes and rescales the images</li>
<li>yes:<br>
bi_encoder.eval()<br>
for batch in loader:<br>
\twith torch.no_grad():<br>
\t\tproba = bi_encoder(batch)<br>
…</li>
</ol>

ynjiun_wang: <p>how do you generate validation dataset?</p>
<p>does it come from the 8000 training dataset?</p>

milan_kalkenings: <p>no i split them up before the training process and stored both the validation and the training set on my harddrive. they are 100% disjoint</p>

ynjiun_wang: <p>there is no magic in Neural Network except it works in very high dimensional simplex plane for classification or regression (ultimately it’s a linear function fit and region cut). If you are so lucky that your validation dataset happens to be linearly “perpendicular” to your training dataset in the high dimensional space. Then no matter how  you train your model, the validation result will be poor by definition. To prove the point, you might swap your validation set and training set and see the phenomena… (well the above thought is just a conjecture not a theorem for your case, because I have very little information about your model… ; ))</p>

J_Johnson: <p>What percent of each category are in your validiation set?<br>
I.e. 10% cats, 10% dogs, etc.</p>

milan_kalkenings: <p>the baseline model is trained on the same training set and evaluated on the same training data (not by me but by facebook ai) so unfortunately we can withdraw that theory</p>
<p>but i never thought of training it on the validation data and testing it on the training data - what kind of experimental results do you expect and what exactly would we derive from that ?</p>
<p>thank you for the input =)</p>

milan_kalkenings: <p>it’s a binary classification problem and the validation set is perfectly balanced (250 of class 0, 250 of class 1)</p>
<p>the training set is a bit screwed. about 35% are of class 1 there. I try to combat that using the WeightedRandomSampler to achieve implicit oversampling. beforehands I tried undersampling by just discarding the overage randomly. both results in more or less (± 1%) same results</p>

J_Johnson: <p>You have a low sample size. I would really up the augmentation and set the dropout to 50%. Use as many transformers as you can before putting the images into the model.<br>
I.e. ColorJitter, RandomAffine, RandomResizedCrop, RandomHorizontalFlip, RandomPerspective, RandomRotation, GaussianBlur, RandomInvert, RandomPosterize, RandomSolarize, etc.<br>
<a href="https://pytorch.org/vision/stable/transforms.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/transforms.html</a></p>
<p>You can’t go wrong with too many transforms, especially with low data. Nvidia’s research team has found aggressive transforms produce excellent results for ML vision.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://blogs.nvidia.com/blog/2020/12/07/neurips-research-limited-data-gan/">
  <header class="source">

      <a href="https://blogs.nvidia.com/blog/2020/12/07/neurips-research-limited-data-gan/" target="_blank" rel="noopener nofollow ugc" title="02:00PM - 07 December 2020">The Official NVIDIA Blog – 7 Dec 20</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/366;"><img src="https://blogs.nvidia.com/wp-content/uploads/2020/12/research-blog-NeuralIPS-styleGAN-hero3-1280x680-1.jpg" class="thumbnail" width="690" height="366"></div>

<h3><a href="https://blogs.nvidia.com/blog/2020/12/07/neurips-research-limited-data-gan/" target="_blank" rel="noopener nofollow ugc">NVIDIA Research Achieves AI Training Breakthrough | NVIDIA Blog</a></h3>

  <p>Data augmentation technique enables GAN to emulate artwork from a limited dataset — and opens up potential applications in healthcare.</p>

  <p>
    <span class="label1">Est. reading time: 4 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>One other question, how many parameters are you using for the entire model? For example:<br>
<code>print(sum(p.numel() for p in model.parameters()))</code></p>

milan_kalkenings: <p>the number of parameters is really high. in total it consists of 2 bert models plus a linear layer of size 768x2x2.<br>
so that’s still a huge capacity.</p>
<p>i didn’t want to decrease the models depth, because i wanted to take advantage of the pretrained parameters (i load both of them from the huggingface model hub)</p>
<p>tried to lower the capacity usin weight decay and dropout</p>
<p>should i remove layers?</p>

J_Johnson: <p>Right. I think for this smaller dataset, you’re not really justified in having such a large model. Larger models may tend to overfit faster, unless you really up the dropout and augmentation to counter that tendency.</p>

milan_kalkenings: <p>hmm i performed some runs with smaller networks (less transformer blocks/less depth) but we still have no score improvement… <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji" alt=":confused:"></p>

ynjiun_wang: <aside class="quote no-group" data-username="milan_kalkenings" data-post="1" data-topic="133976">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/milan_kalkenings/48/42748_2.png" class="avatar"> milan_kalkenings:</div>
<blockquote>
<p>the baseline method achieves validation scores of 80% roc auc…so</p>
</blockquote>
</aside>
<p>According to your conversations, my limited understanding is your training dataset skew (35% 1 and 65% 0) and your validation dataset skew (50% 1 and 50% 0), plus the “best performance” of baseline model performance is 80% (?given the same validation dataset or different validation dataset? what’s the validation dataset skew for baseline model? also 50%/50%? or different?)</p>
<p>Then your model actually already achieved almost the “best performance”  given the training dataset skew: =&gt; train roc 99% means your model will predict pretty much similar to what your training dataset skew is, which is 35% 1 and 65% 0 (This is what it means by training =&gt; you tell the model what the universe looks like: 35%/65% skew!) Then when you validate the dataset with 50%/50% skew with 80% best performance =&gt; what your model best case prediction statistics is (35% 1 and 65% 0) * 80% =&gt; (15% 0 False negative) * 80% =&gt; 85% * 80% val roc =&gt;  68% val roc</p>
<p>You reported that you achieved 64% val roc which is already very close to 68%.</p>
<p>If you reverse val and train dataset (i.e. use 50%/50% skew to train the model and test it with 35%/65% skew), then the result val roc might be approaching similar result of 68% val roc.</p>
<p>The extreme case would be using training skew of 1%/99% or even 0%/100%, then I think you might know the result of val roc with 50%/50% will be very poor…</p>
<p>Again, this is a super simplified analysis and just try to reveal the nature of “linearity” of current NN technology limitation: NN is good for interpolation and not good at extrapolation… ; (</p>
<p>So under the assumption that your network and training has no bugs, then I would suggest to “tune your training set to be roughly 50%/50% skew”, then test again to see if your val roc improve or not…</p>
<p>(Disclaimer: again the above thoughts is not a theorem but a conjecture that deserves further validation ; ))</p>

milan_kalkenings: <ol>
<li>yes the training data has a 35%/65% imbalance, the validation data is perfectly balanced.</li>
<li>yes the baseline model is (according to the authors) evaluated on the exact same validation data</li>
</ol>
<p>but effectively i have a less skewed dataset due to oversampling (and i tried undersampling before to achieve a perfectly balanced training set) but I achieve more or less (maybe 1% difference in both cases) the same results</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji only-emoji" alt=":confused:"></p>

OrielBanne: <p>what’s your optimizer? optimizer parameters?</p>

milan_kalkenings: <p>I tried Adam and AdamW with varying learning rates from a range between 10**-2 to 10**-6, and varying weight decays.</p>
<p>turns out that in my case an initial learning rate of 10**-4 seems to be the best solution.</p>
<p>moreover, i use a learning rate scheduler that lowers the learning rate via lr_i = 0.9*lr_{i-1} but it doesn’t seem to have an impact on the model performance and overfitting</p>

OrielBanne: <p>can you please try it with SGD and see if you get the same overfitting? plus can you remove the scheduler and the learning rate update you have used?</p>
<p>1 - I would loose all learning rate scheduling and see if there is a difference between both optimizers with a few different parameters<br>
2 - once you know which one is best in this case add back the LR.</p>

milan_kalkenings: <p>i will perform some runs with sgd, brb</p>

OrielBanne: <p>also - what are the results when all categories are the same size? it’s a classification problem, right? try out when all categories have the same size of dataset, 4 epochs or so.</p>