Bigger dataset not helping in accuracy for BERT model

Utkarsh_Upadhyay: <p>I am pre-training a BERT model on a custom dataset. The accuracy of the model gets stuck at ~ 20%. Since it is recommended that one should try to overfit a model on a small dataset before moving to a bigger dataset, I tried it and my model easily overfits on a small dataset (100 samples) you can see the plots below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff768423176db0898faa3a78678acbb5d671650e.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff768423176db0898faa3a78678acbb5d671650e" title="Validation Accuracy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff768423176db0898faa3a78678acbb5d671650e_2_690x362.png" alt="Validation Accuracy" data-base62-sha1="ArVFPq5trqIBTm5kukDAbiW2Ce2" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff768423176db0898faa3a78678acbb5d671650e_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff768423176db0898faa3a78678acbb5d671650e_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/ff768423176db0898faa3a78678acbb5d671650e_2_1380x724.png 2x" data-dominant-color="F9FAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation Accuracy</span><span class="informations">2528×1328 372 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/2/72c6023b9e1f5412c17f0838f55301cb510e6523.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/72c6023b9e1f5412c17f0838f55301cb510e6523" title="Training Accuracy"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72c6023b9e1f5412c17f0838f55301cb510e6523_2_690x362.jpeg" alt="Training Accuracy" data-base62-sha1="gnkF7XJ91JRJ8wIozbSGLdxTFJx" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72c6023b9e1f5412c17f0838f55301cb510e6523_2_690x362.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72c6023b9e1f5412c17f0838f55301cb510e6523_2_1035x543.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/2/72c6023b9e1f5412c17f0838f55301cb510e6523_2_1380x724.jpeg 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training Accuracy</span><span class="informations">2528×1328 249 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10c04fe7002d73280e2f83fe5fe41570fa2551ed.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/10c04fe7002d73280e2f83fe5fe41570fa2551ed" title="Validation Loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c04fe7002d73280e2f83fe5fe41570fa2551ed_2_690x362.jpeg" alt="Validation Loss" data-base62-sha1="2obFjpHcckoPLJ5Kv35zqtnfamp" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c04fe7002d73280e2f83fe5fe41570fa2551ed_2_690x362.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c04fe7002d73280e2f83fe5fe41570fa2551ed_2_1035x543.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/0/10c04fe7002d73280e2f83fe5fe41570fa2551ed_2_1380x724.jpeg 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation Loss</span><span class="informations">2528×1328 258 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e27fa80b0354672c8b402ec97f217f6eda0d255d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e27fa80b0354672c8b402ec97f217f6eda0d255d" title="Training Loss"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e27fa80b0354672c8b402ec97f217f6eda0d255d_2_690x362.jpeg" alt="Training Loss" data-base62-sha1="wjHptDuY806vp1qDBI61rcmyYKV" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e27fa80b0354672c8b402ec97f217f6eda0d255d_2_690x362.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e27fa80b0354672c8b402ec97f217f6eda0d255d_2_1035x543.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e27fa80b0354672c8b402ec97f217f6eda0d255d_2_1380x724.jpeg 2x" data-dominant-color="FCFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training Loss</span><span class="informations">2528×1328 198 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But when I increase the size of the dataset (250K samples) the accuracy doesn’t improve after a certain point, as you can see below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/9/796ac0569b5d78cec31ef4aac51f136fac0e9f5f.png" data-download-href="https://discuss.pytorch.org/uploads/default/796ac0569b5d78cec31ef4aac51f136fac0e9f5f" title="Validation Accuracy Full"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/796ac0569b5d78cec31ef4aac51f136fac0e9f5f_2_690x362.png" alt="Validation Accuracy Full" data-base62-sha1="hk6ueeYQSe93QOig8scIcYRUt3x" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/796ac0569b5d78cec31ef4aac51f136fac0e9f5f_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/796ac0569b5d78cec31ef4aac51f136fac0e9f5f_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/9/796ac0569b5d78cec31ef4aac51f136fac0e9f5f_2_1380x724.png 2x" data-dominant-color="FEFDFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation Accuracy Full</span><span class="informations">2528×1328 174 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48" title="Training Accuracy Full"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48_2_690x362.png" alt="Training Accuracy Full" data-base62-sha1="u3jOQvLbQd7BeDC5l5ESDP8aEl2" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/2/d2a148ef9f1874fef5ea47862ad3d92e7e65ba48_2_1380x724.png 2x" data-dominant-color="FDFAF8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training Accuracy Full</span><span class="informations">2528×1328 237 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e05849e158aa56fe8b5026799405b50d935a0e02.png" data-download-href="https://discuss.pytorch.org/uploads/default/e05849e158aa56fe8b5026799405b50d935a0e02" title="Validation Loss Full"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e05849e158aa56fe8b5026799405b50d935a0e02_2_690x362.png" alt="Validation Loss Full" data-base62-sha1="w0E6S06dRuy9GRfx8sBGiGDCyRA" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e05849e158aa56fe8b5026799405b50d935a0e02_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e05849e158aa56fe8b5026799405b50d935a0e02_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e05849e158aa56fe8b5026799405b50d935a0e02_2_1380x724.png 2x" data-dominant-color="FEFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Validation Loss Full</span><span class="informations">2528×1328 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9e5d010cabfcb789e6dd840b115014792b20bbed.png" data-download-href="https://discuss.pytorch.org/uploads/default/9e5d010cabfcb789e6dd840b115014792b20bbed" title="Training Loss Full"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e5d010cabfcb789e6dd840b115014792b20bbed_2_690x362.png" alt="Training Loss Full" data-base62-sha1="mAWHrtklJ30HrlydT0vgASFt4F7" width="690" height="362" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e5d010cabfcb789e6dd840b115014792b20bbed_2_690x362.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e5d010cabfcb789e6dd840b115014792b20bbed_2_1035x543.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/e/9e5d010cabfcb789e6dd840b115014792b20bbed_2_1380x724.png 2x" data-dominant-color="FDFBFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Training Loss Full</span><span class="informations">2528×1328 212 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I also tried with bigger-size models but still, the same problem persists. Please any suggestions in this regard would be very helpful. Also, let me know if more details are required.</p>