My model works on small random subsets(12k) of the dataset but not the whole dataset(200k)

sepid_kh: <p>Hi,<br>
I have a dataset of 200k extracted patches from medical images. My model works satisfying on a random subset of the dataset(e.g. 12k patches).<br>
However, it looks like it does not work on my whole dataset, and overfitting happens. I have tried different learning rates and augmentations, but it still does not work.<br>
Please see attached diagrams of training and validation accuracy and losses after 50 epochs of training applying the whole dataset(including 200k patches).<br>
I cannot understand why the accuracy for validation started too high from the beginning?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/0/60d160f2f9258c8a5682d23de9960167c7daa448.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/60d160f2f9258c8a5682d23de9960167c7daa448" title="picmesal1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d160f2f9258c8a5682d23de9960167c7daa448_2_690x372.jpeg" alt="picmesal1" data-base62-sha1="dOuqAYKFO8So5xoL62GxCP4xZDa" width="690" height="372" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d160f2f9258c8a5682d23de9960167c7daa448_2_690x372.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d160f2f9258c8a5682d23de9960167c7daa448_2_1035x558.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/0/60d160f2f9258c8a5682d23de9960167c7daa448_2_1380x744.jpeg 2x" data-dominant-color="F3F1EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">picmesal1</span><span class="informations">3301×1781 337 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also, as seen in another diagram, by applying a very low learning rate 10e-6, it looks like overfitting happens the model does not learn at all.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d" title="picmesal"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d_2_690x363.jpeg" alt="picmesal" data-base62-sha1="vfmDl1iOmVHQ2DNB4ThckcPslop" width="690" height="363" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d_2_690x363.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d_2_1035x544.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db00082e7bc2e2ca8193a6cc8d8c20c1dd42680d_2_1380x726.jpeg 2x" data-dominant-color="F3F2F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">picmesal</span><span class="informations">3274×1725 261 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would appreciate it if anybody could help me.</p>

sepid_kh: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Could you please help me If you have any idea about this?</p>

tiramisuNcustard: <p>Can you post your code, where you think the problem might be? This will help him to figure out what might be wrong.</p>

sepid_kh: <p>My model is based on the ResNet50 backbone. I am not sure where the problem might be. I think the model faces overfitting after applying more data. I was wondering if someone has experienced a similar problem or has any suggestions.</p>