Overfit for validation set and test set?

username: Gears_Gears
cooked: <p>Hi!</p>
<p>May I get some advice for the following situation please?:<br>
I’m doing cross-validation. In each cross-validation round, I return the lowest training loss and the lowest validation loss of that epoch. Then, I choose the model for testing by lowest validation loss.</p>
<p>The problem is: while my test loss (0.05) is close to the validation loss (0.04), in each validation round, my validation error is always 1.5 to 2 times that of the training error.</p>
<p>My question is:<br>
1 Is this overfitting? It looks like my model overfitting for the cross-validation round (because training loss much smaller than testing loss), but not-very-overfitting for the testing round (because the validation loss is close to testing loss))? This is confusing because the data used for testing and cross-validation are the same. Perhaps the model just generalizes poorly in general?<br>
2 What should I do? My model is for regression, with input being a long 1d vector and output being a single number. (I have early stopping)<br>
I used dropout but that made the model converge at high loss<br>
I tried to ‘strengthen’ the regularization by reducing early stopper patience from 5 to 3, and got higher loss again</p>

username: Gears_Gears
cooked: <p>My training loss is blue, and my testing loss is orange.<br>
In each validation round, the loss look like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/2/a20c3a00d3e174ab2d7a17afe36505e39ac020c7.png" data-download-href="https://discuss.pytorch.org/uploads/default/a20c3a00d3e174ab2d7a17afe36505e39ac020c7" title="Screenshot 2023-05-31 at 11.00.59 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a20c3a00d3e174ab2d7a17afe36505e39ac020c7_2_687x500.png" alt="Screenshot 2023-05-31 at 11.00.59 AM" data-base62-sha1="n7xxP0S0iNrdtieQdmhUh3jgC2z" width="687" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a20c3a00d3e174ab2d7a17afe36505e39ac020c7_2_687x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a20c3a00d3e174ab2d7a17afe36505e39ac020c7_2_1030x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/2/a20c3a00d3e174ab2d7a17afe36505e39ac020c7_2_1374x1000.png 2x" data-dominant-color="BABAB9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-05-31 at 11.00.59 AM</span><span class="informations">1684×1224 156 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It looks my model converged, but by the time I stop it, the model has overfit (because train error drop below the test error). It looks like the only thing I could make the model stop earlier is to just do ‘more’ early stopping</p>
<p>But ‘more’ early stopping (I am counting the number of times the model doesn’t give lower validation loss) means lower patience, and my patience is already 5 (as described above, going down to patience=3 only makes me stop at higher loss, and I don’t think patience=1 is a good idea, since the model’s loss might go up and down)</p>