What is the best way to validate a model?

morphism: <p>Dear everybody:</p>
<p>Usually, we separate input data into three, which are training set, validation set, test set in deep learning.</p>
<p>For each epoch, I want to do the best way to get a better model using validation set.</p>
<p>For example, for each epoch, after finishing learning with training set, I can select the model parameter which has the lowest loss w.r.t. validation set by saving the model parameter each time when the loss w.r.t. validation set has lower value than the<br>
lowest loss by then.</p>
<p>Also, I can select the lowest validation loss while using different validation set each time<br>
by mixing training set and validation set and randomly selecting some fraction of the mixture as a new<br>
validation set</p>
<p>Is this the best or at least recommended way to get a better or the best model parameter???</p>
<p>I will appreciate if you can help me with this problem<br>
Thank you</p>

ptrblck: <aside class="quote no-group" data-username="morphism" data-post="1" data-topic="110716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morphism/48/28101_2.png" class="avatar"> morphism:</div>
<blockquote>
<p>For example, for each epoch, after finishing learning with training set, I can select the model parameter which has the lowest loss w.r.t. validation set by saving the model parameter each time when the loss w.r.t. validation set has lower value than the<br>
lowest loss by then.</p>
</blockquote>
</aside>
<p>Yes, this is a common technique and is called early stopping. You would store the model’s <code>state_dict</code> for the lowest validation loss (or highest validation accuracy).</p>
<aside class="quote no-group" data-username="morphism" data-post="1" data-topic="110716">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morphism/48/28101_2.png" class="avatar"> morphism:</div>
<blockquote>
<p>Also, I can select the lowest validation loss while using different validation set each time<br>
by mixing training set and validation set and randomly selecting some fraction of the mixture as a new<br>
validation set</p>
</blockquote>
</aside>
<p>No, you shouldn’t do it, since mixing the training and validation set would give you a better “validation” loss than you would expect to see on “unseen” data. After your training is finished, the “validation” loss (which is not really the validation loss anymore) would most likely be much lower than the test loss.</p>

morphism: <p>Dear ptrblck:<br>
Thank you so much for your kind answer!!!<br>
Now I understand the validation process much better</p>
<p>May I ask you one more question? Actually it may be the same question again.<br>
(Sorry if it is the same question…)</p>
<p>Actually, what I wanted to do is as follow:</p>
<p>I separate input data into two parts, training set and test set.</p>
<p>And for each epoch, I randomly jumble training set and separate it into two parts,<br>
and I train the model using the first part and, after that, evaluate the model using the second part by calculating validation loss with this second part and trace when it has the minimum validation loss</p>
<p>and when finishing epochs, get the model parameter at that epoch having lowest validation loss.</p>
<p>Why I wanted to do this is, because I randomly jumble training set and select newly created validation set inside the training set each epoch and hope it may make sense theoretically…</p>
<p>So, should I not do this???</p>
<p>Have a nice day…</p>

BramVanroy: <p>No, you should not do that because your model will be biased. It means that in epoch 1 some of the data will be used to train, but then in the second epoch that data may be part of the validation. The model will perform very well on that data then as it has seen it before, which you do not want. The validation set should always be data that the model has never seen during training.</p>
<p>What you seem to want is something like k-fold cross-validation. You can Google it, there’s quite a lot of information about it on the internet.</p>

morphism: <p>Thank you, Bram.<br>
Now I understand fully.<br>
Validation should not be a part of training…(which I didn’t know correctly)<br>
whereas cross validation is permitted…</p>
<p>Have a nice day and see you again<br>
Thank you and<br>
sorry that I can select only one answer(solution) to my question</p>

Abhinav_Gupta: <p>I have a slightly different question on the same topic. Say I find out out that epoch 25 gives the minimum loss on validation set. I was wondering if I should train another model using training + validation data with 25 epochs and call this my final model?</p>
<p>I want to do this so that I am able to use maximum possible data for training.</p>