Doubt about cross-validation with training, validation and test set

PedroG: <p>hi all,<br>
I have a question where I am apparently not able to find any answer.<br>
Let’s say I have a dataset which is relatively small and I want to be able to test on the entire dataset to reduce the bias of my model.<br>
I thought I could use something like k-fold cross-validation, but no matter wher I look, i only find the case where for each fold, the data is split in only training and test set. What I am interested in is to have for each fold a training, validation AND test data, where I train the model on the test data, determine when to stop training on the validation data and then test on the test data. I would then report the average of the scores obtained on the different test sets.<br>
However, I am unsure on how to split the data and if the mentioned procedure is actually correct.<br>
Would this for example be correct? (I fix [p] to be the validation set) and test on the other different test sets)<br>
Dataset = [o, d, p, x]<br>
Fold1: Train = [o, d], Validation = [p], Test=[x]<br>
Fold2: Train = [o, x], Validation = [p], Test=[d]<br>
Fold3: Train = [d, x], Validation = [p], Test=[o]<br>
Thank for help!</p>

ptrblck: <p><a class="mention" href="/u/rasbt">@rasbt</a> published <a href="https://sebastianraschka.com/faq/docs/evaluate-a-model.html">this great post</a> a while ago where he explains cross validation in detail (you should also check other resources from his blog).<br>
To perform the actual splitting you could use e.g. scikit-learn’s methods.</p>

PedroG: <p>Hi ptrblck,<br>
thanks for the useful link!<br>
But I am still not sure if what I want to do is correct. I want to use k-fold CV for model-evaluation, I am not interested in any hyper-parameter selection. However, because I am using neural networks, I still need a validation set to decide when to stop training.<br>
In the slides of the link you posted, this case is covered:<br>
Dataset = [o, d, p, x]<br>
Fold1: Train = [o, d, p], Test=[x]<br>
Fold2: Train = [d, p, x], Test=[o]<br>
Fold3: Train = [p, x, o], Test=[d]<br>
Fold4: Train = [x, o, d], Test=[p]<br>
from which I could estimate the performance of my model with fixed hyper-parameters as the average of the performance on the test sets across the different folds.<br>
But no validation set is used here and I am wondering if I should keep the validation set fixed across the different folds (e.g. [p] as in my previous post) or if this should vary as well or also use some inner loop.<br>
What do you think?</p>

my3bikaht: <p>IMO, cross-validation is flawed by the definition. So, If possible, test set should be fully separated from  training loop. If not, then I’d choose scenario 3: run outer loop picking test subset, then run inner loop on remaining data training model via cross-validation.</p>
<p>Yet, I’d look first into possibility of generating additional samples, either fully synthetic or augmented.</p>

PedroG: <p>hi Sergey,<br>
unfortunately, generating additional samples is not a possible option for my application.<br>
So you think this approach<br>
Dataset = [o, d, p, x]<br>
Fold1: Train = [o, d], Validation = [p], Test=[x]<br>
Fold2: Train = [o, x], Validation = [p], Test=[d]<br>
Fold3: Train = [d, x], Validation = [p], Test=[o]<br>
is good? I only want to use [p] for checking when to stop training the network on [o,d], for example.</p>
<p>or is it better to do like this?<br>
Dataset = [o, d, p, x]<br>
Fold1: Train = [o, d], Validation = [p], Test=[x]<br>
Fold2: Train = [d, p], Validation = [x], Test=[o]<br>
Fold3: Train = [p, x], Validation = [o], Test=[d]<br>
Fold4: Train = [x, o], Validation = [d], Test=[p]<br>
Thank you!</p>

SebGruber1996: <p>Both of what you have suggested gives an unbiased estimate of the test set error, which is the most important aspect.<br>
But your second approach has probably lower variance error, since you are also switching up the validation set.</p>
<p>But keep in mind your procedure differs from cross-validation and I also would not call it that way, or else you are confusing people.<br>
At least in your second approach, I can tell what you are doing is nested resampling, where the outer resampling is cross validation and the inner resampling is holdout (see <a href="https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html" rel="noopener nofollow ugc">here</a> for more)</p>