GridSearchCV for neural networks that allows evaluation of the validation set after every epoch

### Describe the workflow you want to enable

I would like to be able to perform a grid search of hyperparemeters for a neural network using crossvalidation such as with GridSearchCV. I was able to set everything up in Scikit-Learn and the KerasRegressor wrapper but I then found out that my model was not evaluating on the validation set until the end of the number of epochs I set. This does not enable to track training and validation loss throughout training hence, assessing the performance of my model and taking action to e.g., use early stopping to prevent overfitting.

### Describe your proposed solution

enable GridSearchCV to pass the validation data to the estimator during fitting so validation loss can be tracked throughout training. Actually, If I understand how GridSearchCV works correctly, I think that perhaps an entire new DeepLearningGridSearchCV class would be more appropriate because I think that GridSearchCV follows (at least roughly) these steps:

1. Create data folds using CV strategy provided.
2.  fit training data to estimator (pipeline including scaling and model in my case)
3.  evaluate on validation data.

The process I was looking for would merge steps 2 and 3 so that:

1. Create data folds using CV strategy provided.
2.  fit training and validation data to estimator so validation data can be used for model assessment after every epoch throughout training

### Describe alternatives you've considered, if relevant

 In my case, I was training an autoencoder so both my X and y data needed to be scaled using the scaling factors calculated on the training data. I've created a custom scoring function to do so but my main problem seemed to be requiring GridSearchCV to pass the validation data during training  andhaving to pass more than one output from one step to another within the pipeline.

### Additional context

Has anyone done anything similar or is this something that would be possible?

Thanks

COMMENT:
no, I want to pass my entire dataset, then GridSearchCV would perform the splitting as per the cross-validation strategy I set (group k fold in this case). Then my pipeline includes a custom scaler and an autoencoder that I am trying to optimise hyperparameters for. For each iteration of the cross-validation process, I want to be able to scale the training and vaidation set (validation set scaled with training mean and std as I think it's already done by default within the piipeline and GridSearchCV process) and pass these two sets of the data to the autoencoder fitting step. The autoencoder is going to be trained on the trianing set but after every completed epoch, I want to evaluate how the current instance of my model performs on the validation set. This would allow me to assess training loss vs validation loss curves so I can decide to e.g., use early stopping to terminate training based on validation loss not improving or getting worse to prevent overfitting and recover the best weights.

Instead, the current GridSearchCV does not allow to do so since training is completed and then the final model is evaluated on the validation set. In my case for example, it means that I would need to run my model training based on a fixed number of epochs or use earlystopping based on training loss. This makes it harder to control for overfitting during training and is also suboptimal for training time as a model may be training for 100 epochs when validation loss is not improving since epoch 20. Actually validation loss might increase throughout training, so the returned weights of my model are not the ones that would the best so not only am I wasting time but also getting a suboptimal model.

Thank you for your reply and I hope that this message makes sense and clarifies my issue. I ended up creating a custom script with loops that I am currently using but I am sure that I am not making the most of my computer and the parallelism implented in sci-kit learn, which would make things faster for me

Labels: New Feature, Needs Triage