## 2 Theory and Calculation
### The K-Fold Cross Validation  
In machine learning, the available data set is usually divided into subsets for training, validation, and testing. After the model's hyperparameters have been set up (for example, the number of hidden layers, number of neurons in each layer, learning rate, etc. ), the training data set, generally made up of 60% to 80% of the data, is used to train the model itself. If the model performs satisfactorily on the training data, then it is run on the validation data set, which usually ranges from 5% to 20% of the data. The validation data set helps to provide an assessment of the model's capability. If the error (i.e., the mean squared difference between the predicted and experimental data, \(MSE\)) on the validation data set significantly increases (or decreases) compared to the training error, then an overfitting (rather than an underfitting) condition has occurred. The test data set, typically consisting of 5% to 20% of the data, is employed to complete the analysis and assessment of the model prediction capacity. The procedure described above, considered as \"conventional\", poses two issues: On the one hand, the subdivision of the data into subsets with different purposes involves the risk of leaving out some relevant trends from the training data set and, on the other hand, a model can lead to a lower prediction error on training data but at the same time to high test error due to data sample variability [(82)]. An alternative to the conventional approach is the so-called k-fold cross validation (CV). It is a statistical technique that allows a more accurate estimation of a model's performance and then to compare and select a model for a given predictive modeling problem [(83; 84; 85; 86)]. It is strongly recommended to follow such a procedure in the case of a relatively small data set [(81; 78)]. It involves a random division of data into k folds of equal sizes: One of these folds is used as a test set to evaluate the model's performance and others (\(k\)-1) are grouped to form the training data set (Figure 2). The training and test phases are performed k times and each time a fold is assumed as a test set while the remaining ones are used to train the model. Performing multiple rounds of CV with different subsets from the same data allows a significant reduction in sample variability: All data are included in a training set (\(k\)-1) times, and in a test set once. Usually, the value of k is set to 5 or 10, as these values have been shown to result in more accurate estimations of a model's capacity [(82; 86)]. By calculating the average of the testing error, \(MSE_{i}\), over these multiple rounds, an estimation of the model's predictive performance is obtained:  
\[MSE_{cv}=\frac{1}{k}\sum\
olimits_{i=1}^{k}MSE_{i}=\frac{1}{k}\sum\
olimits_{ i=1}^{k}\left(\frac{1}{M}\sum\
olimits_{l=1}^{M}(t_{l}-y_{l})^{2}\right)_{i}, \tag{4}\]  
where \(M\) is the number of samples in the \(i\)-th fold, \(t_{l}\) is the experimental targets, and \(y_{l}\) is the ANN-calculated outputs. Including a measure of the variance of the skill scores is also recommended [(82; 83)], such as the standard deviation (i.e., the standard deviation is the square root of the variance) of \(MSE_{i}\):  
\[\sigma_{cv}=\sqrt{\frac{1}{k-1}\sum\
olimits_{i=1}^{k}\left|MSE_{i}-MSE_{cv} \right|^{2}}. \tag{5}\]  
However, to find the hyperparameters of the model that best fit the experimental data and that satisfactorily generalize the problem, it is necessary to define a model selection procedure. This consists of designing several neural networks by assigning numerical values, within a range, to one of the hyperparameters and by fixing those of the remaining ones. This operation is repeated by varying at least one pair of hyperparameters. Resulting models have to adapt well to the data used for training. Subsequently, a k-fold CV is performed on each of the models thus defined.