## 2 Materials and Methods
### Machine Learning Methods
#### 2.1.6. Validation Process and Criteria Used for Model Selection  
Validation of the model is a critical stage in machine learning. There are several different validation techniques (e.g., holdout, random subsampling, resubstitution, \(k\)-fold cross-validation, and leave-one-out cross-validation) [69]. In this work, we use random  
Figure 5: Flowchart of the SVR algorithm. subsampling and repeated \(k\)-fold cross-validation. Random subsampling or Monte Carlo cross-validation is based on random splitting of the dataset into several training and test sets. A prediction model is obtained for each training dataset, and is evaluated on the corresponding test dataset. To analyse the performance of a model, the average values of the goodness-of-fit measures are considered (see Figure 6a). In \(k\)-fold cross-validation, the dataset is randomly partitioned into \(k\) subsamples of equal size. At each iteration, one of these \(k\) subsamples is used as a test set, and the remaining \(k-1\) subsamples as a training set. The performance of this approach is obtained based on the average values of the goodness-of-fit measures at each iteration (see Figure 6b). In repeated \(k\)-fold cross-validation, this process is repeated several times with different random data partitions, and the mean performance across all repeats is considered. To assess the performance of the models presented in this study, the following goodness-of-fit measures were considered (where \(y_{i},\ i=1,2,\ldots,n\), denote the observed values; \(\hat{y}\) is the mean of the observed values; \(\hat{y}_{i},\ i=1,2,\ldots,n\), are the predicted values; and \(d\) is the number of predictive variables in the model):  
* Relative error, \(\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i=1}^{n}(y_{i}-\hat{y})^{2}}\). * Determination coefficient, \(R^{2}=1-\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i=1}^{n}(y_{i}-\hat {y})^{2}}\). * Mean squared error, \(\text{MSE}=\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{n}\). * Root mean square error, \(\text{RMSE}=\sqrt{\frac{\sum_{i=1}^{n}(y_{i}-\hat{y}_{i})^{2}}{n}}\). * Mean absolute error, \(\text{MAE}=\frac{\sum_{i=1}^{n}|y_{i}-\hat{y}_{i}|}{n}\). * Mean absolute percentage error, \(\text{MAPE}=\frac{100}{n}\sum_{i=1}^{n}|\frac{y_{i}-\hat{y}_{i}}{y_{i}}|\). The determination coefficient is one of the most frequently used measures of the degree of linear correlation between two variables. This measure can be interpreted as the proportion of variability explained by the model. Note that the determination coefficient in a linear least-squares regression with an intercept is equivalent to the square of the correlation coefficient and, therefore, takes values between zero and one, where a value of one indicates perfect correlation. However, outside of this context, the determination  
Figure 6: Random subsampling and \(k\)-fold cross-validation procedures. (**a**) A 10 random subsampling procedure by randomly partitioning the sample into 75% for training and 25% for testing; (**b**) \(k\)-fold cross-validation procedure, example with \(k=4\). coefficient may be negative. The RMSE, MAE and MAPE are also very suitable measures for comparing the performance of the models, where lower values indicate a better fit of the model. The MAPE and MAE are robust to the effects of outliers, due to the use of absolute values in the formulae, while the RMSE is more sensitive. Nevertheless, there are some circumstances in which the use of the RMSE is more beneficial (for example, when the errors are normally distributed), meaning that it would be hard to argue that one measure is better than another. Hence, a combination of these metrics is used to analyse the performance of the models. One of the main concerns in machine learning is overfitting or underfitting the models. Underfitting occurs when the model is not able to represent the training data or new inputs that are not included in the training dataset.