## 2 Materials and Methods
### Features Selection  
In order to identify the most informative features and to reduce complexity for improving performance [22], three different features selection techniques were applied: F-Score, Recursive Feature Elimination (RFE) and model-based selection.  
The feature selection method based on the F-Score seeks to determine which independent variables have the greatest ability to explain variability in the target [23; 24; 25; 26]. Variables with a high F-Score are those that are most strongly related to the target, while those with a low F-Score may not have a significant relationship and may be excluded from the model. In addition, the \(p\)-Value measures the statistical significance of the F test. A low \(p\)-Value (less than 0.05 for this work) indicates that the characteristic has a significant correlation with the target, suggesting that it is important in the model.  
The RFE method aims to identify the most important variables for the model, by recursively removing those that are least significant [27; 28; 29]. RFE is particularly useful for improving the accuracy of a model by reducing the dimensionality of the dataset. Once the model is trained, RFE ranks each feature based on a metric that depends on the type of model used. In the case of Linear Regression, the importance of a feature can be determined by its coefficients.  
Model-based selection is another feature selection approach that exploits the ability of a model to calculate the importance of features during model training [30; 31; 32]. In the current work, the selected model is Random Forest. Once the Random Forest is trained, each tree in the model calculates how much each feature contributes to the final prediction. The importance of a feature is generally calculated using the decrease in purity of each tree node during its construction.  
\begin{table}
\begin{tabular}{c l} \hline \hline
**Model** & \multicolumn{1}{c}{**Search Space**} \\ \hline Linear Regression & \{} \\ Ridge Regression & alpha: [0.1, 1, 10, 100] \\ Lasso Regression & alpha: [0.01, 0.1, 1, 10] \\ Random Forest & n\_estimators: [50, 100, 200] \\  & max\_depth: [None, 10, 20] \\ Gradient Boosting & n\_estimators: [50, 100, 200] \\  & learning\_rate: [0.01, 0.1, 0.2] \\ Support Vector Regression & C: [0.1, 1, 10] \\  & kernel: [linear, bfl] \\  & max\_depth: [None, 10, 20, 30] \\ Decision Tree & min\_samples\_split: [2, 5, 10] \\  & min\_samples\_leaf: [1, 5, 10] \\ k-Nearest Neighbours & n\_neighbors: [3; 5; 7; 10] \\ \hline \hline \end{tabular}
\end{table}
Table 5: For each model, the following search spaces weretested.  
After applying the feature selection techniques, only the features that were considered important by at least two feature selection methods were chosen. For this reason, a top ten list was drawn up. To obtain the best regression performance, PCA (Principal Component Analysis) was performed on the best selected features.  
PCA, also known as Karhunen-Loeve expansion, is the oldest multivariate technique developed before the Second World War. PCA is a classical feature extraction and data representation technique [33; 34]. It analyzes a dataset that represents variations described by different dependent variables, which are generally interrelated [35; 36; 37]. Its purpose is to extract important information from the matrix and express this information as a set of new orthogonal variables called principal components. The correlation between a principal component and a variable quantifies the extent of shared information. Within the PCA framework, this correlation is referred to as a loading (or score). Notably, the sum of the squared correlation coefficients between a given variable and all principal components equals 1. Consequently, squared loadings offer a more intuitive interpretation than raw loadings, as they represent the proportion of the variable's variance explained by the components.