## 5 Results and Discussion
### Feature-Importance Results  
#### 5.2.1 Tree-Based Feature-Importance Results  
ML models can be seen as mathematical functions that map given inputs to desired outputs. Their performance depends on numerous factors, including the nature of the data, the underlying algorithm, and the chosen hyperparameters. Most algorithms offer hyperparameters that let users tune how the model learns, aiming for a balance between accurate predictions and generalizability. This study focuses on two key hyperparameters for our tree-based model: max_features and max_depth of the random forest regressor. This procedure can be generalized for other algorithms and their hyperparameters. Figure 10 shows the \(R^{2}\) heatmap of different combinations of'max_features' and'max_depth' values, with each color tone representing a clustered performance region. The dominant cells marked with * are those whose (max_features, max_depth) settings yield \(R^{2}\) performance at least as good as all configurations to their right (increasing'max_features') and below (increasing'max_depth'). Focusing on these dominant points ensures that we maintain or improve model accuracy while avoiding unnecessary complexity. In practice, selecting a dominant combination helps minimize training overhead and mitigate \begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Parameter** & **Best 1** & **Best 2** & **Best 3** & **Best 4** & **Best 5** \\ \hline apparent\_temperature & ✓ & ✓ & & & ✓ \\ \hline diffuse\_radiation\_instant & ✓ & ✓ & ✓ & & \\ \hline wind\_speed\_10m & & & & ✓ & ✓ \\ \hline direct\_radiation\_instant & & & & ✓ & \\ \hline terrestrial\_radiation\_instant & & & & ✓ & \\ \hline Average \(R^{2}\) & 0.9097 & 0.9093 & 0.9091 & 0.9090 & 0.9086 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The top five agents by \(R^{2}\) obtained from including features. Figure 9: Convergence of scores over 30 iterations for 30 agents. overfitting, ensuring that additional depth or features do not inflate model complexity without providing a meaningful performance boost. Each dominant combination of max_features and max_depth offers a valid option for training the model and evaluating feature importance. For example, a particularly effective configuration is set with max_features and max_depth values of 3 and 7, respectively, which achieves an \(R^{2}\) of roughly 92%. Because our random forest ensemble consists of 100 individual trees, we obtain 100 feature-importance evaluations, some of which are highly similar. Applying Principal Component Analysis (PCA) to these importance measurements helps us understand how many distinct \"patterns\" of feature importance truly exist among the ensemble. Figure 11 illustrates the resulting PCA plot, focusing on the first two principal components to reveal key clusters or groupings of feature-importance profiles. Figure 11: PCA and clustering on feature-importance profiles. Figure 10: Heatmap model performances as \(R^{2}\) percentage for hyperparameters. (The dominant cells are marked with *). Following the PCA analysis, four natural groupings emerged among the random forest's feature-importance profiles. K-means clustering was applied with k = 4 to formalize these patterns, and the resulting clusters are shown by color in Figure 11. Each cluster represents a subset of estimators with a distinct view of which features contribute most strongly to the model's accuracy. By identifying these clusters, we gain a better understanding of how different trees within the ensemble prioritize certain features over others. Figure 12 provides a clearer picture of these variations by illustrating the average feature-importance values for each cluster. In some clusters, a few features dominate; in others, their importance is distributed more evenly among several parameters. This result demonstrates that even when overall performance metrics are similar, the paths to achieving those results can differ in how features are focused.