# Application of Machine Learning Ensemble Methods to ASTRI Mini-Array Cherenkov Event Reconstruction
## 6 Tests on New Methods for Energy Reconstruction
### Ensemble Learning Methods  
As already completed for gamma/hadron separation, we applied Stacking Ensemble Learning on different combinations of models. Stacking Ensemble Learning is a technique in which multiple models are trained on the same dataset and their predictions are combined using a meta-model. The objective of stacking is to leverage the individual strengths of different models to produce more accurate predictions. As such, stacking is an effective way to improve the performance of Machine Learning models.  
We tested Stacking Ensemble models composed as follows (see Figure 11 and Table 5):  
* EN ENSEMBLE 0: 0.65 XGB, 0.21 Extra Trees, 0.14 Random Forest.
* EN ENSEMBLE 1: 0.80 XGB, 0.16 HGB, 0.04 Extra Trees.
* EN ENSEMBLE 2: 0.34 XGB, 0.33 Extra Trees, 0.33 HGB.
* EN ENSEMBLE 3: 0.55 XGB, 0.45 Extra Trees.
* EN ENSEMBLE 4: 0.45 XGB, 0.275 Random Forest, 0.275 Extra Trees.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Adj R-Squared** & **R-Squared** & **RMSE** & **Training Time** \\ \hline XGB & 0.33 & 0.33 & 0.02 & 0.83 \\ Extra Trees & 0.30 & 0.31 & 0.02 & 0.74 \\ Random Forest & 0.30 & 0.30 & 0.02 & 7.07 \\ NuSVR & 0.29 & 0.30 & 0.02 & 21.90 \\ Grad Boost & 0.28 & 0.28 & 0.02 & 1.67 \\ HGB & 0.27 & 0.28 & 0.02 & 1.76 \\ Bagging & 0.24 & 0.24 & 0.02 & 0.46 \\ KNeigh & 0.15 & 0.15 & 0.02 & 0.05 \\ Tran Target & 0.11 & 0.12 & 0.02 & 0.03 \\ Linear & 0.10 & 0.12 & 0.02 & 0.01 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Evaluation metrics for different Machine Learning models for energy estimation. Only the first 10 models out of 42 are shown. It is worth noticing that this is a fast test run on 10,000 samples on the benchmark set of parameters (“EN SET 0”) and the results are only approximate. R-squared measures the proportion of variance in the dependent variable explained by the independent variables, while adjusted R-squared takes into account the number of independent variables in the model. Root Mean Squared Error (RMSE) measures the average deviation of predicted values from actual values.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**EN Ensemble** & **XGB** & **ET** & **RF** & **HGB** \\ \hline
0 & 0.65 & 0.21 & 0.14 & 0.00 \\
1 & 0.80 & 0.04 & 0.00 & 0.16 \\
2 & 0.34 & 0.33 & 0.00 & 0.33 \\
3 & 0.55 & 0.45 & 0.00 & 0.00 \\
4 & 0.45 & 0.275 & 0.275 & 0.00 \\ \hline \hline \end{tabular}
\end{table}
Table 5: The table displays the percentage of each Machine Learning model in five different Stacking Ensembles consisting of XGB, Extra Trees (ET), Random Forest (RF), and HGB models.  
These combinations were derived using a general method for creating Stacking Ensemble models, which involves training multiple base models on the same dataset and then combining their predictions using a meta-model. The specific weights assigned to each base model in the ensembles were determined through a process of hyperparameter tuning and cross-validation, which seeks to identify the optimal combination of models and weights for the specific task at hand.  
Our primary objective was to improve the bias of the ensembles, as there was a substantial scope for enhancement in this area. Therefore, we prioritized the optimization of the ensembles based on bias rather than energy resolution. Consequently, the ensembles and their weights were selected to optimize the bias and not energy resolution.  
In our tests, all these methods computed on \"EN SET 1\" show better performance than any method made using a single bagging or boosting ensemble.