# Application of Machine Learning Ensemble Methods to ASTRI Mini-Array Cherenkov Event Reconstruction
## 5 Tests on New Methods for Gamma/Hadron Separation
### Ensemble Learning Methods  
Ensemble Learning is a general meta approach to Machine Learning that involves training multiple models and combining their predictions to make a final prediction. The goal of Ensemble Learning is to improve the accuracy, stability, and robustness of the final prediction by using the strengths of multiple models.  
There are several types of Ensemble Learning methods, including Stacking. Stacking is a type of Ensemble Learning in which multiple models are trained on the same data  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Adj R-Squared** & **R-Squared** & **RMSE** & **Training Time** \\ \hline Grad Boosting & 0.30 & 0.31 & 0.41 & 2.09 \\ LGBM & 0.29 & 0.30 & 0.41 & 0.68 \\ HGB & 0.29 & 0.29 & 0.42 & 0.97 \\ Random Forest & 0.28 & 0.28 & 0.42 & 7.16 \\ Extra Trees & 0.27 & 0.28 & 0.42 & 0.84 \\ MLP & 0.25 & 0.26 & 0.43 & 2.39 \\ XGB & 0.23 & 0.23 & 0.43 & 0.38 \\ SVR & 0.22 & 0.22 & 0.44 & 3.55 \\ Ada Boost & 0.21 & 0.22 & 0.44 & 0.14 \\ Bagging & 0.20 & 0.20 & 0.44 & 0.47 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Evaluation metrics for different Machine Learning models for gamma/hadron separation. Only the first 10 best-performing models out of 42 are shown. It is worth noting that this is a fast test run on 10,000 samples on the benchmark set of parameters (“GH SET 0”) and the results are only approximate. R-squared measures the proportion of variance in the dependent variable explained by the independent variables, while adjusted R-squared takes into account the number of independent variables in the model. Root Mean Squared Error (RMSE) measures the average deviation of predicted values from actual values.  
and their predictions are combined using a meta-model. The goal of stacking is to use the strengths of different models to make more accurate predictions.  
We tested Stacking Ensemble models composed as follows (see Figure 4 and Table 2):  
* GH ENSEMBLE 0: 1.00 Random Forest (same model as the benchmark).
* GH ENSEMBLE 1: 0.65 XGB, 0.21 Extra Trees, 0.14 Random Forest.
* GH ENSEMBLE 2: 0.60 Extra Trees, 0.40 Random Forest.
* GH ENSEMBLE 3: 0.42 Extra Trees, 0.30 XGB, 0.28 Random Forest.  
The ensembles mentioned above were chosen through a series of tests aimed at optimizing the quality factor of the models (see next paragraph). In order to identify the best combination of models, we conducted several experiments using different ensembles and evaluated their performance using the quality factor metric. Based on the results of these experiments, we selected the four ensembles mentioned above. ENSEMBLE 0 comprises a single model, while the other ensembles are composed of a combination of XGB, Extra Trees, and Random Forest models, with varying weights assigned to each model. These ensembles were found to produce the best results in terms of the quality factor and were thus chosen for further analysis.  
\begin{table}
\begin{tabular}{c c c c} \hline
**GH Ensemble** & **RF** & **XGB** & **ET** \\ \hline
0 & 1.00 & 0.00 & 0.00 \\
1 & 0.14 & 0.65 & 0.21 \\
2 & 0.40 & 0.00 & 0.60 \\
3 & 0.28 & 0.30 & 0.42 \\ \hline \end{tabular}
\end{table}
Table 2: The table displays the percentage of each Machine Learning model in four different Stacking Ensembles consisting of Random Forest (RF), XGB, and Extra Trees (ET) models.  
Figure 4: Heatmap of gamma/hadron separation Stacking Ensembles. The figure displays the percentage of each Machine Learning model in four different Stacking Ensembles consisting of Random Forest (RF), XGB, and Extra Trees (ET) models. The ensembles were chosen through a series of tests aimed at optimizing the Quality factor of the models. Our best model is Ensemble 3. The heatmap visualization provides an easy-to-read overview of the model composition of each ensemble, with darker shades of blue indicating a higher percentage of the model.