## 4 Experiments and Results
### Results and Discussion  
Based on the evaluation results of individual models in the previous step, the combination of the base learner and meta learner were determined through repeated experiments. First of all, since most ML models do not have much of a difference between the \(R^{2}\) values for testing data, these models can be selected as the base learners. Given that the heterogeneous integration of different types of base learners can enhance the generalization ability of the model, we selected RF, SVR and CatBoost as base learners. Adaboost, XGboost, LightGBM and CatBoost all belong to the Boosting Ensemble, but since CatBoost is the most recently released as an improvement on the common boosting algorithm, it was selected as the base learner.  
In the first-level prediction model, RF, SVM and CatBoosting were generated individually using the scikit learn package in Python. In the second-level forecast, an LR algorithm was selected to combine the three basic-learning algorithms and to generate final prediction results. As we saw earlier, since there is no significant difference in the performance of individual models, we did not use the method of assigning weights to the output of the base learner.  
Table 4 shows the five-fold cross-validation results (RMSE and \(R^{2}\)) calculated for the training dataset and the testing dataset by the proposed model. The results demonstrate that the values of RMSE and \(R^{2}\) are almost constant regardless of the number of cross-validation executions. The average \(R^{2}\) of the five-fold cross-validation was calculated to be 0.91, which is an improvement of 0.02 from the previously calculated \(R^{2}\) value of 0.89 for individual models. As a result of optimizing the hyperparameter using Bayesian optimization in the process of training the base learner, the \(R^{2}\) values of RF, SVM and Catboost were calculated to be 0.900, 0.897 and 0.906, respectively. Since the result value of the meta learner was 0.91, it can be seen that the \(R^{2}\) value of each base learner is improved.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Cross-Validation**} & \multicolumn{2}{c}{**Training Dataset**} & \multicolumn{2}{c}{**Testing Dataset**} \\ \cline{2-5}  & **RMSE** & \(R^{2}\) & **RMSE** & \(R^{2}\) \\ \hline
1 & 1775.01 & 0.92 & 1644.83 & 0.91 \\
2 & 1706.94 & 0.92 & 1933.24 & 0.90 \\
3 & 1776.34 & 0.92 & 1649.95 & 0.92 \\
4 & 1729.90 & 0.92 & 1821.91 & 0.91 \\
5 & 1695.12 & 0.92 & 1951.54 & 0.91 \\ Average & 1736.66 & 0.92 & 1800.29 & 0.91 \\ Stdv 1 & 37.75 & 0.002 & 148.15 & 0.007 \\ \hline \hline \end{tabular} \({}^{1}\) Standard deviation.  
\end{table}
Table 4: Five-fold cross-validation accuracy results of the proposed model.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{2}{c}{**Training Dataset**} & \multicolumn{2}{c}{**Testing Dataset**} \\ \cline{2-5}  & **RMSE** & \(R^{2}\) & **RMSE** & \(R^{2}\) \\ \hline ANN & 1973.05 & 0.90 & 2012.30 & 0.89 \\ LR & 1991.61 & 0.89 & 2026.00 & 0.89 \\ SVR & 2006.20 & 0.89 & 2041.48 & 0.89 \\ BDT & 937.08 & 0.98 & 2115.27 & 0.88 \\ RF & 1632.91 & 0.93 & 2018.72 & 0.89 \\ Adaboost & 1700.85 & 0.92 & 2086.06 & 0.88 \\ XGboost & 1557.83 & 0.93 & 1974.73 & 0.89 \\ LightGBM & 1509.55 & 0.94 & 1977.16 & 0.89 \\ CatBoost & 1712.98 & 0.92 & 2003.19 & 0.89 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Performance comparison of the individual ML models.