## 4 Development of Predictive Models
### Model Optimization by Tuning the Hyperparameters
\begin{table}
\begin{tabular}{l c c} \hline \hline
**Hyperparameters** & **Search Space** & **Model (Optimal Value)** & **Description** \\ \hline \({}^{*}\)n\_estimators\({}^{\text{a}}\) & (\(10,1000\)) & RFR (335), ESTR (334), ABR (551) & The total count of decision trees in the random forest. \\ \hline \({}^{*}\)max\_depth\({}^{\text{a}}\) & (\(1,50\)) & RFR (26), XGB (34), DTR (16), ESTR (47) & The maximum depth up to which each decision tree can grow during the training process. \\ \hline \({}^{*}\)min\_samples\_split\({}^{\text{a}}\) & (\(2,20\)) & DTR (3), RFR (2), ESTR (2) & The minimum number of samples required to split an internal node. \\ \hline \({}^{*}\)min\_samples\_leaf\({}^{\text{a}}\) & (\(1,20\)) & DTR (2), RFR (1), ESTR (1) & The minimum number of samples required to be at a leaf node. \\ \hline \({}^{*}\)learning\_rate\({}^{\text{a}}\) & (\(0.01,0.3\)) & XGBR (0.3), ABR (0.4292) & Parameter which controls the step size at each iteration when updating the weights of the model. \\ \hline \({}^{*}\)min\_child\_weight\({}^{\text{a}}\) & (\(1,20\)) & XGBR (7) & The minimum sum of weight of instances (hessian) required in a child node. \\ \hline \({}^{*}\)subsample\({}^{\text{a}}\) & (\(0.1,1\)) & XGBR (1.0) & The part of samples used for training each tree. \\ \hline \({}^{*}\)colsample\_bytree\({}^{\text{a}}\) & (\(0.1,1\)) & XGBR (1.0) & The part of features (columns) used for training each tree. \\ \hline \({}^{*}\)gamma\({}^{\text{a}}\) & (\(0,10\)) & XGBR (0.0) & Parameter which regulates the minimum reduction in the loss function essential to split a node during the construction of a tree. \\ \hline \({}^{*}\)reg\_alpha\({}^{\text{a}}\) & (\(0,10\)) & XGBR (0.95065) & L1 regularization term on the weights. \\ \hline \({}^{*}\)reg\_lambda\({}^{\text{a}}\) & (\(0,10\)) & XGBR (10.0) & L2 regularization term on the weights. \\ \hline \({}^{*}\)C\({}^{\text{a}}\) & (\(0.1,100\)) & SVR (100) & Parameter defining the penalty for misclassified data points. \\ \hline gamma & (\(0.0001,10\)) & SVR (0.0271) & Parameter that characterizes the reciprocal of the range within which the model selects samples as support vectors. \\ \hline epsilon & (\(0.01,1\)) & SVR (0.01) & The epsilon-insensitive loss function of the Support Vector Regression (SVR) algorithm. \\ \hline \({}^{*}\)L1 ratio\({}^{\text{a}}\) & (\(0,1\)) & ENR (1) & L1 ratio (mixing parameter). \\ \hline \({}^{*}\)max\_iter\({}^{\text{a}}\) & \([100,200,500]\) & ENR (500) & Maximum number of iterations. \\ \hline \({}^{*}\)n\_neighbors\({}^{\text{a}}\) & (\(1,20\)) & KNN (3) & Number of neighbors. \\ \hline \({}^{*}\)weights\({}^{\text{a}}\) & [’uniform’, ‘distance’] & KNN (’distance’) & Weight function used in prediction. \\ \hline \({}^{*}\)p\({}^{\text{a}}\) & [\(1,2\)] & KNN (1) & Power parameter for the Minkowski metric. \\ \hline \({}^{*}\)fit\_intercept\({}^{\text{a}}\) & [’True’, ‘False’] & RR (’True’), LR (’True’), ENR (’True’) & Controls whether to calculate intercept for the model or not. \\ \hline \({}^{*}\)alpha\({}^{\text{a}}\) & (\(1\times 10^{-4}\), \(1\times 10^{6}\), \({}^{*}\)log-uniform’) & RR (0.79193), LR (0.03341), ENR (0.03352) & Regularization strength. \\ \hline \({}^{*}\)solver\({}^{\text{a}}\) & [’auto’, ‘svd’, ‘choeks’, ‘lsq’] & RR (’auto’) & Solver for optimization. \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance metrics for regression models trained with default hyperparameters. During the optimization process, different combinations of hyperparameters were used for 300 iterations to ensure the proper convergence of the optimization process. The optimized models with the best hyperparameters were then trained and fit using the training data and evaluated with five-fold cross validation. Subsequently, the optimized models were then used to make predictions on the testing dataset and the respective metrics along with Cl\({}_{0.95}\) were computed and recorded. Out of the ten regression models, only six black box models demonstrated high accuracy levels.