# Prediction of Unconfined Compressive Strength in Cement-Treated Soils: A Machine Learning Approach
## 3 Results and Discussion
### Model Performance Comparison
#### 3.2.3 Bayesian Hyperparameter Optimization Results  
During nested cross-validation, all three top-performing algorithms (Random Forest, Gradient Boosting, and XGBoost) underwent comprehensive hyperparameter optimization using Optuna's Bayesian approach. This systematic optimization across all candidates ensured that final model selection was based on each algorithm's optimal performance rather than default configurations, thereby providing a fair and rigorous comparison.  
The optimization process employed 50 trials per cross-validation fold for each algorithm, systematically exploring hyperparameter spaces tailored to each method's specific characteristics. Optuna's Tree-structured Parzen Estimator (TPE) sampler adaptively focused on promising regions of each hyperparameter space, while MedianPruner terminated unpromising trials early to enhance computational efficiency.  
The optimization results (Table 5) reveal distinct patterns in how each algorithm adapted to the cement-stabilized soil dataset. Random Forest converged to a substantial ensemble size (261 estimators) with moderate tree depth (\(max\_depth=5\)), reflecting a balanced approach between model complexity and generalization. The conservative node splitting criteria (\(min\_samples\_split=6\), \(min\_samples\_leaf=2\)) demonstrate careful adaptation to the dataset size, preventing overfitting while preserving predictive capability.  
Gradient Boosting optimization yielded the most conservative learning configuration with the largest ensemble (289 estimators), shallow trees (\(max\_depth=4\)), and an exceptionally low learning rate (0.012). This ultra-conservative learning rate indicates that Optuna identified the need for very gradual model building to achieve optimal performance, characteristic of datasets where subtle patterns require careful, incremental learning. The high subsample rate (0.963) provides minimal regularization, allowing most training data to contribute to each boosting iteration.  
XGBoost adopted an intermediate approach with a substantial ensemble (277 estimators) and deeper trees (\(max\_depth=7\)), balanced by moderate regularization through minimum child weight (3) and feature sampling (\(colsample\_bytree=0.820\)). The significantly higher learning rate (0.173) compared to Gradient Boosting reflects XGBoost's enhanced regularization capabilities that permit more aggressive learning steps while maintaining stability.  
The convergence to relatively large ensemble sizes across all algorithms (261-289 estimators) suggests that the cement-stabilized soil relationship benefits from extensive model averaging to  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Hyperparameter** & **Random Forest** & **Gradient Boosting** & **XGBoost** \\ \hline \(n\_estimators\) & 261 & 289 & 277 \\ \(max\_depth\) & 5 & 4 & 7 \\ \(min\_samples\_split\) & 6 & - & - \\ \(min\_samples\_leaf\) & 2 & - & - \\ \(learning\_rate\) & - & 0.012 & 0.173 \\ \(min\_child\_weight\) & - & - & 3 \\ \(subsample\_{}\) & - & 0.963 & 0.866 \\ \(colsample\_bytree\) & - & - & 0.820 \\ \(random\_state\) & 42 & 42 & 42 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Optimal hyperparameter configurations for machine learning models used in UCS prediction.  
capture the complex interactions between stabilization parameters. The dramatic difference in learning rates between Gradient Boosting (0.012) and XGBoost (0.173) highlights how different boosting implementations require distinct optimization strategies for the same dataset.  
The moderate tree depths across algorithms indicate that while the relationships are non-linear, they do not require extremely deep Decision Trees, suggesting that the key interactions can be captured through ensemble diversity rather than individual tree complexity. This finding supports the suitability of tree-based methods for this application while validating the experimental design's effectiveness in capturing the essential patterns in UCS development.