# Prediction of Unconfined Compressive Strength in Cement-Treated Soils: A Machine Learning Approach
## 2 Materials and Methods
### Machine Learning Methodology
#### 2.3.2 Algorithm Selection and Initial Screening  
A diverse range of regression algorithms was initially evaluated to identify approaches best suited for UCS prediction. This comprehensive selection included traditional linear models, Non-Linear Regression techniques, and ensemble methods:  
_Linear Methods_: Linear Regression, Ridge Regression, Lasso Regression, and Elastic Net were included to establish baseline performance and evaluate whether linear relationships could adequately capture UCS behavior. Linear Regression models capture relationships between a dependent variable \(y\) and one or more independent variables \(X\). A Multiple Linear Regression model takes the form \(y_{i}=\beta_{0}+\beta_{1}X_{i1}+\beta_{2}X_{i2}+...+\beta_{p}X_{ip}+e_{i},i=1,...,n\), where \(n\) is the number of instances, \(\beta_{k}\) are coefficients, \(X_{ij}\) represents the \(i\)th instance of the \(j\)th predictor, and \(e_{i}\) is the error term [60]. For simple Linear Regression (\(p=1\)), the least squares method typically determines optimal coefficients. Ridge Regression extends Linear Regression by adding an L2 regularization term to the cost function, penalizing large coefficients to prevent overfitting. This approach is particularly effective with highly correlated independent variables [61]. In contrast, Lasso Regression employs L1 regularization, adding a penalty proportional to the absolute value of coefficients, which can reduce some coefficients to zero, effectively performing feature selection [62]. Elastic Net combines both Ridge and Lasso approaches, balancing feature selection and coefficient shrinkage by incorporating both penalty terms [63]. _Tree-Based Methods_: Decision Tree, Random Forest, Gradient Boosting, XGBoost, LightGBM, and CatBoost were selected for their ability to model complex non-linear relationships and feature interactions without requiring explicit feature engineering. Regression Decision Trees create a hierarchical structure by recursively partitioning data based on feature values [64]. Each internal node represents a feature-based decision, each branch an outcome, and each leaf node a predicted value. Training involves (i) selecting features and thresholds that minimize target variable variance, (ii) splitting datasets accordingly, (iii) continuing until stopping criteria are reached such as minimum instances per leaf, and (iv) optionally pruning branches that contribute minimally to predictive power. Random Forest builds upon Decision Trees by constructing multiple trees through bootstrap aggregation (bagging) and training each on a random subset of data [65]. To enhance diversity, each tree considers only a random subset of features at each split. Predictions from all trees are averaged, reducing overfitting while improving accuracy and resilience compared to a single Decision Tree. Gradient Boosting creates sequential trees where each tree corrects errors made by previous trees [66]. By optimizing along the gradient descent, it progressively reduces overall error. XGBoost enhances the Gradient Boosting framework with regularization strategies and parallel tree construction for improved speed and performance [67]. LightGBM improves efficiency through leaf-wise tree growth rather than level-wise growth, employing techniques like Exclusive Feature Bundling and Gradient-Based One-Side Sampling for faster training with large datasets [68]. CatBoost optimizes categorical data handling through permutation-driven approaches, reducing extensive preprocessing requirements while supporting GPU training and providing model analysis tools [69]. _Other Non-Linear Methods_: Support Vector Regression (SVR) with radial basis function (RBF) kernel, k-nearest neighbors (KNN), Gaussian Process Regression, and AdaBoost were included to explore alternative non-linear modeling approaches. SVR extends Support Vector Machine concepts to predict continuous target values [70]. It aims to find a function that approximates target values within a specified tolerance margin while minimizing model complexity.