# Integrating Machine Learning Techniques for Enhanced Safety and Crime Analysis in Maryland
## 3 Methodology
### Model Comparisons and Performances
For instance, the random forest model reached an average R\({}^{2}\) of about 0.85, with an RMSE near 8.4, a substantial improvement over any linear model. The gradient boosting and XGBoost models likewise showed strong performance (R\({}^{2}\approx\) 0.80-0.83, RMSE 9-10), although the CatBoost algorithm slightly edged them out (R\({}^{2}\approx\) 0.84). Extra trees (an ensemble of extremely randomized trees) performed comparably to the random forest model (R\({}^{2}\approx\) 0.82). Notably, these ensemble approaches not only achieved higher predictive accuracy but also exhibited lower variance in R\({}^{2}\) and RMSE across the different data splits. For example, the random forest model's R\({}^{2}\) variance across the four splits was only about 0.003, compared to 0.012 for the lasso model. This suggests that the random forest model's performance was consistently high and less sensitive as to which subset of data was used for training, whereas the lasso model's results fluctuated more--an indication that the linear model was less robust. Among non-tree models, the support vector regression (SVR with an RBF kernel) also yielded better results than linear regression (R\({}^{2}\approx\) 0.75; RMSE \(\approx\) 12.3), although it did not reach the accuracy levels of the ensemble tree models. The SVR's performance variance was moderate, reflecting some sensitivity to data splits (likely due to the need to tune the kernel parameters for different data subsets). Finally, the stacking ensemble proved to be the top performer: by combining multiple algorithms (in our case, \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Best Hyperparameters** & **Test R\({}^{2}\)** & **R\({}^{2}\) Variance** & **Test RMSE** & **RMSE Variance** \\ \hline Gradient Boosting & learning\_rate = 0.1, \(n\_\)estimators = 100, & 0.80 & 0.004 & 9.7 & 0.81 \\  & max\_depth = 3 & 0.83 & 0.002 & 9.1 & 0.50 \\ XGBoost & learning\_rate = 0.1, max\_depth = 4, & 0.83 & 0.002 & 9.1 & 0.50 \\  & subsample = 0.8 & & & & \\ CatBoost & iterations = 200, depth = 6, & 0.84 & 0.003 & 8.8 & 0.55 \\  & learning\_rate = 0.05 & & & & \\ SVR (RBF kernel) & kernel = RBF, C = 10, \(\gamma\) = 0.1 & 0.75 & 0.009 & 12.3 & 1.60 \\ Stacking Regressor & base models = [RF, XGB, SVR], & 0.88 & 0.002 & 7.0 & 0.36 \\  & meta-model = Ridge & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 6: The best hyperparameters and test performance for advanced models (boosting, SVR, and stacking), along with cross-validation variance. Figure 8: Variations in R-squared values across different train–test split ratios. random forest, XGBoost, and SVR as the base learners, with a ridge regression meta-learner), the stacking regressor achieved the highest overall R\({}^{2}\) (0.88) and the lowest RMSE (~7.0) among all the tested models. This stacked model effectively leveraged the complementary strengths of its constituents and it maintained a very low variance in performance (R\({}^{2}\) variance-0.002), indicating excellent stability across the various train-test splits. To ensure that these complex models did not overfit, a rigorous cross-validation and hyperparameter tuning procedure was employed. For each model, we performed an extensive grid search over key hyperparameters, using k-fold cross-validation (typically k = 10) on the training data of each split. This means that model configurations were chosen based on their average validation performance on k-folds, rather than just on training performance, which guards against selecting an overly complex model that performs well on training data but poorly on unseen data. For example, the best random forest model was found to have a maximum tree depth of 10 and around 100 trees (estimators)--a configuration that balances model complexity and generalization. Deeper or unbounded trees could memorize the training data, but cross-validation revealed that a depth \(\approx\) 10 was optimal, likely because this prevents overfitting the smaller training sets. Similarly, the gradient boosting models (XGBoost, CatBoost, etc.)