# BoxRF: A New Machine Learning Algorithm for Grade Estimation
## 3 Results

Table 6: Top 5 performances for NR = 2, FH = 1. \begin{table}
\begin{tabular}{c c c c c c c c} \hline
**R\({}^{2}\)** & **R\({}^{2}\_\)std** & **RMSE** & **RMSE\_std** & **BL** & **BW** & **BH** & **MS** & **T** \\ \hline
0.749 & 0.001 & 0.170 & 0.000 & [50, 100] & [50, 100] & [50, 100] & 3 & 11 \\
0.744 & 0.002 & 0.172 & 0.001 & [100, 300] & [50, 100] & [25, 50] & 5 & 9 \\
0.744 & 0.003 & 0.172 & 0.001 & [100, 300] & [50, 100] & [25, 50] & 5 & 11 \\
0.744 & 0.002 & 0.172 & 0.001 & [100, 300] & [50, 100] & [25, 50] & 5 & 9 \\
0.743 & 0.003 & 0.173 & 0.001 & [100, 300] & [50, 100] & [50, 100] & 5 & 11* Out of the 20 results shown, 12 of them contain 11 trees, with a median R\({}^{2}\) of 0.75. Four rows contained 9 trees, with a median R\({}^{2}\) of 0.747, while three rows contained seven trees, with a median R\({}^{2}\) of 0.735. Only two rows contained three trees; the median R\({}^{2}\) was 0.735. * When sorted by box length, 11 rows contained box length [100, 300] with a median r\({}^{2}\) of 0.75, while 8 rows contained box length [50, 100] with a median r\({}^{2}\) of 0.737. Only one row contained the box length of [300, 500], though it had an R\({}^{2}\) of 0.75. Given the variogram ranges, it is not surprising that the longer boxes [300, 500] did not perform well. * When sorted by box width, four rows contained a box width [100, 200] with a median R\({}^{2}\) of 0.75. Moreover, the performance was consistent at an R\({}^{2}\) = 0.75. The rest contained box width [50, 100] with a median R\({}^{2}\) of 0.744. * Looking at box heights, the top three performances (all tied with the best performance of R\({}^{2}\) = 0.75) came from a box height of [100, 300] and a box width of [50, 100]. _Comparison with Machine Learning Methods: RF, Neural Nets, K-Nearest Neighbors, XGBoost_  
BoxRF was compared to several machine learning methods available from scikit-learn (version 1.5.2), a popular public domain toolkit [27]. These included RF, neural nets (NN), and k-nearest neighbors (kNN). An additional comparison was also made with XGBoost [28]. The training and testing subsets were identical for all the methods (the same as shown in Table 3). GridSearchCV, a cross-validation tool also from scikit-learn, was used to select the hyperparameters for the four traditional machine learning methods based on 5-fold cross-validation. Prior to setting the hyperparameter combinations for GridSearchCV, the authors had modeled the methods individually to determine parameter bounds. The details are as follows:  
* The NN function used was MLPRegressor. The hyperparameters tested were the following:
* Activation functions: tanh, logsig, and relu;
* Neurons and layers: Up to 2000 neurons, in a combination of single, double, and triple hidden layers;
* Solvers: sgd, adam, lbfgs. Early stopping was practiced with a validation fraction of 0.25. The data were normalized prior to NN modeling. GridSearchCV indicated that the best hyperparameters were relu as the activation function, two hidden layers of 50 neurons each, and adam as the solver. * KNN was applied using KNeighborsRegressor. A variety of neighborhood sizes were tested, from 3 to 500. The best neighborhood size was 55. * The function RandomForestRegressor was used for scikit-learn random forest (SRF). Two key parameters for the SRF models, depth and minimum samples in a leaf, were tested. Depth options explored were (8, 12, 16, 20, 24, 30, 32), while the _minimum samples in a leaf_ option explored were (1, 2, 3, 4). GridSearchCV indicated that the best options for depth and minimum samples in a leaf were 30 and 4, respectively. * The function XGBRegressor was used for XGBoost. The number of estimators varied in the range of 5-150, and the learning rate varied between 0.05 and 1.2. The best hyperparameters were determined to be 80 for the number of estimators, with a learning rate of 0.05. The results of applying the four machine learning methods were as follows:
* NN performance was r\({}^{2}\) of 0.392 and RMSE of 0.0265. * KNN performance was r\({}^{2}\) of 0.476 and RMSE of 0246.