# Investigating the Applications of Machine Learning Techniques to Predict the Rock Brittleness Index
## 3 Modelling Process and Results
### Evaluation of the Developed Models  
Once the models have been developed, the accuracy performance of each model was evaluated using five well-known indices i.e., coefficient of determination (\(R^{2}\)), root mean square error (\(RMSE\)), mean absolute error (\(MAE\)), variance account for (\(VAF\)), and _a20-index_. The formula that was used for calculating the mentioned performance indices are presented in Equations (2)-(6). This study also employed an easy to understand ranking system which ranked each model developed using the above-mentioned performance criteria for both training and testing stages. For each criteria, the ranking system first sorted the models based on their obtained values, then assigned the highest rank (5) to the best value and the lowest rank (1) to the worst value. Final rank of each model was calculated through summing the ranking values for both training and testing stages (Equation (6)):  
\[R^{2}=1-\frac{\sum_{i}(y_{i}-f_{i})^{2}}{\sum_{i}(y_{i}-\overline{y})^{2}} \tag{2}\]  
\[VAF=\left[1-\frac{\mathit{var}\left(y-y\
u\right)}{\mathit{var}\left(y\right) }\right]\times 100 \tag{3}\]  
\[MAE=\frac{1}{N}\sum_{j=1}^{N}\left|y_{i}-y_{j}\right| \tag{4}\]  
\[RMSE=\sqrt{\frac{1}{N}\sum_{i=1}^{N}(y-y^{\prime})^{2}} \tag{5}\]  
\[a20-index=\frac{m20}{N} \tag{6}\]  
where \(y\) denotes the measured values, \(\overline{y}\) and \(y^{\prime}\) indicate mean and predicted of the \(y\), respectively, \(N\) denotes the total number of data, \(m20\) shows number of samples with value of experimental value/predicted value between 0.8 and 1.20:  
\[\mathit{Final\ rank\ of\ model}=\sum_{1\leq i\leq 5}^{1\leq j\leq 2}R_{ij} \tag{7}\]  
where \(i\) denotes the indices, \(j\) shows the dataset, the \(R\) shows model's ranking. The values and ranks of the performance indices and models are presented in Table 2. The results of this evaluation showed that the KNN model achieved the highest final rank (37). This model was followed by RF (34) and ANN (33), respectively. For the training dataset, the RF obtained the highest rank (25) while the SVM obtained the lowest rank (5). For the testing dataset, the ANN achieved the highest rank (22) while the CHAID achieved the lowest rank (6). Turning to the performance indices, the RF outperformed other models developed for the training dataset. However, for the testing dataset, the ANN achieved the best ranks for three indices including \(R^{2}\), \(RMSE\), and _a20-index_. The ANN also achieved the second-best rank for \(VAF\). Based on these discussions and rank values, three models of RF, KNN, and ANN were selected to be discussed in more details in the following sections. The authors also used a gain chart (Figure 5) to compare the performance of the models proposed for both training and testing datasets. Gains are estimated as (number of hits in quantile/total number of hits) \(\times\) 100%. Here, it is necessary to mention that \"hit\" refers to the success of a model to predict the values greater than the midpoint of the fields range (\(BI>\) 16.458). In this chart, the blue line denotes the perfect model which has perfect confidence (where hits = 100% of cases), the diagonal red line denotes the at-chance model, and the other five lines in the middle represent the models developed in this study.