# Accident Probability Prediction and Analysis of Bus Drivers Based on Occupational Characteristics
## 5 Methods
### Boosted Trees
#### 5.1.5 Evaluation  
The problem of predicting the risk of accidents for bus drivers with the target variable of whether or not an accident occurs in the current month is a typical binary classification problem. A confusion matrix is one of the most basic tools for assessing the effectiveness of classification in binary classification problems. It classifies the actual observations according to the predicted categories into the following: true-positive (TP), the number of samples predicted to be positive and actually positive; false-positive (FP), the number of samples predicted to be positive but actually negative; true-negative (TN), the number of samples predicted to be negative and actually negative; and false-negative (FN), the number of samples predicted to be in the minus category but actually in the plus category. This is shown in Table 2.  
The confusion matrix provides us with a comprehensive view of model performance. Based on the four metrics in the confusion matrix, multiple evaluation metrics can be computed to understand how the model performs in different aspects.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Prediction**} & \multicolumn{2}{c}{**Actual Results**} \\  & **Positive Examples** & **Negative Examples** \\ \hline Positive examples & TP & FP \\ Negative examples & FN & TN \\ \hline \hline \end{tabular}
\end{table}
Table 2: Confusion matrix.  
Precision--This is concerned with the proportion of positive cases that the model predicts will actually be positive. It provides a measure of how accurately the model predicts \"positive cases\" (in this case, accurately predicting accidents).  
\[Precision=\frac{TP}{TP+FP} \tag{6}\]  
Recall--The main focus is on the proportion of positive examples that the model is able to identify out of the entire set of positive examples. It measures the ability of the model to identify all actual positive example samples (to predict all transit crashes).  
\[Recall=\frac{TP}{TP+FN} \tag{7}\]  
F1-score--This is a metric that combines precision and recall and provides an assessment of the balance of the model.  
\[F1=\frac{2\times Precision\times Recall}{Precision+Recall} \tag{8}\]  
ROC curve--This focuses on the ability of the model to distinguish between positive and negative cases. It is a performance curve plotted at different thresholds with a true-positive rate (TPR) on the vertical axis and a false-positive rate (FPR) on the horizontal axis.  
\[TPR=Recall=\frac{TP}{TP+FN} \tag{9}\]  
\[FPR=\frac{FP}{FP+TN} \tag{10}\]  
AUC value--This is the area under the ROC curve; the closer the AUC value is to 1, the better the model performance.  
PR curve--The horizontal axis indicates recall and the vertical axis indicates precision; by plotting the performance curve of the model under different thresholds, it brings more attention to the balance between the precision and recall of the model under different thresholds.  
AUC-PR--This is the area under the PR curve; the closer the AUC-PR is to 1, the better the model can balance precision and recall under different thresholds.  
[PERSON]'s kappa (k)--This is a metric that provides an assessment of the degree of agreement between the classifier's predictions and actual observations. A [PERSON]'s kappa (k) < 0.00 indicates that the model classifies poorly, 0.00-0.20, slightly, 0.21-0.40, fairly, 0.41-0.60, moderately, 0.61-0.80, substantially, and 0.81-1.00, almost perfectly.  
\[k=\frac{p_{0}-p_{c}}{1-p_{c}} \tag{11}\]  
\[p_{0}=\frac{TP+TN}{TP+FN+FP+TN} \tag{12}\]  
\[p_{C}=\frac{(TP+FP)(TP+FN)+(FN+TN)(FP+TN)}{\left(TP+FN+FP+TN\right)^{2}} \tag{13}\]