# Improving Machine Learning Identification of Unsafe Driver Behavior by Means of Sensor Fusion
## 4 Experimental Results
### Classification Results  
Both SVM and neural network classifiers have been trained using the k-fold cross-validation technique with k = 5. To measure the performance of the classifier, the following four metrics have been used:  
\[Precision=\frac{TP}{TP+FP} \tag{2}\]  
\[Recall=\frac{TP}{TP+FN} \tag{3}\]  
\[F_{1}score=2\cdot\frac{Precision\cdot Recall}{Precision+Recall} \tag{4}\]  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{5}\]  
where \(TP\) are the true positives, \(TN\) the true negatives, \(FP\) the false positives, and \(FN\) the false negatives. We also conducted incremental experiments to evaluate the relative importance of the proposed features with respect to the classification performances. In particular, we added, step by step, the groups of features calculated from different sensors in order to highlight their contributions to the classes discrimination. We started training the model with only the in-vehicle sensors' features, which we believe to be the minimum set; then we added the magnetometer and gyroscope features. Both training and testing phases have been carried out using Matlab(r) on an Asus N56] laptop PC equipped with an Intel(r) CORE i7 @ 2.5 GHz and 16 GB of RAM running Windows 10 Home edition. **SVM results:** Table 1 reports the confusion matrices and the classification performances of the SVM model when varying the adopted features. The Table 1a shows the results obtained using only the in-vehicle sensors' features (i.e., speed, engine load, engine rotation speed, and throttle position). In this case, the average classification accuracy reached about 76% and the average precision and recall were about 75%. Notice that the performances of the two classes were appreciably unbalanced, as demonstrated by higher values of the recall and of the F1 score of the \"safe\" class with respect to the \"unsafe\" one. Figure 6: Scatter plots of several features calculated starting from the dataset tracks. Adding magnetometer features (see Table 1b) increased all average performances, which then became over 82%, by about 7% and also reduced the unbalance about the two classes. For instance, F1 score and recall of the \"unsafe\" class reached, respectively, 80% and 78%. Gyroscope features (Table 1c), on the other hand, increased performance metrics even more leading to values close to 87%. Table 1d reports the results obtained when training the SVM model using all available features. Interestingly, in this configuration, the average classification performances were exceeded by just 1% those obtained when using only in-vehicle sensors plus gyroscopic features as if the signals recorded from the magnetometer did not add useful information compared to those already present in the gyroscope. Moreover, another appreciable variation, obtained in this case appears to be a better balance of the classification performances between the two classes. In the case scenario that consists of using all available features, the execution time of the training phase was 14.925 s, while the prediction of a sample (i.e., of a single vector computed within a 5 s window) was accomplished in 0.357 milliseconds. **Neural network results:** The results of the same classification experiments conducted with the feed-forward neural network are reported in Table 2. With respect to the SVM results, the neural network achieved better performances (measurable in values from about 2% to 5%) in all features configurations, even if essentially it reflects the results obtained by the SVM tool. In fact, also in this case, using only in-vehicle sensors led to a classification performance that sis not exceed the 78% on average, confirming the inadequacy of the extracted features to accurately identify driver behavior. On the other hand, adding a simple gyroscope can strongly compensate for this lack of information, bringing the classification performance up to an average value of about 89%.