# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 4 Results and Evaluation
### Failure Analysis of the Proposed System  
This section investigates the failure analysis and limitation of the proposed LRCN model. Normally, for driver activity recognition, RGB images collected from the RGB sensor and IR images collected from the IR sensors are used for real-time applications during the day- and nighttime, respectively. However, in this work, we further studied the use of RGB images gathered at nighttime and IR images collected during the daytime. For this purpose, we collected RGB images at nighttime and IR images during the daytime for two drivers in a real-time scenario. However, the model has shown poor performance or failure for both drivers, with an accuracy of 63.69% and 30.87% for the RGB images collected at nighttime. This is because the RGB sensor is sensitive to sunlight and illumination, and therefore cannot completely capture the full posture of the driver during the nighttime. A more statistical result is presented in Appendix A, Figure A3. On the contrary, the IR images result shows a satisfactory performance, with an accuracy of 89.6% and 86.4% in the daytime. Based on the result obtained, it can be concluded that the usage of RGB sensors (RGB color images) for driver behavior recognition overnight are impractical. However, IR sensors (IR images) can be applied for both day- and nighttime driver activity recognition, as acceptable results were obtained in both scenarios.  
On the other hand, despite the higher performance of our model, the system still has a few limitations in identifying the driver activities that have common body posture and orientation, such as talking and smoking, and head nodding and fainting. Specifically,  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Approaches** & **Accuracy** & **Participants** & **Platform** & **Operation** \\ \hline GMM segmentation and transfer learning [17] & Recognition: 81.6\% & & Daytime & Nighttime \\  & Binary classification: 91.4\% & 10 drivers & Real Vehicle & ✓ & \(\times\) \\ \hline InterCNN with MobileNet [30] & Recognition: 73.9\%—9 class, Recognition: 81.66\%—5 class & 50 drivers & \
\begin{tabular}{c} Mockup car \\ (Simulator) \\ \end{tabular} & ✓ & \(\times\) \\ \hline VGG16 with regularization [31] & Recognition: 96.3\% (Train–Test Split) & 31 drivers & Real Vehicle & ✓ & \(\times\) \\ \hline AlexNet,VGG16, GoogleNet, and ResNet [33] & Recognition: 86-92\% & 10 drivers & Simulator & ✓ & \(\times\) \\ \hline AdaBoost classifier and Hidden Markov Model [45] & Recognition: 85.0\%, Binary classification: 89.8\% & 8 drivers & Simulator & ✓ & \(\times\) \\ \hline Modified SVM, KNN, and RUSBoost [46] & Recognition: 66.7–87.7\% & 20 drivers & Real Vehicle & ✓ & \(\times\) \\ \hline Our proposed model &
\begin{tabular}{c} Recognition: 88.7\% for RGB images \\ and 92.4\% for night-time datasets. \\ \end{tabular} & 35 drivers & Real Vehicle & ✓ & ✓ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of the proposed model with other existing state of the art approaches.  
talking is recognized when the driver raises their right arm to engage with a passenger, while in real-life scenarios, the driver may also engage in conversation with both hands on or off the wheel. Also, looking outside is recognized when the driver is only looking to the left outside direction; however, in real-life practice, drivers might engage in various non-driving related activities and look in various directions. Therefore, we defer this study to future work, to enrich our dataset with several driver activities and more driver samples in a complex environment and, moreover, to comprehensively confirm the robustness of IR data under various temperature conditions and with different drivers, thus maximizing the recognition and generalization ability of the proposed system to be fully deployed on a real-time basis. We also plan to incorporate the depth sensor information in the future work to evaluate its impact on the model performance.