# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 4 Results and Evaluation
### Real-Time Implementation  
In this section, the real-time assessment is reported. After thoroughly evaluating the developed model, the model was tested on an actual real vehicle environment to determine  
Figure 8: Average confusion matrix generated to validate the proposed model for seven-fold cross validation: (**a**) RGB image and (**b**) IR image results. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline
**Image Type** & \multicolumn{2}{c}{**Fold-1**} & **Fold-2** & **Fold-3** & **Fold-4** & **Fold-5** & **Fold-6** & **Fold-7** & **Average** \\ \hline \multirow{3}{*}{RGB Images} & Driving & 98.9 & 85.8 & 88.7 & 45.0 & 99.0 & 97.9 & 92.6 & 86.8 \\ \cline{2-9}  & Distracted & 96.2 & 96.5 & 98.4 & 99.8 & 97.1 & 98.3 & 96.5 & 97.7 \\ \cline{2-9}  & Average & 97.4 & 91.1 & 95.5 & 83.3 & 97.5 & 98.1 & 95.1 & 93.9 \\ \hline \multirow{3}{*}{IR Images} & Driving & 96.1 & 98.4 & 100.0 & 98.5 & 99.2 & 100.0 & 100.0 & 98.9 \\ \cline{2-9}  & Distracted & 99.6 & 98.8 & 99.8 & 99.7 & 99.2 & 99.9 & 100.0 & 99.6 \\ \cline{2-9}  & Average & 99.1 & 98.7 & 99.9 & 99.5 & 99.3 & 99.9 & 100.0 & 99.2 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance evaluation of the proposed model for binary classification (non-distracted driving and distracted driving) based on seven-fold cross validation. the capability of prediction during the day- and nighttime conditions. At this moment, four new people (three male and one female) were recruited and verbally instructed to perform the predefined actions. Normally, the vehicle environment is subjected to various limitations such as size and power. As a result, instead of using size-constrained GPU boards, we deployed an embedded NVIDIA Jetson AGX Xavier board. This device consumes less power and is much cheaper than other GPU boards. The embedded board is equipped with a 512-core GPU with tensor cores, 8-core ARM 64-bit CPU, and 32 GB 256-bit LPDDR4x memory. Ubuntu 18.06 and Tensor flow were used for the OS and deep learning frameworks, respectively. The completed trained model weight was first stored in hd5 format in our original workstation environment. Then, the hd5 weight file was converted to Tensor RT format to deploy on the embedded board. In the conversion process, a few optimizations were made to fit the operation on the embedded board. During the real-time streaming, the RGBDT sensor module and Xavier embedded board are interconnected using transmission control protocol/internet protocol (TCP/IP) communication to exchange data. At this time, the embedded board receives the driver activity image from the RGBDT sensor and creates a single sequence based on the most recent five frames of images, and uses it as an input to the loaded model. The prediction result is not immediately returned to the RGBDT sensor because the model prediction could be temporarily wrong. Instead, it selects the most repeated action or actions that appear more than three times among the five frames (single sequence) and returns them as the final predicted driver action result. This also helps to reduce the misclassification that can occur during the transition scene from one type of driver class to another type of driver class. The computational time to execute a single sequence input (composed of five frames) is around 90 ms/sequence and the general inference speed of the system is around 12 fps on the Xavier embedded board. Meanwhile, to report the statistical results, a video was recorded during the real-time streaming in day- and nighttime conditions. Similarly to the real-time evaluation setting reported previously [17], we compared the difference between the ground truth label and the predicted driver behavior class, as shown in Figures 9 and 10. In Figure 9a,b, the multiclass result of driver 1 for daytime with RGB images and nighttime with IR images are reported, respectively.