# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 4 Results and Evaluation  
### Evaluation of the LRCN Model Based on Train-Test Split  
For fair performance evaluation of the unified LRCN and single CNN architecture, we performed a comprehensive experiment based on the data collected in the second phase that contains 17 participants for RGB and IR images. A train-test split method was implemented in which fifteen drivers were set as training and the remaining two as testing. At this time, single image input was considered for all VGG16, ResNet50, MobileNet, Xception, DenseNet169, NASNet-Mobile, and InceptionResNetV2 pre-trained CNN architectures. In contrast, a stack of five sequence inputs was used for the unified LRCN model including VGG16+LSTM, ResNet50+LSTM, MobileNet+LSTM and Xception+LSTM. To demonstrate the superiority of the models, the experiment was conducted under the same hyperparameters for both architectures.  
As such, a stochastic gradient was applied to update the parameters with a learning rate of 0.001. The batch size and the number of epochs were set to 24 and 60, respectively. The dense layer has two 1024 nodes, each followed by a dropout probability of 0.5, and one final 512 node dense layer was added to interpret the summary of the input sequence. Note that this section of the experiment was carried out for two main reasons. First, the effect of the LSTM model was evaluated based on the sequence inputs. Particularly, the superiority of the models between a single input image of the CNN architecture and a sequential input of the CNN-LSTM model was evaluated.  
Second, an effective pre-trained CNN feature extractor that generalizes well for our datasets among the experimented pre-trained models was selected. For this reason, the same hyperparameters were set for all architectures, as described above. As Table 1 reveals, the overall performance of the pre-trained CNN with LSTM is superior to all other single pre-trained CNNs. Specifically, the accuracy comparison depicted VGG 16 combined with LSTM as the most accurate. Although ResNet showed higher classification accuracy in the 2015 ILSVRC (ImageNet Large Scale Visual Recognition Challenge) competition among pre-trained models, it is a very deep architecture compared to VGG16 and the other most pretrained models. As a result, it requires relatively large-scale datasets to perform well. Based on this experiment, the VGG16 combined with LSTM was chosen as the proposed model and further modified, including the hyperparameters, to improve its classification accuracy for all driver participants.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Model** & **Test Accuracy (\%)** \\ \hline TimeDis VGG16+LSTM & 86.53 \\ VGG16 & 77.08 \\ TimeDis ResNet50+LSTM & 77.46 \\ ResNet50 & 65.83 \\ TimeDis MobileNet+LSTM & 78.84 \\ MobileNet & 68.57 \\ TimeDis Xception+LSTM & 73.29 \\ Xception & 70.12 \\ DenseNet169 & 74.65 \\ NASNet-Mobile & 66.92 \\ InceptionResNetV2 & 66.68 \\ \hline \hline \end{tabular}  
* The CNN+LSTM corresponds to the CNN concatenated to the LSTM unit through the TimeDistributed (TimeDis) layer. For all the CNN architecture and CNN+LSTM models, the same hyperparameters were set for fair comparison purposes.  
\end{table}
Table 1: This performance evaluation between CNN and TimeDis CNN+LSTM based on the train–test split.