# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 3 Methodology
### Model Training  
The experiments for RGB and IR images were first performed separately with the same networks. This gave us a baseline to compare the results of the two different image sources. The classifier network inputs are set to five time steps and fed to the unified LRCN architecture. To perform end-to-end fine tuning, the pre-trained CNN architecture is kept trainable throughout the whole experiment. When training a deep learning model, the network hyperparameters such as the optimizer, learning rate, the number of hidden units in LSTM, and the dense layer can greatly affect the tradeoff accuracy. Particularly, the number of dense layers and learning rate exert large effects on the complexity of the designed model, as well as the accuracy. As such, the hyperparameters are carefully analyzed and selected in this work. Regarding the LSTM hidden units, we compared 256 units and 512 hidden units and found that the model with 256 hidden units yielded superior results. Therefore, the number of hidden units for the LSTM layer in this proposed work was set to 256. The dense layer was set to single 1,024 nodes and 512 nodes, each followed by batch normalization and dropout of 0.5 and 0.9, respectively. An aggressive dropout (0.9) was applied to combat the overfitting problem. The batch size and number of epochs were selected to be 24 and 300, respectively. To optimize the weights of the proposed network, the Adam optimizer with a learning rate of \(1\times 10^{-6}\) was set as an objective function. In addition, the model recognition performance was measured using categorical cross-entropy loss, whose result output was a probability value between 0 and 8. The standards loss function \(\text{C}_{\text{loss}}\) was computed using the number of driver activity label classes I, the target label class \(\text{m}_{\text{i}}\) and the logarithm of the \(\text{i}^{\text{th}}\) value in the final model output \(\widehat{\text{m}}_{\text{i}}\) as shown in Equation (11). \[\text{C}_{\text{loss}}=-\sum_{\text{x}=1}^{1}\text{m}_{\text{i}}\cdot\log \widehat{\text{m}}_{\text{i}}. \tag{11}\]  
To learn robust features from the unified LRCN model, overcome the imbalanced data between the 9 classes, and increase the diversity of our training set, we applied various data augmentation strategies. This also helped to reduce the dependence of the proposed model on spurious characteristics and increase the generalization ability. As such, rotation either clockwise or counterclockwise, zooming in, and zooming out augmentation techniques were applied to regulate the changes in the pose of the camera sensor and to cope with changes in the size of the driver due to fluctuations of the driver's seat position which can occur while driving. A gamma correction was also utilized with a gamma value of 1.0 to manipulate the brightness and intensity of the sequence image caused by sunlight and illumination. The remaining data augmentation techniques were saturation changing and adding salt and pepper noise. Note that the input data of our model is not a single image but a sequence of five series images. Thus, the augmentation strategy has to be performed sequence by sequence instead of image by image. In this sense, a randomly selected augmentation method is applied for every sequence input. The adjustable parameters used in each augmentation method were selected randomly within the predetermined range. For instance, when applying a rotation to the five sequence inputs, the parameter is the angle at which the image randomly rotates between the range of [\(-\)10, +10] degrees. At this moment, all five input images rotate at an equal angle without losing the original information. To implement the augmentation system, Keras provides a data generator application programming interface (API), where we can load all of the image data and perform the augmentation process for a single type of input image.