# An Intelligent Real-Time Driver Activity Recognition System Using Spatio-Temporal Features
## 3 Methodology
### Spatial Feature Extraction  
This CNN is the most dominant source and powerful tool for feature extraction and representation of images. When the inputs are sequence data (video data), each input frame is represented by the CNN feature. Training a deep CNN for image classification requires a thousand large-scale datasets. Thus, it is common to reuse a pre-trained architecture such  
Figure 5: The sliding window process for the sequence input data. The window width is fixed to five frame images. \(N\) denotes the final sliding window number.  
as VGG-16 [32], AlexNet [34], ResNet [35], or MobileNet [39] by utilizing transfer learning, to leverage the knowledge gained for one task to solve the related ones to overcome the isolated learning system. As such, this study adopts VGG-16, which was originally proposed for image classification and has proven its effectiveness in feature extraction.  
Figure 6 illustrates the schematic diagram of the proposed unified LRCN network. Originally, the VGG16 was composed of 13 convolutional layers, 4 pooling layers, and 3 fully connected layers, and the input size was 224 \(\times\) 224 \(\times\) 3, which represents the width \(\times\) height \(\times\) channels, respectively. Note that the fully connected layer of this architecture is removed in this work. This allows us to vary the input size image differently from the original input size. As such, our input size image was downsized to 128 \(\times\) 128 \(\times\) 3 with five time step inputs; this is 42% less than the original CNN architecture input size and can highly contribute to reducing the computational memory. In Figure 6, b, n, h, w, and c correspond to the batch size, time steps, height, width, and the dimension of the output feature map, respectively. Every spatial location in the output feature map corresponds to the same locations in the input feature map.  
All of the thirteen convolutional layers of the CNN include 3 \(\times\) 3 filters with stride one and padding one, 2 \(\times\) 2 max-pooling with a stride of two, and a ReLU activation function to minimize the feature maps to half their size. Here, the pre-trained CNN network plays a key role in facilitating a robust feature extraction using both the convolutional 2D (Conv2D) and max-pooling layers. The other main reason for removing the fully connected layer is that the neuron activation occurs in the dense layer and is not capable of encoding the spatial information features, which is very essential for the next LSTM unit to capture the motion information of the sequential inputs. Unlike the Conv3D in [40] that performs the spatio-temporal operation which can be computationally intensive, the deployed Conv2D only performs the spatial feature operation to extract the primary spatial information of the driver activity image inputs and leaves the temporal extraction task to the LSTM unit. The Conv2D operation performed to extract the spatial features is expressed as follows:  
\[\text{C}_{\text{2D}}(\text{a},\text{b})=(\text{K}\text{*I})(\text{a},\text{b} )=\sum_{\text{x}=-\text{k}}^{\text{k}}\sum_{\text{x}=-\text{k}}^{\text{k}} \text{K}(\text{x},\text{y})\text{I}(\text{a}-\text{x},\text{b}-\text{y}), \tag{1}\]  
where I, K, and * indicate the two-dimensional input image, 2D kernel, and convolutional operation, respectively. \((\text{a},\text{b})\) is the element of the 2D input image tensor I, and \((\text{x},\text{y})\) is the index of the kernel filter. To achieve the Conv2D operation for all time-step inputs, the time distributed layer is applied as described in Section 3.5.