# Driver Behavior Analysis via Two-Stream Deep Convolutional Neural Network
## 1 Introduction
The authors applied a histogram of gradients (HOG) and an AdaBoost classifier trained for each side of the face regions to classify cellphone usage as right hand, left hand, or no usage [PERSON] et al. [11] introduced a video-based hand detection dataset in an automotive environment and used the aggregate channel features object detector. In recent years, with the great success achieved by deep learning networks in computer vision [12; 13; 14], some deep-learning-based object detection models were applied to detect hand location. In 2016, [PERSON] et al. [15] trained a faster R-CNN model [16] to classify whether the hands are holding a steering wheel or not, and the results showed that the model could achieve a higher accuracy rate than found in Reference [10]. In 2016, [PERSON] et al. applied AlexNet to perform head pose estimation and used the stacked hourglass network in the refinement stage to estimate facial landmarks and refine the face localization [17]. The detected results can provide a basis for the estimation of the driver's state in terms of distraction and drowsiness. In addition, some studies were proposed to recognize the driver's behavior to detect distraction. Then, in Reference [18], the authors proposed DarNet and investigated the mixing of different models, i.e., CNNs, recurrent neural network (RNNs), and SVMs, to detect driver distraction. In 2018, [PERSON] et al. proposed Drive-Net to classify 10 distracted behaviors, which was composed of a convolutional neural network (CNN) and a random decision forest [19]. [PERSON] et al. [20] utilized four different CNN models including VGG-16 [21], AlexNet [12], GoogleNet [22; 23], and a residual network to classify 10 distracted behaviors. In addition, they developed a warning system that can alert the driver in real time when a distraction behavior is detected. According to their results, the authors observed that the deeper models could provide higher detection accuracy, but the cost of inference time was increased. The trade-off between accuracy and efficiency remains an issue. In addition, only spatial information was considered, but temporal information, related to important cues for behavior recognition, was ignored in recent works [19; 20; 24; 25; 26]. Although behavior analysis or action recognition was studied for many years in the computer vision field, driver behavior analysis is a specific issue and is challenging due to the light changing, occlusion, clutter, and subtle actions. In this study, we aimed to analyze driver behavior and detect manual distractions in order to develop a warning system for drivers. For action recognition, temporal information is important, as well as spatial information, but it was ignored in recent works of distracted behavior recognition. Hence, inspired by the famous network architecture, using a two-stream CNN model [27] composed of a spatial and a temporal stream ConvNet, the spatial and temporal information was extracted using powerful CNN models for the classification of common 10 distracted behaviors inside the vehicle. For the spatial information, an average pooling in the temporal domain was performed on 10 consecutive RGB images, and then the resulting map was input to the following convolutional layers. The feature map from the last convolutional layer was taken as the spatial features for further processing. On the other hand, TVL\({}^{1}\) optical flow [28] was firstly applied to extract the horizontal and vertical motion information between input frames, and then two stacks of flow images were concatenated and input to the temporal stream ConvNet. The feature map from the last convolutional layer was extracted as the temporal features. Note that, instead of using a three-dimensional (3D) ConvNet, which would suffer from large parameters and the lack of a pre-trained model, two-dimensional (2D) ConvNet was used to construct the spatial and temporal ConvNet streams, and they were pre-trained using the large-scale ImageNet [12].