# Driving Fatigue Detection Based on the Combination of Multi-Branch 3D-CNN and Attention Mechanism
## 3 Materials and Methods
### 3D-CNN Module  
In the following, we describe our 3D-SE-Net fatigue detection architecture in depth. As shown in Figure 4a, the proposed framework consists of three branches, each with the same convolution and pooling operations, which are trained on images of the driver's left eye, right eye and mouth. After acquiring the grey-scale, gradient and optical flow images of the driver's video frame, the feature maps are passed into the corresponding 3D-CNN branches, each containing two 3D convolution operations and two max-pooling operations. The feature maps of each branch are merged together to generate a feature map cube and finally to enter the SE module [20] to compress and excite the features on the feature channels and then re-assign the weights to each channel to obtain the final fatigue recognition result. In order to prevent the gradient from disappearing and to keep the convergence rate of the model in a steady state, each 3D convolution operation is followed by a linear rectification function (ReLU) [21] as the activation function, which facilitates the fitting of the training data while preventing overfitting to a certain extent.  
The parameters and structure of each 3D-CNN branch are shown in Figure 4b. First, in the input layer, we select 10 consecutive frames of the driver's face of size \(48*48\) from the video stream as the input to the 3D-CNN network and then obtain an image containing five channels of information from the original image of the driver in the hardwired layer, following the method used by [PERSON] et al. [13] to encode the original image with optical flow and gradient features. They are greyscale, optical flow-x, optical flow-y, gradient-x and gradient-y. Of these, the greyscale and gradient images we can obtain directly by frame  
Figure 3: Schematic diagram of frame aggregation strategy.  
Figure 2: Schematic diagram of privacy protection mechanism.  
by-frame computation, whereas the optical flow image is special in that it contains motion information between consecutive frames, so it must be obtained by two consecutive frames. Thus, in the hardwired layer we get 48 frames (\(10*3+(10{-}1)*2=48\)) of the driver's face image containing 5 channels of information. These sample images from different channels actually encode our prior knowledge of the features, which helps to improve the convergence and generalization of the network.  
The first 3D convolution layer, Conv1, contains two sets of 3D convolution kernels of size \(7*7*3\), where \(7*7\) denotes the spatial dimension and 3 denotes the temporal dimension, so that we obtain two sets of feature maps of size \(42*42\) and 38 channels. After activation by the ReLU function, we apply the max pooling layer Pool1 with the kernel size of \(2*2\) to downsample the feature maps, which maintains the number of feature maps but decreases the spatial resolution. To enhance the number of feature maps, we employ three sets of convolution kernels of size \(7*7*3\) for the second 3D convolution operation, yielding six sets of feature maps of channel number 28 and size \(15*15\). After activation by the ReLU function into the second maximum pooling layer Pool2, this layer has exactly the same parameters as the Pool1 layer, so we get a feature map with no change in the number of channels and a size of \(8*8\).  
After two convolution and pooling operations, we stack the feature maps of the three tributaries according to the channel arrangement and transform the 10 input images of size \(48*48\) into a feature map cube of \(8*8*504\). Subsequently, the feature map cube is sent to the SE module for weight optimization and reassignment; the specific operations are described in Section 3.3 in the expansion. Finally, we use stochastic gradient descent(SGD) [22] to train the whole network model.