# Convolutional Two-Stream Network Using Multi-Facial Feature Fusion for Driver Fatigue Detection
## 2 Methods
### Face Detection and Key Area Positioning  
Driver fatigue detection in real driving videos can be challenging because faces are affected by many factors such as the lighting conditions and the driver's gender, facial gestures, and facial expressions, etc. However, low-cost in-car cameras can only take low-resolution videos. Therefore, a high-performance face detector was needed. Even with a specific face area, positioning of the mouth and eye area was also very important, which contained important fatigue characteristics of the driver. The Adaboost face detection algorithm [27], based on Haar features of the face, is not effective enough in a real, complex environment. It also cannot determine the eye area and mouth area. MTCNN [28] is known as one of the fastest and most accurate face detectors. With a cascading structure, MTCNN can jointly achieve rapid face detection and alignment. As a result of face detection and alignment, MTCNN obtained the facial bounding box and facial landmarks. In this paper, we used MTCNN for face detection and face alignment tasks. MTCNN consists of three network architectures (P-Net, R-Net, and O-Net). In order to achieve scale invariance, the given image was scaled to different scales to form an image pyramid. In the first stage, shallow CNNs quickly generated candidate windows; in the second stage, more complex CNNs filtered candidate windows and discarded a large number of overlapping windows; in the third stage, more powerful CNNs were used to decide whether the candidate window should be discarded, while displaying five facial key positionings. Proposal Network (P-Net) (shown in Figure 2): The main function of this network structure was to obtain the regression vector of the candidate window and bounding box in the face area. At the same time, it used the bounding box to do the regression and calibrate the candidate window, and then it merged the highly overlapping candidate boxes by non-maximum suppression (NMS). Figure 1: Algorithm flowchart for driver fatigue detection. All input samples were first resized into \(12\times 12\times 3\), and finally the P-Net output was obtained by a 1 \(\times\) 1 convolution kernel of three different output channels. P-Net output was divided into three parts: (1) face classification--the probability that the input image was a face; (2) bounding box--the position of the rectangle; and (3) facial landmark localization--the five key points of the input face sample. Refine Network (R-Net) (shown in Figure 3): This network structure also removed the false positive region through bounding box regression and non-maximum suppression. However, since the network structure had one more fully connected layer than the P-Net network structure, a better effect of suppressing false positives could be obtained. All input samples were first resized to \(24\times 24\times 3\), and finally the R-Net output was obtained by the fully connected layer. R-Net output was divided into three parts: (1) face classification--the probability that the input image was a face; (2) bounding box--the position of the rectangle; and (3) facial landmark localization--the five key points of the input face sample. Output Network (O-Net) (shown in Figure 4): This network structure had one more convolutional layer than R-Net, so the result of the processing was finer. The network worked similarly to R-Net, but it supervised the face area and obtained five coordinates representing the left eye, right eye, nose, left part of lip, and right part of lip. All input samples were first resized to \(48\times 48\times 3\) dimensions, and finally the O-Net output was obtained by the fully connected layer. O-Net output was divided into three parts: (1) face classification--the probability that the input image was a face; (2) bounding box--the position of the rectangle; and (3) facial landmark localization--the five key points of the input face sample. All three networks set a threshold that represented the degree of overlap of face candidate windows in non-maximal suppression.