# Driving Fatigue Detection Based on the Combination of Multi-Branch 3D-CNN and Attention Mechanism
## 3 Materials and Methods
### Face Tracking and Privacy Protection Processing  
When the driver's video is introduced into our fatigue detection system, the first thing we need to do is to detect and track the key points on the driver's face. We use the Dlib [18] library to detect and track the key areas of the face image. Dlib is a platform-independent programming library whose source code is based on a C++ implementation, but which provides a Python interface. It is more versatile and superior in image processing and facial feature extraction, classification and comparison. The Dlib library provides a facial landmark detector containing 68 coordinate points of a human face. The Dlib library detects and tracks the driver's face in real time and at the same time accurately obtains the driver's key point coordinates. According to the coordinate output of the feature points, the micro movements of the human face can be accurately tracked. It is worth mentioning that a Dlib library-based application was designed to monitor the driver's face and head features in real time while protecting the driver's facial privacy. The driver's blinking, nodding and yawning behaviors are monitored and recorded in real time. The interface of the software is shown in Figure 1. The video recording device is located on the upper part of the vehicle's front windshield and can record the driver's facial changes in real time without affecting the driver's normal driving. For the processing of driver's facial privacy protection, the method of template fusion is used to fuse face features. First, we obtained a face template database through web crawling and dataset integration. After that, we classified the attributes of faces (e.g., age, expression, skin quality,skin tone, angle, etc.,) and after the driver's face enters the collection range, the model will traverse the database to match templates with similar attributes to the driver's image; the matching algorithm is shown below. \[y=\underset{\text{Lab}}{\text{argmin}}\Bigg{(} \begin{array}{c}(O_{1}-M_{1})^{2} \times\theta_{1}+(O_{2}-M_{2})^{2}\times\theta_{2}+\\ (O_{3}-M_{3})^{2}\times\theta_{3}+\dots(O_{n}-M_{n})^{2}\times\theta_{n}\end{array}  \Bigg{)} \tag{1}\]  
where Lab denotes the template library, \(O_{1\sim n}\) denotes the original image face attributes, \(M_{1\sim n}\) denotes the matching template image attributes and \(\theta_{1\sim n}\) denotes the coefficients of the corresponding attributes. After matching the most suitable face template, we have to segment the face. We segment the face region based on the key points of the face identified by Dlib using the Delaunay triangulation algorithm [19]. The Delaunay triangulation algorithm can segment the points on the plane into triangles with good properties. The segmentation of the template image and the original image is consistent, which facilitates the fusion of facial key points afterwards. The generation of the fused image is based on the following equation:  
\[I_{p}=(1-\xi)I_{o}+\xi I_{m} \tag{2}\]  
where \(I_{p}\), \(I_{o}\) and \(I_{m}\) represent the fused face image, the original image and the matched image in the template library, respectively. \(\xi\) represents the deformation intensity, which is an artificially set parameter greater than 0 and less than 1. Since the key regions for fatigue detection are eyes and mouth, the weights in these two regions are set to 0 to ensure the accuracy of fatigue detection. The fused images are presented in the software interface after 3D simulation as shown in Figure 2, which solves the problem of driver facial information leakage from the bottom and better ensures the security of driver facial privacy. In the actual driving environment, the phenomenon of missing detection of the driver's facial features (camera occlusion, driver nodding, etc.) may occur. In order to ensure the accuracy of detection effect and maximize driver safety, frame aggregation is used to overcome the phenomenon of missing detection.