# Convolutional Two-Stream Network Using Multi-Facial Feature Fusion for Driver Fatigue Detection
## 3 Experimental Results
### Experiment  
We trained our models using a training dataset with a stratified five-fold cross-validation [32], where data folds were chosen such that each fold had nearly the same class distribution as the original dataset, and it used an evaluation dataset for the test. Images were extracted one frame from every three frames in the videos, and they were labeled into five driver states: normal, drowsiness, nodding, talking, and yawning; the distribution of these classes in the dataset was around 5:9:2:5:3. Input images were first resized to a \(50\times 50\) size. The model input of the experiment without gamma correction was the original image, while the model with gamma correction input images had an average gray value of 120. We developed our models using the Keras framework and run experiments on GTX 1080 Ti. All of the layer weights were randomly initialized. We chose the hyper-parameters using a grid search. The network was trained using a batch gradient descent with a batch size of 128 and a dropout rate of 0.2. An initial learning rate of 0.1 was used in the optimization function Adadelta. Training was stopped when the validation loss did not improve for 50 iterations. The model was trained for around 230 iterations. The results are shown in Tables 1 and 2. We tested the model at four different levels: (1) different scenarios; (2) different driving states; (3) different derived models; (4) average performance. Performances of the models without gamma correction, fatigue detection network (FDN), and with gamma correction, gamma fatigue detection network (GFDN), were comparable to state-of-the-art methods. In Table 1, we showed a comparison with state-of-the-art methods. Our models outperformed the existing methods in all of the scenarios, and the average performance surpassed all the state-of-the-art methods, which achieved a 97.06% accuracy. In addition, GFDN increased accuracy by 2% compared to FDN in a night environment. We showed another result in Table 2. We obtained features before the softmax layer in the GFDN model as inputs, and we tested them with other classifiers. We chose and tuned four classification algorithms including k-nearest neighbors (KNNs) [33], centroid displacement-based k-nearest neighbors (CDNNs) [34], support vector machine (SVM) [35], and random forest (RF) [36]. Parameter k was  
Figure 9: Sample image from the National Tsing Hua University-Driver Drowsiness Detection (NTHU-DDD) dataset. (**a**) Bareface (nodding); (**b**) wearing glasses (normal); (**c**) wearing sunglasses (yawning); and (**d**) bareface at night (normal). It is not suitable to directly input the entire image, which contains extraneous information. The driver's face is only about \(200\times 150\) pixels, compared with the actual \(640\times 480\) image in the camera shot. The full image contains useless information and has a negative effect on classification. Furthermore, after the image is resized and sent to CNN, it carries much less correlative information, which makes it difficult to learn features. Inputting partial facial images can avoid inaccurate classifications. Therefore, inputting the left eye and mouth area, which are closely related to driver fatigue, is a sensible idea. Using only static images is not accurate enough for driver drowsiness detection; this is an action recognition task instead of an image recognition task. For instance, people are opening their mouths when both yawning and speaking. There is no difference in static images, while optical flow can reflect the difference. When the mouth and left eye are used as the inputs of the network, dynamic information between the continuous frames is not used. Better results can be obtained using a two-stream neural network that contains dynamic information in optical flow. Poor image quality caused by non-uniform gray-level distribution can have negative effects on calculation results, for example, images taken in too bright or too dark environments.