# Driver Drowsiness Detection by Applying Deep Learning Techniques to Sequences of Images
## 3 Materials and Methods
### Alternative II: Deep Learning Combined with Fuzzy Logic
#### 3.2.1 Preprocessing  
The parameters that we are interested in discovering from the driver's image are the following:  
* Number of blinks (_blinks_).
* Average blinking duration (\(avg\_blink\)).
* Number of microsleeps, i.e., blinks with a duration of over 500 ms (_microsleeps_).
* Number of yawns (_yawns_).
* Time spent yawning (\(avg\_yawn\)).  
The first step to calculate these parameters is locating the driver's face. To do this, DLIB's face detector is used once again. However, in this case, we also use the landmark detector that DLIB provides, which is an implementation of [32], where the shape predictor uses an ensemble of regression trees. DLIB's landmark detector represents the facial features with a group of 68 points, 12 of them representing the eyes (6 points per eye), which allows us to know the position of the driver's eyes. Figure 7 shows an example of landmark detection.  
To calculate the parameters related to eyes and blinks, it is necessary to calculate first how opened or closed the driver's eyes are. To do this, we use the technique proposed in [33]: we calculate the eye aspect ratio (EAR) by measuring the distance between the top eyelid and the bottom eyelid, and we divide it by the eye width, thus obtaining openness values that usually range between 0.16 and 0.36. Experimentally, it is determined a threshold of 0.20, so that every time that the measured EAR is under 0.20, it is considered that the driver has closed his or her eyes. By calculating how many times and for how long the driver blinks, we gather the first three measures.  
Next, we want to use the face to detect if the driver is yawning. To do this, the face of the driver is cropped and preprocessed following the same process as described in Section 3.1.1. This image is then given as an input to a CNN that we have trained specifically to detect whether a person is yawning or not. This network is based on the EfficientNetB0 model: we use its architecture and its trained weights over the ImageNet domain, and we add new layers to the model to train the network on the detection of the specific actions that we are interested in.  
Figure 6: Alternative II overview.  
Figure 8 shows the architecture chosen for this network.  
To train this neural network, the YawDD dataset [24] is used, which provides videos from 30 different people yawning and driving normally. In particular, we use the videos that were recorded by a camera placed on the dashboard, in front of the driver. Videos from 18 people are used at the training phase, while 6 are used for validation and the other 6 are used for testing. The accuracy when classifying the drivers' yawns reached a notable 88% accuracy on the testing data used. To avoid possible errors, to classify a frame, the system calculates the average class of the last 20 frames (2 s), and it returns the class that appears more times.  
Thanks to this deep learning model, we can calculate how many seconds the driver has spent yawning, thus obtaining the remaining parameters that we need for the fuzzy inference system.