# Convolutional Two-Stream Network Using Multi-Facial Feature Fusion for Driver Fatigue Detection
## 2 Methods
### Optical Flow Calculation  
In real, three-dimensional space, the physical concept that describes the state of motion of an object is motion field. In the space of computer vision, the signal received by the computer is often two-dimensional image information. Because one dimension of information was lacking, it was no longer suitable for us to use motion fields to describe motion state. The optical flow field was used to describe the movement of three-dimensional space objects in the two-dimensional image, reflecting the motion vector field pixel. As indicators of driver fatigue, yawning, blinking, etc. are not a static state but a dynamic action. Therefore, just a static image was not enough. By utilizing the change of pixels over time in the image sequence and the correlation between adjacent frames, optical flow was determined based on the corresponding relationship between the previous frame and the current frame, and it contained the dynamic information between adjacent frames. Unlike the method of using continuous frames for action recognition, such as LSTM and 3D-CNN, we used the dynamic information contained in optical flow data to replace the dynamic information provided by successive frames. In this paper, we fused the features of static and dynamic information to make driver fatigue detection better than using only static images. Optical flow is observed in the imaging plane, and it is the instantaneous velocity of the pixel motion of an object moving through space. It uses the change in pixels over time in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, then it calculates the motion information of objects between adjacent frames. Suppose there is a vector set \((x,y,t)\) in every moment, which represents the instantaneous velocity of the specified coordinate \((x,y)\) at the moment of \(t\). Let \(I(x,y,t)\) represent the pixel brightness of point \((x,y)\) at the moment of \(t\), and in a very short period of time \(\Delta t\), \((x,\,y)\) increases \((\Delta x,\,\Delta y)\) respectively, thus, we can get:  
\[ \begin{array}{l}I(x+\Delta x,y+\Delta y,t+\Delta t)=\\ I(x,y,t)+\
icefrac{{\partial I}}{{\partial x}}\Delta x+\
icefrac{{\partial I }}{{\partial y}}\Delta y+\
icefrac{{\partial I}}{{\partial t}}\Delta t\end{array}  \tag{4}\]  
Figure 6: Original and gamma correction renderings. (**a**) Underexposed image. (**b**) Underexposed image after gamma correction. (**c**) Overexposed image. (**d**) Overexposed image after gamma correction. At the same time, taking into account that the displacement of two adjacent frames is short enough, represented by:  
\[I(x,y,t)=I(x+\Delta x,y+\Delta y,t+\Delta t) \tag{5}\]  
we get:  
\[\
icefrac{{\partial I}}{{\partial x}}\Delta x+\
icefrac{{\partial I}}{{\partial y }}\Delta y+\
icefrac{{\partial I}}{{\partial t}}\Delta t=0 \tag{6}\]  
\[\
icefrac{{\partial I}}{{\partial x}}\Delta x/\Delta t+\
icefrac{{\partial I}} {{\partial y}}\Delta y/\Delta t+\
icefrac{{\partial I}}{{\partial t}}\Delta t =0 \tag{7}\]  
Since:  
\[\Delta x/\Delta t=v_{x},\
icefrac{{\Delta y}}{{\Delta t}}=v_{y} \tag{8}\]  
The final conclusion can be drawn as:  
\[\
icefrac{{\partial I}}{{\partial x}}v_{x}+\
icefrac{{\partial I}}{{\partial y }}v_{y}+\
icefrac{{\partial I}}{{\partial t}}=0 \tag{9}\]  
where \(v_{x},v_{y}\) is the speed of \(x,y\) respectively, which is called the optical flow of \(I(x,y,t)\). The Farneback algorithm [31] is a method of calculating dense optical flow. First of all, it approximates each neighborhood of two frames with a second-degree polynomial, which can be done efficiently with polynomial expansion transformations. Next, by observing how an exact polynomial transforms under translation, a method of estimating the optical flow is derived from the polynomial expansion coefficients. With this dense optical flow, image registration at the pixel level is possible.