# Lip2 Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features
## 3 Gabor-Based Visual Feature Extraction
### Proposed Feature Extraction Approach  
Figure 1 illustrates the feature extraction flow chart, which involves tracking the ROI, filtering, and calculating Gabor lip region features, which will be explained in more detail below. Gabor Lip Image Generation  
Given a sequence of images \(I_{n}\) (where \(n=1\dots N\)) extracted from a video file, the widely used Dlib toolkit [50], along with a trained model file, Shape-Predictor-68-Face-Landmarks, used in previous research by the authors [3; 11], is used to identify 68 landmark face features. To detect the mouth region (ROI), four features are selected, represented by four \(x\) and \(y\) coordinate pairs \(L_{nx}(1,2,3,4)\) and \(L_{ny}(1,2,3,4)\), respectively, and \(C_{ntL}(X,Y)\), the centre point of the ROI (mouth region), is calculated as shown in Figure 2.  
Figure 1: Flowchart of the lip feature extraction system showing frame extraction, ROI identification and extraction, Gabor transform, and lip feature extraction. Figure 2: (**left**) Landmark features detected by the dlib toolkit, and (**right**) ROI extracted using the 4 landmark features. Horizontal Gabor filters of greyscale images are then calculated, using the real component of a Fast Fourier Transform:  
\[g(x,y;\lambda,\theta,\psi,\sigma,\gamma)=exp(-\frac{x\
u^{2}+\gamma^{2}y\
u^{2}} {2\sigma^{2}})\cos(2\pi\frac{x\
u}{\lambda}+\psi) \tag{1}\]  
where:  
\[x\prime=x\cos\theta+y\sin\theta\]  
\[y\prime=-x\sin\theta+y\cos\theta\]  
This is implemented using the Opencv Python function:  
\[cv2\_getGaborKernel((Ksize,Ksize),\sigma,\theta,\lambda,\gamma,\psi) \tag{2}\]  
This has six parameters:  
_Ksize_ is the Gabor kernel size;  
\(\sigma\) is the standard deviation of the Gaussian function used in the Gabor filter;  
\(\theta\) is the orientation of the normal to the parallel stripes of the Gabor function, which is 90 degrees;  
\(\lambda\) is the sinusoidal factor wavelength in the equation;  
\(\gamma\) is the spatial aspect ratio;  
\(\psi\) is the phase offset, defined as 0 by default. The above six parameters control the shape and size of the Gabor function. The role of each parameter is discussed in detail below. To illustrate the effects of parameters, the following values were chosen as a starting point: _ksize_ = 50, \(\theta\) = 90, \(\lambda\) = 10, \(\sigma\) = 5, \(\gamma\) = 0.5, \(\psi\) =0. Figure 3 is a GRID dataset image that is used here to demonstrate Gabor feature extraction in the following illustrations. _Ksize_: When varying _ksize_, the size of the convolution kernel varies. Figure 4 shows example results when applying _ksize_ with 5, 10, 25, 50 pixels. As shown in Figure 4b, when _Ksize_ = 15, more efficient information about facial features can be extracted. That means a proper Gabor parameter is vital for feature extraction. Wavelength (\(\lambda\)): The wavelength governs the width of the strips of the Gabor function. Increasing the wavelength produces thicker strips and decreasing the wavelength produces thinner strips. Keeping other parameters unchanged and changing the wavelength to 5, 10, 15, and 20, the stripes shown in Figure 5 get thicker. Orientation (\(\theta\)): Theta (\(\theta\)) controls the orientation of the Gabor function. The zero-degree theta corresponds to the vertical position of the Gabor function. Figure 6 shows the different orientations of the Gabor kernel. Different orientations of the Gabor filter highlight different features of the face. In Figure 6b, the vertical filter (\(\theta\) = 0) obtains more vertical information, such as ears and neck, whereas the Horizontal filter (\(\theta\) = 90) receives information about the eyes and mouth. The 45 and 135-degree filters can highlight other information. Figure 3: A sample image used to demonstrate Gabor feature extraction. Figure 4: Gabor kernels and facial Gabor features with different kernel sizes. Figure 5: The Gabor kernel and facial Gabor features with different wavelengths. Figure 1:0 The Gabor kernel and facial Gabor features with different orientations.