# Diverse Pose Lip-Reading Framework
## 1 Introduction  
Lip-reading is a skill to comprehend spoken words from sequences of lips movements. Human speech is bimodal: one is acoustic and other one is visual. The lip-reading affects the actual speech perception, and it is elaborated in [1]. Acoustic speech is closely related to the phoneme where phoneme is the smallest unit of the sound which distinguishes one word from other, for example, the phoneme \(k\) against words cat, kit, scat, skit and phoneme \(p\) against words pin, spin and tip. The visual unit is related to the viseme where viseme is a phoneme identical in appearance on lips. Visemes affect the acoustic/phoneme perception, and it alone can be used to recognize speech as shown in Figure 1.  
In the early ages, human lip reader experts comprehend the visual speech. There were professional lip-readers who used to interpret many insights from the historic silent movies. In crime scene investigations, the professional lip-readers would interpret conversation between offenders for clear evidence. However, there were problems with the human lip-readers like only experts could interpret the visual speech and the cost of interpretation was high. In addition, they were not reliable as there were chMoreover, the human lip-reader required more time to understand visual speech because different people have different speaking styles. Traditional lip-reading systems work in two steps: feature extraction and classification. In the first step, pixels from the mouth region of interest (ROI) are extracted as visual information. Then, discrete cosine transform (DCT) [2; 3], discrete wavelet transform (DWT) [3] and principal component analysis (PCA) [3; 4] are used to extract abstract images features. The advance geometric features obtained in this way possess the characteristics of low dimensionality and high robustness. In the second step, the extracted features are fed into the classifiers such as support vector machine (SVM) [5] and hidden Markov model (HMM) [6]. At present, machine learning has made significant progress in the field of computer vision (image representation, target detection, human behavior recognition and video recognition). Therefore, it is an inevitable trend of scientific research to shift the direction of automatic lip-reading technology from the traditional manual feature extraction classification model to the end-to-end machine learning architecture. Machines learning based lip-reading (MLR) methods extracts features from ROI abstract images which are more discriminative than the features extracted with DCT, DWT and PCA. These features are then fed into a classifier. Most of the current MLR methods performed lip-reading in frontal-view videos. However, relatively lesser works focused on multiple-view lip reading. Multiple-view MLR performed well in high quality videos, but they performed poorly in low quality videos because MLR requires high resolution videos for training. Moreover, in multiple-view MLR, the mouth region is often not clearly visible due to the view angle which significantly degrades the performance of these methods. This issue has been addressed by some methods. However, such methods failed to achieve high accuracy in extreme view angles such as 90\({}^{\circ}\) and beyond. How to achieve high accuracy in machine learning based lip reading for low-quality extreme view angles is the research question our method aims to answer. Our work focuses on lip-reading in multiple poses. In this research work, a new dataset containing low as well as high quality videos of multiple poses is created. Secondly, a speaker dependent lip-reading method has been proposed which can recognize sentences from low as well as high quality videos with multiple poses of a single person. The multiple poses are frontalized using a face frontalization method (FF-GAN) that has not been used in the lip-reading field till now. For improving the quality of video, the proposed lip-reading method uses the CLAHE method. Next, the FF-GAN is used to make frontal faces from multiple poses.