# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 2 Classification of Lip-Reading Methods
### Lip-Reading Methods Based on Machine Learning  
Lip-reading methods based on machine learning include two main stages: feature extraction and classification. As illustrated in Figure 4a, the process begins with the application of various transformations, such as the Discrete Cosine Transform (DCT) [15], Wavelet Transform (WT) [16], Principal Component Analysis (PCA) [17], Fourier Transform (FT) [18], and Linear Discriminant Analysis (LDA) [19,20], to extract feature descriptors from each region of interest (ROI) in video frames [21]. These transformations may also be used to extract motion features, such as optical flow [22,23], or to derive geometric features such as height, width, area, lip shape, histograms [24,25], color (texture) features [26], or model-based features (e.g., Active Appearance Model (AAM) [27], Active Shape Model (ASM) [28], Active Contour Model (ACM) [29]). After feature extraction, these descriptors are concatenated and fed into classifiers such as Template Matching [30], Hidden Markov Models (HMM) [17,31,32], Gaussian Mixture Models (GMMs), Dynamic Time Warping (DTW) [33,34], K-Nearest Neighbors (KNN) [35], Random Forest Method (RFM) [36], Support Vector Machines (SVM) [35], and Naive Bayes (NB) [35] for classification. Figure 3: Classification of lip-reading methods. Feature extraction methods in machine learning, as illustrated in Table 1, primarily concentrate on extracting pixel features, transformation features, optical flow features, geometric features, texture features, and model-based features from lip images. Due to the effective capability of capturing visual features of lip images, machine learning-based feature extraction methods contribute to the long-term development of lip-reading. Nevertheless, the process of constructing these features is heavily complex and dependent on the structure of the model. This reliance on manual construction limits the scalability and broader application of these methods. Notably, these features are primarily spatial and fail to adequately reflect the temporal dynamics of lip movements, resulting in \begin{table}
\begin{tabular}{c c c} \hline
**Classification** & **Type of Features** & **Characteristic** \\ \hline pixel feature & multistage linear transformation, local pixel feature & The method uses lip-centered scanning lines as feature vectors, but it is sensitive to lighting variations and underperforms in high-complexity computations. \\ \hline image transformation feature & Discrete Cosine Transform (DCT), Wavelet Transform (WT), Principal Component Analysis (PCA), Fourier Transform (FT), and LDA & The method uses the transformation results of all pixels as feature vectors and extracts high-frequency components for detailed information. \\ \hline optical flow feature & optical flow field & The method extracts lip motion parameters and analyzes motion patterns but relies on precise positioning during preprocessing. \\ \hline color feature & color space & The method focuses on color images. \\ \hline model-based feature & active appearance model (AAM), active shape model (ASM), active contour model (ACM) & The method adjusts model parameters to approach the target but is prone to local minima and sensitive to the initial position. \\ \hline \end{tabular}
\end{table} 
Table 1: Main feature extraction methods used in machine learning-based lip-reading. Figure 4: The illustration of the principles for lip-reading methods. (**a**) is the principle of machine learning-based methods; (**b**) is the principle of visual feature-based methods; (**c**) is the principle of spatiotemporal feature-based methods; (**d**) is the principle of pulse feature-based methods. Arrows represent data flow. limited feature expressiveness and low model accuracy. Consequently, the development of simple and effective feature extraction methods is becoming increasingly crucial.