# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Other Spatiotemporal Feature Enhancement Methods  
Building upon these advancements in integrating visual information for improved speech recognition, researchers have also incorporated various techniques such as restricted Boltzmann machines, moment features, optical flow, graph convolution, and variational temporal masking into convolutional neural networks (CNNs). These elements work together to construct hybrid front-end modules that significantly enhance spatiotemporal feature extraction capabilities. By blending these diverse methodologies, the aim is to refine the analysis of intricate data patterns, thereby advancing the effectiveness of applications in both speech and visual recognition. Figure 14 shows the schematic diagram of other spatiotemporal feature enhancement methods. A restricted Boltzmann machine (RBM) is a class of stochastic neural networks designed to model and learn the underlying probability distribution of its input data for tasks such as feature extraction, dimensionality reduction, and collaborative filtering. To jointly learn visual features of the lip region from multiple perspectives, [PERSON] et al. embraced the multi-branch approach to learn the visual features of lips from diverse perspectives. This architecture enabled the dynamic modeling of the temporal characteristics inherent in each perspective, ensuring that the subtleties of lip movements were accurately captured, resulting in a comprehensive representation that significantly enriches the understanding of lip dynamics in the context of speech recognition [126]. Orthogonal Hahn moments, as a feature-extracting tool of lip movement, is merged in the first layer of the CNN architecture to construct the Hahn Convolutional Neural Network (HCNN) [127]. This method utilizes discrete orthogonal Hahn moments to compute the moments of the input images, resulting in a moment matrix whose dimensions are determined by the order of the moments, providing an optimized representation of the images, significantly reducing the dimensionality of the processed data while preserving crucial features essential for subsequent analysis. Meanwhile, efficient strategies such as the enhanced visually derived Wiener filter (EVWF) and a Variational Time Masking (VTM) module are effective in enhancing spatiotemporal feature extraction capabilities in a backend structure. Specially, EVWF is used to estimate the clean audio power spectrum and eliminate voice. Activity detection (VAD) and noise estimation for visual-driven speech recognition [123]. The VTM module automatically analyzes the frame-level feature importance, aiming to restrict the flow of irrelevant or noisy visual features to the back end network without compromising prediction accuracy, which contributes to enhancing the interpretability and generalization of lip-reading models [128]. Optical flow, as a computer vision technique that estimates the motion of objects between consecutive frames in a video for accurate tracking motion-based recognition tasks, has demonstrated satisfactory performance levels comparable to video data. Building on this insight, optical flow is integrated as an additional input alongside grayscale video to construct a dual-stream fusion method for extracting a more comprehensive extraction of spatiotemporal features [129]. Moreover, at the front end of this methodology, a 3D Inception network is employed to learn these features from both the grayscale video and optical flow data. The significant classification accuracy on large-scale image and video datasets underscores the effectiveness of multiple data modalities in the advancement of lip-reading. Figure 14: _Cont._  
Graph convolution is a neural network operation consisting of the learning of features by aggregating information, the relational structure, and connectivity inherent in graph-structured data.