# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Audio-Visual Assisting
#### 3.5.1 Audio Features Assisting Visual Recognition  
In the realm of lip-reading, the incorporation of auditory features to strengthen visual recognition has emerged as a pivotal strategy for overcoming the shortcomings of cross-modal enhancement methods. Extensive research has been dedicated to multimodal feature enhancement techniques that capitalize on the synergistic potential of auditory and visual signals to elevate recognition performance. Among these, cross-modal fusion and audio-visual memory augmentation stand out as prominent methodologies that facilitate audio-assisted visual recognition, effectively bridging the gap between the two modalities and enriching the overall comprehension of spoken communication. (1) Cross-modal fusion enhancement  
Cross-modal fusion is established through the careful design of feature enhancement networks meticulously crafted for the distinct characteristics of auditory and visual modalities. By harmonizing the strengths of both types of data, this intricate architecture empowers auditory features to augment and amplify visual cues, cultivating a more comprehensive and robust recognition system, and simultaneously paving the way for more intricate interpretations of multimodal information. In 2018, [PERSON] et al. [120] unveiled SynNet, a dual-stream convolutional neural network that employs cross-modal self-supervised learning to effectively capture the joint embedding of auditory signals and lip movements from unlabeled data, demonstrating superior performance compared to existing methodologies across prominent datasets and excelling in tasks such as audio-visual synchronization, active speaker detection, and multi-frame temporal lip-reading. By deftly weaving together disparate data sources, SynNet has not only propelled the field forward but also redefined the possibilities of multimodal recognition. In 2020, [PERSON] et al. [121] presented an innovative approach by introducing a 128-dimensional subspace designed to encapsulate the feature vectors of speech signals alongside their corresponding lip movements. They conceptualized lip-reading as an unconstrained processing of natural speech signals within video sequences. To achieve this, features were meticulously extracted from both audio and visual channels: the audio channel employed 1D convolution to derive temporal features, while the visual channel utilized 3D convolution to generate rich visual feature vectors. These temporal and visual features were subsequently transformed into long-term audio and visual representations using Long Short-Term Memory (LSTM) networks, culminating in feature alignment and model training and enhancing the robustness and efficacy of lip-reading. Figure 13: The typical spatiotemporal features enhancement methods based on audio-visual assisting. (**a**) The architecture of Ref. [120]. (**b**) The architecture of Ref. [121]. (**c**) The architecture of Ref. [122]. (**d**) The architecture of Ref. [123]. (**e**) The architecture of Ref. [124]. These enhancement methods for cross-modal fusion hinge upon the effective alignment of features from both modalities, necessitating a high quality of speech signal input. This critical requirement imposes a significant influence on the overall performance of the model, as the integrity of the sound data serves as the foundation for accurate and robust multimodal processing. (2) Audio-visual memory enhancement  
The challenges posed by articulatory movements in lip-reading can be attributed primarily to two interrelated factors: the inherent lack of information conveyed by lip movements and the presence of homophonic words that appear similar on the lips but differ in meaning. These issues represent significant roadblocks to achieving high accuracy in lip-reading. In response to these obstacles, researchers have sought to address the limitations by exploring dual-modal approaches that enhance audiovisual memory, thereby striving to improve the accuracy and effectiveness of lip-reading.