# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Attention
#### 3.3.3 Spatiotemporal Feature Enhancement Based on Self-Attention  
With the development of self-attention and Transformer, multi-head attention mechanisms have gained prominence as a vital component in the realm of lip-reading. (1) Spatiotemporal feature enhancement based on multi-head attention  
In 2020, [PERSON] et al. [57] integrated self-attention modules into the front end, allowing the model to weight features across different temporal steps through a series of self-attention blocks. On the back end, self-attention, multi-head attention, and Vanilla Attention Blocks were employed, enabling the model to maintain sensitivity to the entire input sequence while capturing correlations between different parts of the sequence. This approach achieved optimal recognition performance in multilingual lip-reading. In the context of a large vocabulary system under noisy conditions, a feature-enhanced network constructed with multi-head self-attention blocks effectively harnesses auxiliary information streams, making it suitable for a large vocabulary audio-visual lip-reading across diverse scenarios. Experimental validations demonstrate that this approach exhibits robust performance in large vocabulary recognition, even in noisy environments [106]. In 2023, to recognize the complexity of a single lip shape corresponding to multiple words, [PERSON] et al. [107] expanded upon the framework of multi-head attention blocks to design a multi-head K-V memory network, adept at modeling the intricate one-to-many mapping relationships between lip movements, thereby enhancing lip-reading performance even in challenging noisy environments. (2) Spatiotemporal feature enhancement based on TransformerThe application of Transformer models in the enhancement of audio and visual features has increasingly centered on the domain of multimodal feature enhancement, yielding significant advancements in recent years that further enrich the interplay between disparate sensory modalities. For example, in 2022, to harness the capabilities of multi-head self-attention mechanisms, a 3D Visual Transformer (3 DCvT) is introduced to adeptly learn the latent distribution of the input modality, facilitating a more profound extraction of spatiotemporal features across successive frames [108]. In the realm of audio-visual multimodal lip-reading, in 2022, [PERSON] et al. [100] utilized the Transformer framework for self-supervised learning, where the front end employs six-layer Transformers to separately extract visual and audio features, succeeded by another six-layer Transformer that refines the fused audio-visual representations; the back end similarly incorporates six-layer Transformers for decoding, implementing CTC loss to facilitate the self-supervised learning process. For facilitating multimodal alignment within the phoneme space, in 2023, [PERSON]. [101] harnessed Transformer for feature extraction and enhancement; they proposed an open-modal speech recognition (OpenSR) system that permits models trained on a singular modality, such as pure audio, to be effectively applied across multiple modalities, including both pure visual and audio-visual inputs. This innovative system not only enables modal transfer between modalities but also achieves exceptionally competitive zero-shot performance, outpacing conventional few-shot and full-shot lip-reading methods. A comparative analysis of three Transformer-based lip-reading models, Transformer-CTC, Av-HuBERT, and Moco-word2 vec, revealed that the Moco framework, which elegantly intertwines convolutional networks with Transformers, consistently outperformed its counterparts across video, audio, and audio-visual recognition tasks [109]. This finding underscores the notion that the synergy of convolutional and Transformer networks substantially enriches feature expressiveness, leading to remarkable advancements in model performance.