# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Attention
#### 3.3.3 Spatiotemporal Feature Enhancement Based on Self-Attention
(3) Spatiotemporal feature enhancement based on Transformer + CNN  
The Transformer architecture excels at capturing temporal features within lengthy sequences, showcasing its remarkable proficiency in modeling long-range dependencies. In contrast, convolutional networks (CNNs) are particularly effective at extracting local and spatial features, finely tuned for the retrieval of nuanced details. Thus, the harmonious integration of Transformer and CNN models facilitates the extraction of a richer tapestry of both temporal and spatial attributes, thereby significantly enhancing the performance of lip-reading. In 2022, the integration of 3D convolution into visual Transformers marked a notable advancement in the extraction of spatiotemporal features from sequential video frames. This innovative hybrid architecture exploits the temporal dynamics rendered by 3D convolution, while simultaneously harnessing the global feature extraction prowess of the Transformer. As a result, this synergy substantially enhances the model's capacity to decipher intricate patterns of dynamic visual information, i.e., offering a deeper understanding of the fluidity and complexity inherent in video content. not only facilitated the compression of the model but also effectively addressed the multifaceted issues at the intersection of spatial and temporal feature extraction, culminating in a robust and efficient solution for contemporary visual tasks. The advent of regularized Dropout represented a notable improvement in achieving alignment between the training and reducing stages, fostering a more cohesive learning strategy [103]. In a parallel vein, a relaxed attention Transformer is designed to alleviate the overfitting problem [110]. By incorporating regularization terms into the self-attention layers of the encoder and implementing a relaxed decoding approach that constrained the internal language model within the decoder, their method effectively mitigated the risks of overfitting, thereby promoting a more robust and generalizable performance. Certainly, the Visual Transformer Pooling Block (VTP) also can be merged into Transformer framework as an innovative constructor to effectively aggregate prior visual representations [104]. The key concept of this method is to optimize the flow of information, which facilitates a more refined comprehension of visual data by emphasizing relevant features. Experimental results implied that by incorporating this visual feature enhancement module, the VTP exhibited exceptional performance in sub-word-level lip-reading, reflecting the enhanced capacity of attention mechanisms to distill and prioritize critical visual cues in the pursuit of accurate and discriminating recognition. The advent of HVSEM, a hybrid approach that intricately combines visual sub-word and end-to-end modeling, marked a significant evolution in the field. This method employs cascaded Transformers enhanced by cross-layer attention mechanisms, effectively refining visual features and bolstering the exchange of information between the cascaded structures. By doing so, it not only enriches the feature representation but also minimizes error accumulation, a critical challenge in complex modeling scenarios. The efficacy of HVSEM has been demonstrated through its successful application in both visual sub-word collaboration and end-to-end word-level lip-reading, showcasing the potent synergy derived from advanced attention mechanisms in navigating intricate recognition tasks. (4) Spatiotemporal feature enhancement based on multi-attention combination  
A single attention mechanism may confine its focus to limited features, whereas the integration of multiple attention mechanisms significantly broadens the scope and depth of a single attention, thereby enhancing feature representation and facilitating improved model generalization.