# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Spatial Features
#### 3.1.1 Spatial Feature Enhancement Based on ResNet
Figure 5: _Cont._  
Similarly, the spatiotemporal convolution preceding the ResNet-34 architecture incorporates ReLU activation and 2D CNN techniques, which also contribute to the reduction of the spatial dimension at each time step into a one-dimensional tensor. Consequently, this approach effectively captures lip movements that evolve [69]. Thus, the combination of ResNet-34's depth and the spatiotemporal convolution techniques underscores a comprehensive strategy for enhancing lip movement recognition in dynamic contexts. (2) Multi-branch ResNet with 2D convolution  
It is noted that the feature enhancement capabilities of single-branch ResNet are inherently limited, restricting significant improvements in lip-reading performance. To address this, multi-branch enhancement methods have been proposed to integrate both global and local features, to improve the overall effectiveness of visual feature enhancement. [PERSON] et al. [68] introduced a dual-branch system that harnesses the capabilities of ResNet-18 to seamlessly integrate inputs from grayscale video streams and deformation flows, the latter generated through an encoder-decoder process. In this setup, mutual learning performed by knowledge distillation loss is designed to facilitate joint training of two branches of visual features, significantly promoting the overall system performance. In a similar vein, [PERSON] et al. [58] 2022 proposed a multi-branch lip-reading method that integrates spatiotemporal, spatial, and heatmap features. ResNet-18 is responsible for extracting features from both spatiotemporal views and spatial views, combining appearance and shape features to yield more discriminative visual representations. Notably, an adaptive spatial graph model (ASGM) was introduced in the heatmap feature branch to automatically learn the spatial topology and dynamic variations of the lips, demonstrating that the combination of ResNet-18 and graph convolutional networks significantly underscores higher discriminative recognition performance. The introduction of ResNet-18 for visual feature extraction marked a pivotal moment in capturing complementary information, which was subsequently enriched by a dual-stream fusion strategy [59]. This innovative approach promotes mutual learning between the global and local feature streams, ultimately enhancing the overall visual features. In summary, ResNet, celebrated for its robust spatial feature modeling capabilities, has made significant strides in articulating spatiotemporal features through 2D convolution. However, despite these advancements, its ability to effectively capture and describe temporal features remains noticeably limited. This interplay between enhanced spatial representation and the challenges of temporal modeling highlights the ongoing quest for more comprehensive methods in lip movement recognition. (3) ResNet with 3D convolution  
To enhance the modeling ability of temporal features, researchers have attempted to incorporate 3D convolution into the ResNet architecture in that 3D convolution, which processes both temporal and spatial information simultaneously, is especially efficient for  
Figure 5: Spatial feature enhancement methods based on ResNet. (**a**) The architecture of Ref. [58]. (**b**) The architecture of Ref. [61]. (**c**) The architecture of Ref. [63]. (**d**) The architecture of Ref. [66]. (**e**) The architecture of Ref. [68]. capturing the dynamics of video sequences. A straightforward method involves replacing certain layers of ResNet with 3D convolutional layers. In 2021, [PERSON] et al. [70] tracked the issue of the modeling ability of temporal features by transiting the first convolutional and pooling layers of ResNet from 2D to 3D, in which a compact 3D convolutional unit, Stingy Residual 3D (StiRes3D), was also introduced. Heterogeneous convolutional kernels are imposed in this unit across different input channels and then incorporated into both channel-wise and pointwise convolutions within the compact convolutional block.