# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Spatial Features
#### 3.1.1 Spatial Feature Enhancement Based on ResNet  
ResNet, known for its simple structure and effective recognition performance, is one of the earliest convolutional neural networks used in lip-reading and has been extensively applied for feature enhancement [57; 58; 59; 60; 61; 62]. Enhancement based on ResNet is typically implemented using either 2D convolution, 3D convolution, or a combination of both convolutional approaches. Figure 5 shows the schematic diagram of the spatial feature enhancement method based on ResNet. (1) ResNet with 2D convolution  
The 2D convolution-based ResNet architecture proves particularly adept at processing spatial information. Notably, ResNet-18 and ResNet-34 have been employed in the early stages of research to extract and enhance spatiotemporal features in lip-reading. [PERSON] et al. [63] utilized ResNet-18 to refine features derived from 3D convolutional layers, integrating a GRU-based backend for temporal feature modeling, which resulted in superior recognition performance compared to conventional methods on the LRW dataset. Similarly, concurrent studies [64; 65] have typically adopted ResNet-18 as the front end to bolster spatiotemporal feature enhancement. However, a significant limitation of these models is their inability to effectively model temporal features. Building upon these advancements, [PERSON] et al. sought to enhance the temporal modeling capability by utilizing ResNet-18 to refine features extracted from 3D CNNs [57]. In this approach, they output a 512-dimensional vector for each time step, which was subsequently added to the corresponding position encoding and used as input for subsequent self-attention blocks. This integration of a temporal vector improved the model's ability to express features across time steps, thereby enhancing the alignment of spatiotemporal features in lip-reading videos. To improve the recognition of fine-grained lip movements, [PERSON] et al. [61] proposed HP-ResNet18, thereby enhancing the model's ability to describe visual actions. Specifically, HP-ResNet18 replaces the 2D convolutions of ResNet-18 with a Hierarchical Pyramid Convolution (HPConv), which employs a set of convolutional kernels of varying sizes. This modification enables the network to process input information at multiple spatial resolutions, facilitating the capture of both local and global contextual information and demonstrating strong generalization performance on the LRW and LRW-1000 datasets. Thus, both approaches--HP-ResNet18's focus on spatial resolution and [PERSON] et al. 's emphasis on temporal modeling [57]--demonstrate complementary strategies for advancing the effectiveness of lip movement recognition. Primarily, ResNet's convolution mainly uses sliding windows in small regions for feature extraction, and is less efficient in capturing global features for lip-reading in ResNet-18. To address this, [PERSON] et al. [66] proposed a lip-reading framework that incorporated solid dilated convolution by replacing ResNet-18 with the Residual Global Context Network (ResGNet) to overcome the limitations of traditional convolution in capturing global features. As an extension of ResNet-18, WideResNet-18 modified the layer arrangement of ResNet-18 and replaced the ReLU activation function with SiLU [67]. The recognition performance was evaluated on four datasets, including LRW, OuluVS, CUAVE, and SSSD, and demonstrated that the spatial feature extraction method based on ResNet-18 exhibits strong feature representation capabilities. Compared to ResNet-18, ResNet-34 offers greater depth and improved performance, making it particularly suitable for processing highly dynamic and variable lip images [62]. Furthermore, ResNet-34 employs max-pooling layers to progressively reduce the spatial dimensions of features until the output transforms into a one-dimensional tensor for each time step. This architectural enhancement has been verified to yield a significant improvement in word-level visual speech recognition performance.