# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Spatiotemporal Convolution
#### 3.2.1 Three-Dimensional Spatiotemporal Convolution Enhancement  
Three-dimensional spatiotemporal convolutions proficiently extract features from video frames, skillfully capturing the spatiotemporal nuances embedded within the sequences [95; 96]. This remarkable ability to simultaneously model spatial and temporal features render them exceptionally well suited for revealing the distinct characteristics of lip images.  
In 2021, [PERSON] et al. [88] proposed the alternating spatiotemporal and spatial convolution (ALOS) module, which transforms sequences by preserving their length while adjusting the dimensions of sequence elements (e.g., height, width, and channels). The ALOS module consists of 3D convolutions for spatiotemporal feature extraction, 2D convolutions for spatial feature extraction, and components for converting sequences to feature maps. Nestled within the architecture of ResNet blocks, the ALOS framework introduces an innovative front end that significantly enhances the extraction of spatiotemporal features. While this integration bolsters recognition performance, the incorporation of multiple ALOS modules inevitably leads to an increase in both parameters and computational complexity.  
Figure 9: The typical spatiotemporal feature enhancement methods based on spatiotemporal convolution. (**a**) The architecture of Ref. [88]. (**b**) The architecture of Ref. [89]. (**c**) The architecture of Ref. [90]. (**d**) The architecture of Ref. [91]. (**e**) The architecture of Ref. [92]. (**f**) The architecture of Ref. [93]. (**g**) The architecture of Ref. [94].  
In the next year, another front-end spatiotemporal feature enhancement network emerged by stacking 3D densely connected CNN blocks with 3D Transition blocks [89]. By introducing a temporal dimension to the densely connected convolutional kernels and pooling layers, spatiotemporal features can be well preserved. Noteworthy is that diverse spatiotemporal features can be recognized through different CNN structures and depths, and such spatiotemporal features can be further strengthened by virtue of multi-layer feature fusion of 3D CNNs in the front-end network [97].  
To address the limited temporal feature extraction capabilities of standard 3D convolutions, [PERSON] et al. [90] proposed a Temporal Adaptive Module (TAM). TAM consists of a local branch that provides position-sensitive information and a global branch associated with long-term temporal dependencies. This integration contributes to capturing complex temporal structures and enhancing robust temporal modeling ability. Furthermore, to improve the temporal feature representation capabilities, TAM can be integrated into classic network structures.  
The improved feature representation capabilities of multi-granularity features were validated by a dual-branch 3D convolutional structure [11]. Specifically, by employing a 3D convolutional ResNet-52 to extract medium-granularity spatiotemporal features as a spatiotemporal enhancement branch, dynamic changes in video sequences can be effectively captured. To extract fine-granularity spatial features, a ResNet-34 was used to construct a spatial enhancement branch to capture subtle lip shape details. Additionally, unlike the direct concatenation of 2D and 3D convolutions, it incorporated a coarse-granularity spatiotemporal feature fusion module in the back end, and an attention mechanism was designed to integrate fine-grained and mid-level features into a unified representation, offering global latent patterns learning within the sequences. The comparative analysis reveals the heightened sensitivity to subtle variations in the shape of lip movement, excelling in effective distinguishing of visually similar phonemes, such as \"b\" and \"p\".