# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Pulse Features
#### 3.4.1 Spatiotemporal Feature Enhancement Based on Event Cameras  
Event cameras distinguish themselves from conventional frame-based cameras by their ability to continuously capture data, generating asynchronous binary pulses that encode critical information regarding time, position, and intensity. This unique capability results in event images that provide superior recognition accuracy compared to their frame-based counterparts, particularly well-suited for the extraction and enhancement of lip spatiotemporal features. From the principles of spatiotemporal feature enhancement methods based on event cameras illustrated in Figure 11, it can be revealed that these methods resemble spatial feature-based enhancement since it is fundamentally characterized by the use of an event stream as its input, thereby enabling a more dynamic and responsive analysis of lip movements in real-time scenarios.  
Figure 11: The typical spatiotemporal feature enhancement methods based on event cameras. (**a**) The architecture of Ref. [112]. (**b**) The architecture of Ref. [113]. (**c**) The architecture of Ref. [114]. (**d**) The architecture of Ref. [115].  
In 2022, [PERSON] et al. [112] introduced the multi-scale temporal-spatial feature perception network (MSTP), which employs a multi-branch architecture consisting of low-speed and high-speed event processing streams. This design allows the model to concurrently capture comprehensive spatial features through the low-speed branch and intricate temporal characteristics via the high-speed branch. Additionally, the authors developed a multi-flow module (MFM) that utilizes attention-weighted maps to effectively incorporate the multi-scale temporal-spatial features from both event streams, thereby elevating perceptual capability to temporal-spatial features.  
To effectively address the challenge of recognizing isolated phonemes, in 2023, a novel lip-reading technique was introduced by combining frame and event cameras [113]. By employing ResNet-18 for the enhancement of both types of visual data, their experiments revealed that these multi-modalities strategies yielded superior recognition accuracy over the utilization of a single modality alone.  
In 2024, researchers proposed a multi-granularity spatiotemporal feature learning framework designed to extract intricate features from events characterized by microsecond-level temporal resolution [114]. This comprehensive framework includes two perceptual branches. One focuses on low-frame-rate features with complete spatial data but coarse temporal information. The other is on high-frame-rate features with fine temporal resolution at the cost of spatial detail. A temporal aggregation sub-network is incorporated to consolidate features from these branches, thereby fully capturing temporal correlations within event streams.  
Branches operating at different frame rates can capture spatiotemporal features with varying degrees of granularity. However, the aggregation of events into event frames often results in the loss of intricate temporal details. In response to this challenge, [PERSON] et al. [115] proposed a novel event representation method that establishes connections between key local voxels and a graph list, facilitating the alignment and fusion of global spatial features derived from event frames with local spatiotemporal characteristics represented in the voxel graph. Additionally, they developed a time aggregation module incorporating positional encoding to effectively capture both local absolute spatial information and global temporal dynamics. Experimental results revealed that this innovative approach surpassed the performance of both event-based and video-based lip-reading techniques, further building on the previously introduced multi-granularity framework and emphasizing the crucial role of fine-grained temporal information in enhancing recognition accuracy.