# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 3 Spatiotemporal Feature Enhancement Methods
### Spatiotemporal Feature Enhancement Based on Pulse Features
#### 3.4.2 Spatiotemporal Feature Enhancement Based on Spiking Neural Networks  
Spiking neural networks (SNNs) are distinguished by their event-driven processing paradigm and emulation of biological neural architectures, whereby information is conveyed through discrete spikes, mirroring the functionality of biological neurons [55]. This inherent feature renders SNNs particularly adept at processing spatiotemporal data, which is indispensable in the domain of lip-reading. Such data allow for the extraction of critical phonetic characteristics, including articulation positions and rhythmic patterns of speech. The capacity of SNNs to discern phonetically analogous lexical items--differentiated by variations in lip morphology, enunciation processes, and temporal dynamics--underscores the significance of spatiotemporal information in auditory-visual integration tasks, highlighting their utility in lip-reading applications. As illustrated in Figure 12, the enhancement derived from the spiking neural network is intricately tied to its reliance on event-based input processing. Initially, the SNN engages with raw event data, meticulously extracting and amplifying the underlying spatiotemporal features. Once this enhancement is achieved, the temporal dynamics are subsequently modeled through advanced architectures, including Gated Recurrent Unit (GRU) and Bidirectional Gated Recurrent Unit (Bi-GRU), which facilitate a nuanced understanding of the data's temporal progression. The integration of two asynchronous modalities, dynamic vision sensor (DVS) and dynamic audio sensor (DAS), within a spiking neural network provides a harmonious compensation of spatiotemporal feature information [116]. The lip-reading framework could perform training effectively in each modality independently, as well as in a joint training configuration. Empirical results demonstrate that the model with a joint training strategy achieves a 23% higher recognition accuracy than the DVS model. In 2023, a novel solution emerged to tackle the challenge of erroneous spikes generated by identical timestamps in event-driven algorithms [117]. This efficient event-driven learning approach not only addresses this issue but also reinforces the temporal feature processing capabilities inherent in event-driven neural networks. Importantly, this mechanism demonstrates compatibility with a range of spike-based plasticity mechanisms, enhancing its versatility and applicability in diverse neural network architectures. In the same year, event-based sensors were utilized to construct the multi-grained spatiotemporal feature perceived (MSTP) network, designed to capture the events generated by lip movements [118]. This innovative model employs an SNN architecture to classify short event sequences into distinct word categories. Leveraging neuromorphic computing, this approach unfolds the benefits of high efficiency and low latency, making the model particularly well suited for real-time embedded applications. Though SNNs demonstrate a commendable ability to extract spatiotemporal features, particularly temporal characteristics related to lip movements, they still face limitations in distinguishing between visually similar phonetic elements. In 2023, [PERSON] et al. introduced the spatial-temporal attention block (STAB), an event-based lip-reading model that enhances the differentiation of visually similar words by employing spatial and temporal attention branches that dynamically emphasize the most relevant features reflecting lip movements [13]. Through a fusion mechanism, the model establishes a comprehensive and focused representation of lip activity that effectively differentiates between visually similar words, surpassing existing SNN architectures in terms of both recognition accuracy and energy efficiency. The sparse activation and event-driven nature of SNNs render them exceptionally suited for low-power, low-latency edge applications within end-to-end neuromorphic hardware pipelines.