# Spatiotemporal Feature Enhancement for Lip-Reading: A Survey
## 4 Difficulties and Challenges
### Feature Alignment  
Spatiotemporal feature alignment: A foundational challenge in achieving effective lip-reading lies in the alignment of spatiotemporal features. The interdependence and integration of temporal and spatial information are essential for accurate feature representation. That means that disruptions in this relationship readily lead to significant misalignments, resulting in inadequate representation of the lip movements and thus diminishing the model's efficacy. Therefore, aligning spatiotemporal features during both extraction and enhancement is crucial for the success of lip-reading.  
Phoneme-to-viseme mappings: In the realm of audiovisual lip-reading, since phonemes and visemes are often processed separately, a substantial lack of coherence arises between the features associated with these two modes, which significantly and negatively impacts recognition performance. Consequently, the precise alignment of phonetic elements and visual elements is also of paramount importance for lip-reading as well as aligning spatiotemporal features, especially for recognition of subtle lip movements.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline
**Methods** & **Typical Networks** & **Elements** & \begin{tabular}{c} **Enhancement Ability** \\ **Spatial** \\ **Feature** \\ \end{tabular} & \begin{tabular}{c} **Recognition** \\ **Rate** \\ \end{tabular} & \begin{tabular}{c} **Structure** \\ **Complexity** \\ \end{tabular} & **Parameter** \\ \hline spatial features [63; 64; 65; 66; 67; 133] & ResNet, DenseNet, ShuffleNet & Convolution block, residual block & Weak & Weak & Low & Low & Small \\ \hline temporal [88; 89; 90; 91; 92; 93; 94; 95; 96; 97] & \begin{tabular}{c} GCNet, TSM, \\ GSF, TBC \\ \end{tabular} & \begin{tabular}{c} Temporal \\ convolution block \\ \end{tabular} & Weak & Strong & High & High & Small \\ \hline attention [98; 99; 100; 101] & \begin{tabular}{c} CBAM, BAM, \\ Transformer \\ \end{tabular} & Attention block & Strong & Strong & High & High & Large \\ \hline event cameras [112; 113; 114; 115] & Event streams & \begin{tabular}{c} High-speed event \\ stream, \\ Low-speed \\ event stream \\ \end{tabular} & Strong & Relatively & High & High & Large \\ \hline spiking neural  network [116; 117; 118; 119; 120; 121; 122; 123; 124; 125; 126; 127; 128; 129; 130; 131; 132] & Spiking Neural  Networks (SNNs) & Event streams, Pulse signal & Relatively & Strong & High & Low & Fewer \\ \hline audio-visual  assisting [82; 120; 121; 122; 123; 124; 125] & Audio-visual  complementation & \begin{tabular}{c} Visual processing \\ unit, audio \\ processing unit \\ \end{tabular} & Strong & Strong & High & High & Huge \\ \hline others [126; 127; 128; 129; 130; 131; 132] & graph, moment, filter, \
\begin{tabular}{c} Graph convolution, \\ movement feature, \\ filter, \\ variational encoder \\ \end{tabular} &
\begin{tabular}{c} Relatively \\ Strong \\ \end{tabular} & High & - & Large \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of spatiotemporal feature enhancement methods.  
Speaker alignment: This is particularly challenging given the considerable variability among speakers in facial attributes, such as skin tone, head and mouth morphology, inter-organ distances, and posture, as well as differences in speech characteristics like pronunciation tendencies and speech rate.