# Lip2 Speech: Lightweight Multi-Speaker Speech Reconstruction with Gabor Features
## 2 Literature Review
### Speech Reconstruction Using Constrained Datasets  
The first key speech reconstruction system was proposed by [PERSON] and [PERSON] [12], who applied a neural network to estimate a speech signal from a silent video of a speaker's frontal face using a deep-learning network trained on hand-engineered visual features obtained from the mouth region. However, this approach had the limitation of missing certain speech components such as fundamental frequency and aperiodicity. [PERSON] and [PERSON] [5] extended this by using visual features to predict a class label and exploring temporal information using RNNs. Although the intelligibility of the reconstructed speech improved substantially, speech quality was still low. [PERSON] and [PERSON] [1] used a regression-based framework to predict LSP coefficients directly from raw visual data with a CNN and two fully connected layers. They found that no hand-crafted visual features were needed to reconstruct the speaker's voice, using the whole face instead of just the mouth improved performance, and the regression-based method was effective in reconstructing out-of-vocabulary words. However, the signals sounded unnatural because Gaussian white noise was used as excitation to reconstruct the waveform from LPC features. Subsequent studies focused on improving speech quality and intelligibility. [PERSON] and [PERSON] [21] employed a postprocessing network for transforming a learned mel-scale spectrogram into a linear scale spectrogram. This spectrogram was then used with a Griffin-Lim algorithm [22] to regenerate the time-domain signal. Compared with their early study of Vid2 speech [1], their approach demonstrated significant average speech quality improvements. Specifically, during testing on a single GRID speaker, the average PESQ score improved by 38%, from 1.19 to 1.922, as measured by Perceptual Evaluation of Speech Quality (PESQ). [PERSON] et al. [2] focused on establishing a mapping between the speaker's facial cues and speech-related attributes, leveraging a pre-trained deep Autoencoder network. In comparison to their previous work, their methodology led to a 6% enhancement in average PESQ score and a notable 35% increase in speech intelligibility within the GRID Corpus, encompassing speakers S1, S2, S3, and S4. Similarly, [PERSON] et al. [23] and [PERSON] et al. [24] utilised Generative Adversarial Networks (GANs) for direct audio waveform synthesis from video frames. During unseen speaker testing, these approaches achieved word error rates of 40.5% with the GRID Corpus and 42.51 with the LRW dataset. Furthermore, [PERSON] et al. [17] integrated a GAN with an attention mechanism, while [PERSON] et al. [25] employed a stochastic modelling strategy utilising a variational autoencoder. Their methods yielded average PESQ values of 1.961 and 1.932, respectively, with the GRID Corpus. These studies indicate an improvement in PESQ scores over earlier methods, but there are limitations relating to constrained datasets and single speakers. A list of approaches reported in the literature is summarised in Table 1.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c c c c} \hline
**Task** & **Year** & **Paper** & **Inpet** & **Output** & **Model Info** & **Dataset** & **Metrics** & **UD** & **MS** & **MT** & **MV** \\ \hline  & 2015 & Comn and Milner [12] & 2D-DCT/AAM month & LPC or melt-filterbank amplitudes & CMM/FFNN & GRID corpus & WRE/PFSQ & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\  & 2015 & [PERSON] et al. [13] & 2D-DCT & STRAGHT Spectral & NMF & AV-JP & MOS/PFSQ/STO & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\  & 2017 & Comn and Milner [5] & AAM month & (med-filterbank & FPNN/RNN & GRID corpus & MSE/ ESTO/ PFSQ & \(\times\) & \(\times\) & \(\times\) & \(\times\) \\  & 2017 & [PERSON] et al.