# Speech Inpainting Based on Multi-Layer Long Short-Term Memory Networks
## 2 Related Works
### Deep Learning-Based Audio Inpainting Methods  
In recent years, there has been an increase in the audio inpainting literature based on deep learning methods. Several key considerations need to be clarified in the context of the deep learning-based audio inpainting task, which can be summarised by three characteristics: the type of processed audio signal, the duration of the missing audio gap, and the deep learning method used.  
Several studies have explored the efficacy of DNNs in addressing audio inpainting. [PERSON] et al. utilised a DNN architecture to restore the missing part within an instrument's audio signal using time-frequency coefficients (TF coefficients) [22], where the length of the inpainting gap was up to 128 ms. Similarly, [PERSON] et al. proposed an end-to-end DNN with a U-Net architecture to restore missing or distorted speech signals of up to 400 ms [23]. These works have shown that the DNN architecture can perform well in audio inpainting.  
Generative Adversarial Networks (GANs) [24] have recently become a focal point in audio inpainting. [PERSON] et al. proposed to use the Wasserstein GAN (WGAN) to restore the lost part within an instrument's audio signals based on short contexts (1 s from each side of the gap) and long contexts (2.5 s from each side of the gap), where the length of the inpainting gap was up to 550 ms [25]. Furthermore, [26] demonstrated that by using a conditional GAN (cGAN) model, missing audio segments could be restored based on the context of the surrounding audio and the latent variable of the cGAN. The authors aimed at restoring an instrument's audio signals, where the lengths of the inpainting gap were up to 1500 ms.  
It is worth noting that most of the approaches mainly focused on restoring music/instrument signals, which generally have long-term dependency and periodicity. In contrast, speech signals have more features and higher complexity when faced with several practical applications. Recent studies have noted the differences between music/instrumentsignal datasets and speech signal datasets [23; 27; 31]. Increasingly, researchers have started focusing on speech signals and using speech datasets to address audio inpainting.  
As proposed in [27], the architecture of Perceiver IO [28] was used to restore the missing parts of speech signals according to the text transcript. The authors used a transformer-based model [29] to restore gaps of up to 1000 ms. Another study [30] proposed the use of a multi-modal transformer together with high-level visual features to restore an instrument's audio signal, with gap lengths of up to 1600 ms. Other works, like [31], used visual information (facial motions) together with the audio context to restore gaps of up to 1600 ms within speech signals.  
However, when comes to real-life two-way conversational communication scenarios, there are notable limitations shared by these approaches--they are all based on specific conditions. These conditions include the type of processed audio signal [22], relatively shorter gaps [23; 25], the context before and after the gap [24], a text transcript [27; 28], and visual information [30; 31]. This limits the usage of the approaches in the literature to only their specific domains, as these conditions are not always available or feasible to use. This limitation underscores the necessity for further research in developing speech inpainting models that better align with practical communication scenarios.