# Speech Inpainting Based on Multi-Layer Long Short-Term Memory Networks
## 1 Introduction  
In various audio processing tasks, audio signals often experience unexpected damage and information loss, which causes \"gaps\" within the audio signal. The process of restoring the gaps in the audio signal is called audio inpainting [1], which is also known as audio interpolation [2; 3; 4; 5; 6], extrapolation [7; 8], waveform substitution [9], and imputation [10; 11]. The term \"inpainting\" is adopted from image inpainting in the computer vision and image processing fields, which is a technique related to restoring the missing or corrupted parts of an image [12]. Audio inpainting is a general concept that includes restoring any audio signal, such as speech, music, instrument, environmental sounds, or any other auditory signals. It can be referred to as speech inpainting when applied to speech signals. As such, it is a specific form of audio inpainting. In real-life applications, clean speech signals are often affected by many factors, such as degradation of audio quality, loss of information, and unpredictable noise [13], which leads to the incomplete transmission of speech signals, making it difficult for users at both the sending and receiving ends to communicate with each other. Speech inpainting aims to restore coherent and meaningful samples to the missing parts of a speech signal, ensuringthat users at the receiving end do not perceive any information loss. In this paper, we propose an innovative speech inpainting mechanism to restore speech signal losses of up to one second using multi-layer Long Short-Term Memory (LSTM) networks [14]. Currently, there are two kinds of methodology used in the context of the audio inpainting task: mathematical-based approaches and deep learning-based algorithms. Previous research on audio inpainting mainly focused on using mathematical signal-processing techniques [1; 2; 3; 4; 5; 6; 7; 8; 10; 11; 15; 16; 17; 18; 19; 20; 21]. In recent years, with the gradual maturity of deep learning technology, researchers have begun to use deep learning-based audio inpainting algorithms to restore lost information in audio signals [22; 23; 24; 25; 26; 27; 28; 29; 30; 31]. However, it should be noted that existing speech inpainting techniques primarily rely on \"specific conditions\" to restore the missing information. For example, SpeechPainter [27] is designed exclusively for text-based data, which means that during the inpainting process, all the speech transcripts are used to achieve outstanding predictions. Other techniques utilise visual information-based approaches, like facial motions together with the context information, to address the restoration task, as proposed in [31]. It is worth mentioning that during the signal transmission process in communication systems, such \"specific conditions\" do not exist except for the information before and after the missing area, which means that the loss of information is permanent and irreversible. Therefore, the methods mentioned above cannot be universally applied to all types of inpainting tasks due to their inherent limitations, which means that a new approach is needed. In this paper, we propose a method for addressing speech inpainting using multi-layer LSTM networks. A series of LSTM networks containing between two and six layers were designed and trained on four single-speaker datasets, followed by further training on four multi-speaker datasets using an exclusive five-layer LSTM network. Our research is based on two types of datasets--single-speaker and multi-speaker datasets--to investigate the inpainting performance of the proposed LSTM models in terms of the perceived audio quality with different LSTM layers, datasets, and gap lengths. The performance is evaluated through the Mean Opinion Score (MOS) [32] and frequency analysis method [33] to assess the inpainted speech quality. Our proposed models are capable of learning information prior to the gap and inpainting coherently predicted speech samples to fill in missing gaps in the range of 20 ms-1000 ms.