# Speech Inpainting Based on Multi-Layer Long Short-Term Memory Networks
## 3 Proposed Speech Inpainting Methods and Model Architecture
### Performance Evaluation  
We chose the MOS and frequency analysis method [33] to evaluate the inpainted speech quality. The MOS is commonly used in the telecommunications industry to measure the perceived quality of audio and video signals, as defined by the ITU-T Recommendation P.800 [32]. The MOS returns a subjective rating scale that ranges from 1 to 5 to represent the quality, with an MOS of 5 being perceived as excellent, 4 as good, 3 as fair, 2 as poor,  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multicolumn{1}{c}{**Category**} & \multicolumn{1}{c}{**Dataset**} & \multicolumn{1}{c}{
\begin{tabular}{c} **Component (M: Male; F;** \\ **Female)** \\ \end{tabular} } & \multicolumn{1}{c}{**Length (s)**} \\ \hline \multirow{4}{*}{Single-speaker} & LJSpeech [38] & 1 speaker (F) & 400 \\  & RyanSpeech [40] & 1 speaker (M) & 400 \\  & LibriSpeech [41] & 1 speaker (M) & 400 \\  & LibriSpeech [41] & 1 speaker (F) & 400 \\ \hline \multirow{4}{*}{Multi-speaker} & Hi-Fi [39] & 10 speakers (4 F, 6 M) & 400 for each speaker \\  & LibriSpeech [41] & 10 speakers (5 F, 5 M) & 400 for each speaker \\  & LibriSpeech [41] & 10 speakers (5 F, 5 M) & 400 for each speaker \\  & LibriSpeech [41] & 10 speakers (5 F, 5 M) & 400 for each speaker \\ \hline \hline \end{tabular}
\end{table}
Table 1: Datasets built in this research work.  
and 1 as bad. Two types of MOSs were employed in this work, corresponding to both narrow bandwidth (NB) and wide bandwidth (WB) speech signals. NB varies from 300 Hz to 3400 Hz, whereas WB varies from 50 Hz to 7000 Hz.  
The evaluation process utilised a total of 24 models, with 20 models trained on the single-speaker dataset and 4 models trained on the multi-speaker dataset. For each single-speaker dataset, five models were utilised, with LSTM layers ranging from two to six, whereas an exclusive five-layer LSTM model was utilised for each multi-speaker dataset.  
When evaluating each model, 10 speech signals not included in the training and testing datasets were selected, referred to as the original signal. The inpainting performance was assessed across seven different gap ranges for each test speech signal. The selected gap lengths were as follows: 20 ms, 40 ms, 50 ms, 100 ms, 200 ms, 500 ms, and 1000 ms. Each test speech signal was used to generate seven independent signals with different lengths of speech samples set to zero, referred to as zeroed signals. It should be noted that every zeroed signal generated from the same test speech signal had the same gap starting point, and the sample values of the entire gap were set to zero, meaning the signal was continuously missing. Nevertheless, the starting positions of gaps differed among the different test speech signals, which guaranteed that all gaps remained within the voiced part of the speech signal. Subsequently, the inpainted signals were generated. The proposed models were used to restore different lengths of speech samples, and then the inpainted samples were inserted into the corresponding gap locations of the zeroed signals, resulting in seven inpainted signals from each test speech signal.  
The final evaluation results were determined by computing the average MOS between the ten original and ten inpainted signals in seven different gap ranges. It should be noted that the method used to calculate the average MOS was aimed at mitigating the impact of random errors that may arise during a single MOS calculation process, ensuring a more robust and reliable assessment of the inpainting performance.