# Multimodal Unsupervised Speech Translation for Recognizing and Evaluating Second Language Speech
## 4 Experimental Results
### Proficiency Evaluation Test
#### 4.2.3 Data Augmentation  
The proposed VAE-based speech conversion model consisted of a content encoder, style encoders, and a joint decoder. The content encoder comprised two dilated convolutional layers and a gated recurrent unit (GRU) based on a recurrent neural network. In order to remove the speech style information, all convolutional layers were followed by instance normalization (IN) [50]. The style encoder comprised a global average pooling layer, \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Rater** & **2** & **3** & **4** \\ \hline
1 & 0.79 & 0.75 & 0.80 \\
2 & - & 0.81 & 0.83 \\
3 & - & - & 0.80 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Inter-rater correlation. \begin{table}
\begin{tabular}{c c} \hline \hline
**No.** & **Sentence** \\ \hline
1 & My pet bird sleeps in the cage. \\
2 & 1 eat fruits when 1 am hungry. \\
3 & [PERSON] drank a cup of coffee. \\ \(\cdots\) & \\
100 & They suspect that the suspect killed [PERSON]. \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Samples of the scripts. 3-layer multi-layer perceptron (MLP), and a fully connected layer. In the style encoder, IN was not used because it removes the original feature mean and variance that represent speech style information. Then, content and style factors were fed into the decoder to reconstruct or convert the speech. The decoder comprised two dilated convolutional layers and the recurrent neural network-based GRU. All convolutional layers were used with an Adaptive Instance Normalization layer generated by the MLP from the style factor [50]. \[AdaIN(z,s)=\sigma(s)\bigg{(}\frac{z-\mu(z)}{\sigma(z)}\bigg{)}+\mu(s), \tag{7}\]  
where \(z\) is the activation of the previous convolutional layer, and \(\mu(. )\) and \(\sigma(. )\) denote the mean and variance, respectively. Figure 8 shows an example of Mel-spectrograms obtained by the proposed method. Comparing the decoding results, we confirmed that the proposed method reconstructs and converts the spectral features efficiently. We performed the perception test to compare the sound quality and speaker similarity of converted speech between the proposed VAE-based speech conversion method and the conventional conditional VAE-based speech conversion (CVAE-SC) method [29], which is one of the most common speech conversion methods. We conducted an AB test and an ABX test. \"A\" and \"B\" were outputs from the proposed method and the CVAE, and \"X\" was a real speech sample. To eliminate bias in the order, \"A\" and \"B\" were presented in random orders. In the AB test, each listener was presented with \"A\" and \"B\" audios at a time, and was asked to select \"A\", \"B\", or \"fair\" by considering both speech naturalness and intelligibility. In the ABX test, each listener was presented with two audios and a reference audio \"X\", and then, was asked to select a preferred audio or \"fair\" by considering the one closer to the reference. We used 24 utterance pairs for the AB test and another 24 utterance pairs, not included in the AB test, for the ABX test. The number of listeners was 20. Figure 9 shows the results, and we confirmed that the proposed method outperforms the baseline in both sound quality and speaker similarity terms. Figure 8: Waveform and Mel-spectrograms. (**a**) Waveform of the input signal, (**b**) Mel-spectrogram of the input signal, (**c**) reconstructed Mel-spectrogram, and (**d**) converted Mel-spectrogram. We also performed the speech recognition test to validate that the spectral features were converted meaningfully using the English read speech dataset. We used the ESP-net [51] for an end-to-end ASR system. We trained the AM using only the training dataset (\"Train database only\" in Table 5) and evaluated the test dataset, and we compared the recognition results to those obtained by evaluating the same test dataset using the AM trained with the augmented dataset (\"Augmentation\" in Table 5). Table 5 shows the word error rate (WER) results. For comparison, SpecAugment [21], speed perturbation method [20], and CVAE-SC were used as a reference.