# Multimodal Unsupervised Speech Translation for Recognizing and Evaluating Second Language Speech
## 3 Proposed Fluency Scoring and Automatic Proficiency Evaluation Method
### Automatic Proficiency Evaluation with Data Augmentation  
The speech recognition system is optimized for non-native speakers as well as natives for educational purposes and smooth interaction. Speech features for computing fluency scores are extracted and decoded into time-aligned sequences by forced-alignment using the non-native acoustic model (AM). In addition, multiple AM scores are used to evaluate proficiency. In order to improve speech recognition accuracy and time-alignment performance, and to compute AM scores more accurately and meaningfully, we augment the training speech dataset and train non-native AM using the augmented training dataset. In this work, we convert some speech characteristics (style) to generate speech data for augmentation. In the proposed speech conversion model, we assume that each spectral feature of the speech signal is decomposed into a speaker-independent content factor desired to be maintained and each speaker-specific style factor we want to change in latent space. After extracting the content factor from the source speech signal, the proposed conversion model converts the source speech to the desired speech style by extracting the style factor of target speech and recombining it with the extracted content factor. By simply choosing the style factor for this recombination as a source style factor or target style factor, the conversion model can reconstruct or convert speech:  
\[\hat{x}_{s\to s}=D(E_{c}(x_{s}),E_{s}^{s}(x_{s})), \tag{1}\]  
\[\hat{x}_{s\to t}=D\big{(}E_{c}(x_{s}),E_{s}^{t}(x_{t})\big{)}, \tag{2}\]  
where \(\hat{x}_{s\to s}\) and \(\hat{x}_{s\to t}\) are the reconstructed and converted spectra, \(x_{s}\) and \(x_{t}\) are the source and target speech spectra, \(D\) is the decoder, and \(E_{c}\), \(E_{s}^{s}\), and \(E_{s}^{t}\) denote the content encoder, source style encoder, and target style encoder, respectively. The content encoder network \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Word** & **Start Time** & **End Time** & **Stress Value** \\ \hline I & **0.0** & 0.2 & 1 \\ am & **0.2** & 0.4 & 1 \\ still & **0.4** & 0.8 & 2 \\ very & 0.8 & 1.0 & 0 \\ sick & **1.0** & 1.3 & 1 \\ I & 1.4 & 1.5 & 0 \\ need & **1.5** & 1.8 & 1 \\ to & 1.8 & 2.0 & 0 \\ take & 2.0 & 2.2 & 0 \\ some & 2.2 & 2.5 & 0 \\ pills & 2.5 & **3.0** & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Sentence stress pattern in the sentence “I am still very sick. I need to take some pills.” \begin{table}
\begin{tabular}{c c c} \hline \hline
**Proficiency Score** & **Mean Time Interval** & **Standard Deviation of the Time Interval** \\ \hline
1 & 1.12 & 0.62 \\
2 & 0.82 & 0.42 \\
3 & 0.68 & 0.34 \\
4 & 058 & 0.32 \\
5 & 0.56 & 0.31 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Example of the mean rhythm scores. is shared across both speakers, and the style encoder networks are domain-specific networks for individual speakers. Figure 6 shows a block diagram of the proposed speech conversion method. As shown in Figure 6, the content encoder network extracts the content factor and is shared across all domains. All convolutional layers of the content encoder were followed by instance normalization (IN) to remove the speech style information and learn domain-independent content information (phoneme in speech). The style encoder network computes the domain-specific style factor for each domain and is composed of multiple separate style encoders (source style encoder and target style encoder in Figure 6) for individual domains. In the style encoders, IN was not used, because it removes the speech style information. We jointly train the encoders and decoder with multiple losses.