# End-to-End Mispronunciation Detection and Diagnosis Using Transfer Learning
## 4 Experiments and Results
### Experimental Results  
#### 4.4.1 Comparison with Different Amounts and Types of Training Data  
Table 2 lists the results of our implemented wav2 vec2.0 based MDD models. Comparing the first three models shows that the model LV60 outperforms the model BASE. We speculate these improvements of mispronunciation detection benefit from the robust acoustic representation learned from large amounts of unlabeled data. According to the language transfer theory, second language learners tend to transfer the L1 phonetic phenomenon to second language (L2) learning [51], and some researchers argue that using cross-language training corpus can boost MDD performance [52]. This is supported by the results of the current study. As can be seen from Table 2, the multilingual pretrained model XLSR performs better than the monolingual pretrained model LV60 (59.37% vs. 58.75%).  
To examine the MDD performance on ultra-low resource conditions, we compared the four MDD models using the same pretrained model XLSR but different training data configurations. As we can see from the last four rows of Table 2, reducing 33% of training data slightly degrades the MDD performance compared with the _default_ training data configuration. When another 33% of the training data are reduced (i.e., the \(-66\%\) data configuration), the results deteriorate more. However, considering the relatively small size of the training data (here, only one speaker's data for each language is kept), achieving a F1-score of 55.52% is still acceptable. In Figure 4, the upper and lower subplots depict the confusion matrices for the MDD models trained with the _default_ and \(-66\%\) data configurations. The model trained with the \(-66\%\) training data can still retain the ability to detect mispronunciations while identifying correct pronunciations. Despite the difference of overall performances, the MDD models trained with different amounts of training data share similar patterns. This suggests that the acoustic representations extracted from the pretrained wav2 vec2.0 model have robustness to unseen data for the MDD task even when the annotated data are low-resource.  
Finally, we explore the use English native data (i.e., the L1 data) to facilitate the MDD performance. By introducing the additional training data the TIMIT corpus, the performance of the model trained with the +_TIMIT_ data configuration further improves by more than 1% in terms of F1-score (59.37% vs. 60.44%). On the other hand, compared with the \(-33\%\) data configuration, the extra 33% data of the _default_ data configuration does not increase the performance too much (59.27% vs. 59.37%). This suggests that the acoustic information conveyed by the correct pronunciations of the L1 data may break the bottleneck of only using L2 data.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multirow{2}{*}{**Data**} & \multicolumn{3}{c}{**Correct Pronunciations**} & \multicolumn{3}{c}{**Mispronunciations**} \\ \cline{3-8}  & & & & & **True Rejection** & **F1** \\ \cline{3-8}  & & **True Accept** & **False Rejection** & **False Accept** & **Correct Diag.** & **Diag. Error** & \\ \hline wav2 vec2.0-BASE & - & 94.12\% & 5.88\% & 49.53\% & 65.86\% & 34.14\% & 54.28\% \\ wav2 vec2-LV60 & - & 94.01\% & 5.99\% & 43.37\% & 68.08\% & 31.91\% & 58.75\% \\ wav2 vec2-XLSR & - & 94.57\% & 5.43\% & 43.95\% & 65.75\% & 34.25\% & **59.37\%** \\ wav2 vec2-XLSR & \(-33\%\) & 94.11\% & 5.89\% & 41.23\% & 69.13\% & 30.87\% & 59.27\% \\ wav2 vec2-XLSR & \(-66\%\) & 93.35\% & 6.65\% & 46.06\% & 64.67\% & 35.33\% & 55.52\% \\ wav2 vec2.0-XLSR & +TIMIT & 94.30\% & 5.70\% & 41.80\% & 70.72\% & 29.28\% & 60.44\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: MDD performances of different pretrained models and training data configurations.