# End-to-End Mispronunciation Detection and Diagnosis Using Transfer Learning
## 4 Experiments and Results
### Experimental Results
#### 4.4.3 Using Textual Modulation Gate and Contrastive Learning  
Table 4 lists the MDD results of using different textual modulation gates as well as the baseline models. It should be noted that, as mentioned in Section 4.2.2, we used the simplest pretrained model wav2 vec2.0-BASE as the audio encoder for most of the MDD systems, and we only used the most powerful wav2 vec2.0-XLSR for the best textual modulation gate, i.e., the models _TextGateXLSR_ and _TextGateXLSRContrast_. In the stage of examining the benefit of leveraging the prior text, we used three baseline models. The BaselineConventional and BaselineConcatenate models correspond to left and right parts of Figure 2, respectively. The BaselineAdd model only differs in the fusion style from the BaselineConcatenate model (\"add\" versus \"concatenate\"). Under fair architecture and training settings, our baselines based on the pretrained wav2 vec2.0 model surpass the BaselineConventional [20], suggesting the audio encoder wav2 vec2.0 in our baselines provides more powerful representations. Moreover, the \"concatenate\" operation for information fusion is better than the \"add\" operation. As expected, the proposed approaches _DoubleGate_ and _TextGate_ outperform the BaselineConcatenate/BaselineAdd method by +0.5%/1.8% and +1.4%/2.7%, respectively. When leveraging linguistic knowledge, most previous studies directly incorporated textual features into speech representation via a naive attention operation. They did not consider the fact that textual features contribute very differently when they are assigned to attend different acoustic features. The text-audio joint representation should be carefully handled especially when mispronunciation occurs. The gating mechanism can be used to modulate the information flowed from text modality to audio modality. The textual modulation gate proposed in this study successfully plays the role of validating information that comes from texts. Figure 5 shows the attention weights output by the _TextGate_ and _BaselineConcatenate_ model. Since the textual modulation gate can take responsibility for turning on/off textual information flow, attention patterns look neat and natural, whereas audio-text correlation maps for the model without a gate would be chaotic. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Models**} & \multicolumn{3}{c}{**Correct Pronunciations**} & \multicolumn{3}{c}{**Mispronunciations**} \\ \cline{2-6}  & **True Accept** & **False Rejection** & **False Accept** & **True Rejection** & **F1** \\ \hline BaselineConventional & 92.65\% & 7.35\% & 43.88\% & 74.96\% & 25.04\% & 56.08\% \\ BaselineConcatenate & 93.68\% & 6.31\% & 42.87\% & 68.52\% & 31.48\% & 58.87\% \\ BaselineAdd & 94.15\% & 5.85\% & 45.36\% & 63.21\% & 36.79\% & 57.51\% \\ \hline DoubleGate & 94.59\% & 5.41\% & 44.00\% & 68.68\% & 31.32\% & 59.34\% \\ TextGate & 94.50\% & 5.50\% & 42.22\% & 68.26\% & 31.74\% & 60.27\% \\ TextGate\({}^{\prime}\) & 94.29\% & 5.71\% & 41.89\% & 69.86\% & 30.14\% & 60.34\% \\ TextGate & 94.53\% & 5.47\% & 46.33\% & 64.22\% & 35.78\% & 57.48\% \\ TextGate\({}^{\prime}\) & 95.07\% & 4.93\% & 47.66\% & 63.62\% & 36.38\% & 57.47\% \\ \hline TextGateXLSR & 94.94\% & 5.06\% & 43.72\% & 68.79\% & 31.21\% & 60.23\% \\ \hline TextGateContrast & 93.72\% & 6.28\% & 40.43\% & 69.77\% & 30.23\% & 60.32\% \\ TextGateXLSRContrast & 93.81\% & 6.19\% & 38.62\% & 71.08\% & 28.92\% & 61.75\% \\ \hline \hline \end{tabular}  
* * Reversed version of TextGate (maybe we can call it AudioGate); \(g\) is used to multiple \(h_{t}^{\text{O}}\) instead of \(c_{t}\).  
\end{table} 
Table 4: Comprehensive performance comparison between different textual modulation gates.