# Non-Autoregressive End-to-End Neural Modeling for Automatic Pronunciation Error Detection
## 4 Experimental Setup
### Experimental Results  
We used the L2-ARCTIC and SpeechOcean762 datasets to train and test the proposed model. We additionally used the TIMIT dataset for training of mispronunciation detection \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset Name** & **Categories** & **Speakers** & **Utterances** & **Hours** \\ \hline TIMIT & Train & 630 & 6300 & 4.5 \\ \hline \multirow{3}{*}{L2-ARCTIC} & Train & 17 & 2549 & 2.66 \\  & Test & 6 & 900 & 0.88 \\  & Dev & 1 & 150 & 0.12 \\ \hline \multirow{3}{*}{SpeechOcean} & Train & 125 & 2500 & 2.28 \\  & Test & 62 & 1250 & 1.14 \\ \cline{1-1}  & Dev & 62 & 1250 & 1.14 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Dataset summary. and diagnosis. Two feature vectors, FBANK and wav2 vec 2.0, extracted fractures from different speeches in the L2-ARCTIC and SpeechOcean datasets. Figure 7 shows the training and validation accuracy of the three most-used metrics (adjusted Rand index (ARI), normalized mutual information (NMI), and accuracy) on the L2-ARCTIC and SpeechOcean datasets using the FBANK feature extraction model. For the L2-Arctic dataset, the proposed model learns smoothly from epoch 0 to 100, as shown in Figure 7a, but the training and validation curves in Figure 7b for the SpeechOcean dataset are not smooth due to misrecognition of both vowels and consonants in the training phase. However, the accuracy with the SpeechOcean dataset is higher than with the L2-Arctic dataset because SpeechOcean is a balanced dataset, whereas L2-Arctic is an unbalanced dataset. The adjusted Rand index (ARI) metric calculates the similarity between the generated utterance sequence and the annotation sequence. The ARI score is better for the L2-Arctic dataset with both FBANK and wav2 vec feature extraction models, as shown in Figures 7a, 8a, and 9a. The normalized mutual information (NMI) metric measures the dependence between different tokens in a voice signal. A better NMI score predicts that the order of utterances is strongly connected to different tokens. Figures 7-9 show NMI curves of different datasets for both training and validation steps with 100 epochs where the highest NMI achieved for both training and validation on SpeechOcean dataset with wav2 vec 2.0 (large) feature extraction model. Figure 8: Visualization of the effect of training and validation scores along with the number of epochs on different datasets using the wav2 vec 2.0 (base) feature extraction model. (**a**) L2-Arctic dataset. (**b**) SpeechOcean dataset. Figure 7: Visualization of the effect of training and validation scores along with the number of epochs on different datasets using the FBANK feature extraction model. (**a**) L2-Arctic dataset. (**b**) SpeechOcean dataset. We select an utterance from the L2-ARCTIC [27] dataset to visualize the different heads of the last attention block in the encoder, PDS, and decoder blocks. Each attention layer consists of eight attention heads, and we have shown the first four heads in the last attention block in order to understand the model. Figure 10 shows the output of different heads at the last attention level of the encoder block, where different heads give different attention results to the same utterance. Figure 10a,c show the visualization of the speech sequence with respect to different times, whereas Figure 10b,d show the relationship between different speech sequences as acoustic frames. From these figures, we observe that most attention heads learn different patterns to calculate the final attention score. The PDS block summarizes encoder outputs and reshuffles the encoder representation according to the token positions. The PDS module uses the token sequence as the query, and the output of the encoder representation is used as the key and value. Figure 11 shows the relationship between the different tokens and different time frames. Figure 10: Graphical view of the various attention head outputs at the last attention level of the encoder block. (**a**) Output of the first head. (**b**) Output of the second head. (**c**) Output of the third head.