## 4 Results and Discussion
### Query-by-Example Spoken Term Detection
#### 4.2.2 Test Data  
Evaluation system results for the test data are presented in Tables 15-17 for MAVIR, RTVE and SPARL20 data, respectively. The performance for the QoE STD systems show, in general, the same findings as the development data. The performance obtained by the  
Figure 8: DET curves of the QoE STD systems and the text-based STD system for MAVIR development data.  
Figure 9: DET curves of the QoE STD systems and the text-based STD system for RTVE development data.  
end-to-end systems is low, opposite to the experiments carried out on the LibriSpeech dataset. For MAVIR and SPARL20 data, the best performance is obtained by the system that employs target language adaptation (i.e., the \(E2E+LA\) system).  
This best performance is statistically significant for a paired \(t\)-test (\(p<0.001\)) compared to the other QBE STD systems. For RTVE data, the best performance is for the \(E2E+ZNORM\) system, which does not employ language adaptation. This best performance is statistically significant for a paired \(t\)-test (\(p<0.001\)). However, the worse MTWV performance obtained in comparison with the \(E2E+LA\) system for these RTVE data suggests some threshold calibration issues. This is confirmed by the results obtained by most of the systems, whose ATWV performance is below 0, even though the MTWV performance is above 0.  
The text-based STD system (i.e., the _Kaldi SGMM_ system in the STD task) evaluated on the QBE STD queries significantly outperforms the QBE STD systems (\(p<0.001\)) for all the datasets. This suggests that the end-to-end systems for QBE STD need more research to approximate their performance to that of text-based STD. In particular, more research seems to be needed to adapt the system to the more realistic conditions of the evaluation, where the query and test word alignments are unknown.  
The DET curves are shown in Figures 10-12 for MAVIR, RTVE and SPARL20 data, respectively. For MAVIR data, they show that the best performance for a low FA ratio is for the \(E2E+LA\) system, whereas the text-based STD system performs the best for a low miss ratio. For RTVE data, the best performance for a low FA ratio is for the \(E2E\), and the \(E2E+LA\) systems, and the text-based STD system performs the best for a low miss ratio. For SPARL20 data, the \(E2E+LA\) system performs the best for a low FA ratio, and the text-based STD system performs the best for a low miss ratio.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System ID** & **MTWV** & **ATWV** & **p(FA)** & **p(Miss)** \\ \hline E2E + LA & 0.0209 & \(-115.7086\) & 0.00000 & 0.978 \\ \hline E2E & 0.0209 & \(-88.3716\) & 0.00000 & 0.978 \\ \hline E2E + ZNORM & 0.0000 & \(-16.5831\) & 0.00000 & 1.000 \\ \hline Text STD & 0.3427 & 0.3413 & 0.00002 & 0.639 \\ \hline \hline \end{tabular}
\end{table}
Table 16: System results of the QBE STD task for RTVE test data.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System ID** & **MTWV** & **ATWV** & **p(FA)** & **p(Miss)** \\ \hline E2E + LA & 0.0107 & 0.0107 & 0.00000 & 0.989 \\ \hline E2E & 0.0306 & \(-34.2099\) & 0.00000 & 0.961 \\ \hline E2E + ZNORM & 0.0000 & \(-103.6805\) & 0.00000 & 1.000 \\ \hline Text STD & 0.3662 & 0.3583 & 0.00005 & 0.588 \\ \hline \hline \end{tabular}
\end{table}
Table 17: System results of the QBE STD task for SPARL20 test data.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System ID** & **MTWV** & **ATWV** & **p(FA)** & **p(Miss)** \\ \hline E2E + LA & 0.0126 & \(-0.1061\) & 0.00000 & 0.987 \\ \hline E2E & 0.0000 & \(-393.5610\) & 0.00000 & 1.000 \\ \hline E2E + ZNORM & 0.0000 & \(-38.5959\) & 0.00000 & 1.000 \\ \hline Text STD & 0.4734 & 0.4682 & 0.00006 & 0.466 \\ \hline \hline \end{tabular}
\end{table}
Table 15: System results of the QBE STD task for MAVIR test data.  
Figure 11: DET curves of the QBe STD systems and the text-based STD system for RTVE test data.  
Figure 10: DET curves of the QBe STD systems and the text-based STD system for MAVIR test data.