## 4 Results and Discussion
### Query-by-Example Spoken Term Detection  
#### 4.2.1 Development Data  
Evaluation system results for the development data are presented in Tables 13 and 14 for MAVIR and RTVE data, respectively. They show similar findings: (1) The \(E2E+LA\) system obtains the best QoE STD performance for both datasets, since this system is adapted to the target language. This improvement is statistically significant for a paired  
Figure 6: DET curves of the STD systems for RTVE test data.  
Figure 7: DET curves of the STD systems for SPARL20 test data.  
\(t\)-test (\(p<0.001\)) for RTVE data and weak significant (\(p<0.03\)) for MAVIR data compared to the other QBe STD systems. Moreover, this improvement is larger for MAVIR data (see MTVV results), since the language adaptation relies on these data; (2) The \(z\)-norm normalization decreases system performance. We consider that this could be due to the use of a global mean score and therefore global standard deviation score for all occurrences, regardless of the search query.  
Comparing the QBe STD systems to a text-based STD system (i.e., the _Kaldi SGMM_ system in the STD task) evaluated on the QBe STD queries, the text-based STD system obtains better performance for RTVE data, due to the use of the target language for system construction. This improvement is statistically significant for a paired \(t\)-test (\(p<0.001\)) compared to the QBe STD systems.  
Considering the results obtained by the end-to-end systems, it can be said that in general the performance is low. Preliminary experiments with the same systems were carried out on the read speech LibriSpeech dataset [178], following a similar experimental setup as in [137]. In this setup, the test language matched the training language and the query and each of the test word alignments were known in advance, so that words were only compared one to one (not one query against a segment of audio containing a few words, noise or fragments of words).  
The results on this dataset with this setup showed a much better performance for P@20 (precision at 20) metric, similar to that presented in [137]. However, when facing the more challenging data conditions of this evaluation datasets, for which the speech scenarios are more complex, the test language differs from the training language and the query and test word alignments are both unknown, the performance drops dramatically.  
The DET curves for MAVIR and RTVE data are shown in Figures 8 and 9, respectively. For MAVIR data, they show that the \(E2E+LA\) and the \(E2E\) systems perform the best for a low FA ratio, and the \(E2E+ZNORM\) system performs the best for a low miss ratio. For RTVE data, the \(E2E+LA\) system performs the best for a low FA ratio, whereas the \(E2E+ZNORM\) system performs the best for a low miss ratio.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System ID** & **MTWV** & **ATWV** & **p(FA)** & **p(Miss)** \\ \hline E2E + LA & 0.0533 & 0.0491 & 0.00000 & 0.947 \\ \hline E2E & 0.0160 & \(-\)38.5775 & 0.00000 & 0.984 \\ \hline E2E + ZNORM & 0.0000 & \(-\)158.2873 & 0.00000 & 1.000 \\ \hline Text STD & 0.0000 & \(-\)0.0508 & 0.00000 & 1.000 \\ \hline \hline \end{tabular}
\end{table}
Table 13: System results of the QBe STD task for MAVIR development data.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**System ID** & **MTWV** & **ATWV** & **p(FA)** & **p(Miss)** \\ \hline E2E + LA & 0.0465 & 0.0465 & 0.00000 & 0.954 \\ \hline E2E & 0.0414 & \(-\)76.0473 & 0.00000 & 0.954 \\ \hline E2E + ZNORM & 0.0000 & \(-\)51.9993 & 0.00000 & 1.000 \\ \hline Text STD & 0.3101 & 0.3086 & 0.00001 & 0.677 \\ \hline \hline \end{tabular}
\end{table}
Table 14: System results of the QBe STD task for RTVE development data.