## 2 Search on Speech Evaluation
### Evaluation Metrics  
In search on speech systems (both for STD and QBE STD tasks), a hypothesised occurrence is called a _detection_; if the detection corresponds to an actual occurrence, it is called a _hit_, otherwise it is called a _false alarm_. If an actual occurrence is not detected, this is called a _miss_. The actual term weighted value (ATWV) metric proposed by NIST [138] was used as the main metric for the evaluation. This metric combines the hit rate and false alarm rate of each query and averages over all the queries, as shown in Equation (1):  
\[\text{ATWV}=\frac{1}{|\Delta|}\sum_{Q\in\Delta}{(\frac{N_{hit}^{Q}}{N_{true}^{ Q}}-\beta\frac{N_{FA}^{Q}}{T-N_{true}^{Q}})}, \tag{1}\]  
where \(\Delta\) denotes the set of queries and \(|\Delta|\) is the number of queries in this set. \(N_{hit}^{Q}\) and \(N_{FA}^{Q}\) represent the numbers of hits and false alarms of query \(Q\), respectively, and \(N_{true}^{Q}\) is the number of actual occurrences of query \(Q\) in the audio. \(T\) denotes the audio length in seconds, and \(\beta\) is a weight factor set to 999.9, as in the ATWV proposed by NIST [37]. This weight factor causes an emphasis placed on recall compared to precision with a ratio 10:1.  
The time tolerance for query detection was higher than the original proposed by NIST to encourage participants to build end-to-end systems for both STD and QBE STD tasks. To do so, a detection was labelled as correct in case it appeared within \(\pm\)15-s interval from the ground-truth timestamp.  
ATWV represents the term weighted value (TWV) for an _optimal_ threshold given by the system (usually tuned on the development data). An additional metric, called maximum term weighted value (MTWV) [138] is also used in this paper to evaluate the upper-bound system performance regardless of the decision threshold.  
Additionally, p(Miss) and p(FA) values, which represent the probability of miss and FA of the system as defined in Equations (2) and (3), respectively, are also reported.  
\[p(Miss)=1-\frac{N_{hit}}{N_{true}} \tag{2}\]  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Query List** &
\begin{tabular}{c} **Dev-** \\ **MAVIR** \\ \end{tabular} & **Dev-RTVE** & **Test-MAVIR** & **Test-RTVE** & **Test-SPARL20** \\ \hline \#INL (occ.) & 96 (386) & 81 (464) & 99 (1163) & 89 (808) & 87 (903) \\ \hline \#OOL (occ.) & 6 (39) & 22 (110) & 7 (29) & 19 (72) & 13 (30) \\ \hline \#SING (occ.) & 93 (407) & 101 (544) & 100 (1180) & 105 (861) & 100 (933) \\ \hline \#MULTI (occ.) & 9 (18) & 2 (30) & 6 (12) & 3 (19) & 0 (0) \\ \hline \#INV (occ.) & 83 (296) & 76 (480) & 94 (979) & 87 (750) & 65 (788) \\ \hline \#OOV (occ.) & 19 (129) & 27 (94) & 12 (213) & 21 (130) & 35 (145) \\ \hline \hline \end{tabular}
\end{table}
Table 5: Development and test query list characteristics for the MAVIR, RTVE and SPARL20 databases for the QBE STD task. ‘dev’ stands for development, ‘INL’ refers to in-language queries, ‘OOL’ to foreign terms, ‘SING’ to single-word queries, ‘MULTI’ to multi-word queries, ‘INV’ to in-vocabulary queries, ‘OOV’ to out-of-vocabulary queries and ‘occ.’ stands for occurrences.  
\[p(FA)=\frac{N_{FA}}{T-N_{true}}, \tag{3}\]  
where \(N_{hit}\) is the number of hits obtained by the system, \(N_{true}\) is the actual number of occurrences of the queries in the audio, \(N_{FA}\) is the number of FAs produced by the system and \(T\) denotes the audio length (in seconds). These values, therefore, provide a quantitative way to measure system performance in terms of misses (or equivalently, hits) and false alarms.  
In addition to ATWV, MTWV, p(Miss) and p(FA) figures, NIST also proposed a detection error tradeoff (DET) curve [141] that evaluates the performance of a system at various miss/FA ratios. Although DET curves were not used for the evaluation itself, they are also presented in this paper for system comparison.  
The NIST STD evaluation tool [142] was employed to compute both the MTWV, ATWV, p(Miss) and p(FA) figures, along with the DET curves.