## 2 Search on Speech Evaluation
### Databases
#### 2.2.4. Query List Selection  
Query selection plays an important role within search on speech, since it should carefully take into account different search scenarios. To do so, the queries involved both in the STD and the QBE STD tasks include high occurrence queries, low occurrence queries, in-language (INL) (i.e., Spanish) queries, out-of-language (OOL) (i.e., foreign) queries, single-word and multi-word queries, in-vocabulary and out-of-vocabulary queries and queries of different length. In the evaluation datasets, a query may not have any occurrence or appear one or more times in the speech data. Table 4 includes some features of the development and test query lists, such as the number of INL and OOL queries, the number of single-word and multi-word queries and the number of INV and OOV queries, along with the number of occurrences of each set in the corresponding speech database for the STD task.  
Regarding the QBE STD task, three different acoustic examples per query were provided for both development and test datasets. One example was extracted from the same dataset as the one to be searched (hence in-domain acoustic examples). This scenario considered the case in which the user finds a term of interest within a certain speech dataset and he/she wants to search for new occurrences of the same query.  
The two other examples were recorded by the evaluation organizers and comprised an scenario where the user pronounces the query to be searched (hence, out-of-domain acoustic examples). These two out-of-domain acoustic examples amount to 3 s of speech  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Query List** & **Dev-MAVIR** & **Dev-RTVE** & **Test-MAVIR** & **Test-RTVE** & **Test-SPARL20** \\ \hline \#INL (occ.) & 354 (959) & 307 (1151) & 208 (2071) & 301 (1082) & 236 (1521) \\ \hline \#OOL (occ.) & 20 (55) & 91 (351) & 15 (50) & 103 (162) & 16 (39) \\ \hline \#SING (occ.) & 340 (984) & 380 (1280) & 198 (2093) & 383 (1186) & 252 (1560) \\ \hline \#MULTI (occ.) & 34 (30) & 18 (222) & 25 (28) & 21 (58) & 0 (0) \\ \hline \#INV (occ.) & 292 (668) & 312 (1263) & 192 (1749) & 316 (1035) & 204 (1375) \\ \hline \#OOV (occ.) & 82 (346) & 86 (239) & 31 (372) & 88 (209) & 48 (185) \\ \hline \hline \end{tabular}
\end{table}
Table 4: Development and test query list characteristics for the MAVIR, RTVE and SPARL20 databases for the STD task. ‘dev’ stands for development, ‘INL’ refers to in-language queries, ‘OOL’ to foreign terms, ‘SING’ to single-word queries, ‘MULTI’ to multi-word queries, ‘INV’ to in-vocabulary queries, ‘OOV’ to out-of-vocabulary queries and ‘occ.’ stands for occurrences. The term length of the development query lists varies between 4 and 27 graphemes. The term length of the MAVIR and RTVE test query lists varies between 4 and 28 graphemes. The term length of the SPARL20 test query list varies between 3 and 19 graphemes.  
with PCM, 16 kHz, single channel and 16 bits per sample with the microphone of an HP ProBook Core i5, 7 th Gen and with a Sennheiser SC630 USR CTRL microphone with noise cancellation, respectively.  
The queries employed for the QBE STD task were chosen from the STD queries, and the corresponding figures are presented in Table 5. For both the STD and QBE STD tasks, a multi-word query was considered OOV in the case where any of the words that form the query were OOV.