## 2 Search on Speech Evaluation
### Comparison with Previous Search on Speech International Evaluations
#### 2.4.1 Comparison with Previous STD International Evaluations  
Since the SoS ALBAYZIN evaluation integrates the STD task, it is worth mentioning other previous STD international evaluations. In 2006, NIST launched the first STD evaluation [145]. This evaluation involved different languages (i.e., English, Arabic and Mandarin) and different acoustic conditions (i.e., conversational telephone speech (CTS), broadcast news and round-table meetings). The best performance was obtained for the broadcast news condition and English language (ATWV = 0.8485), for which more data for system construction was typically available by that date.  
IARPA BABEL program started in 2011 and addressed KWS/STD tasks to a great extent as well [146]. This program focused on building fully automatic and noise-robust speech recognition and search systems in a very limited amount of time (e.g., one week) and with limited amount of training data. The languages addressed in that program were low-resourced, such as Cantonese, Pashto, Tagalog, Turkish, Vietnamese, Swahili, Tamil and so on, and significant research has been carried out [147; 148; 149; 150; 151; 152; 153; 154; 155; 156; 157; 158; 159].  
Moreover, NIST continued organising STD evaluations in the so-called open keyword search (OpenKWS) evaluations from 2013 to 2016 [160; 161; 162; 163]. These OpenKWS evaluations were very similar to the NIST STD 2006 evaluation, since they included CTS data and microphone speech data. The main novelty was that the evaluation language was unknown up to 4 weeks (or less) before the system submission. The main results of these evaluations are presented in Table 6.  
From 2017, NIST has also been organising the biennial open speech analytics technologies (OpenSAT) evaluations [164; 165; 166; 167], which include keyword search among their tasks, and aim to provide support for developing novel speech analytic technologies for low-resourced languages by including multiple speech analytic tasks (speech activity detection, speech recognition and keyword spotting) and multiple data domains (low-resourced languages, speech from video and public-safety communications). The only publicly available results are those from the 2017 evaluation and show an ATWV = 0.57 for the low-resourced languages and a quite low ATWV (negative value) for the challenging public-safety communication domain.  
In the STD task of the SoS ALBAYZIN evaluation, speech comprises various recording conditions: (1) real talks in real workshops held in large conference rooms with audience, (2) broadcast news speech and (3) parliament sessions speech. In the workshop data, microphones, conference rooms and even recording conditions change from one recording to another, where tabletop and ground standing microphones were typically employed. In addition, the SoS ALBAYZIN evaluation explicitly defines different in-vocabulary and out-of-vocabulary term sets and makes use of Spanish language. These differences in the evaluation conditions make our evaluation pose different challenges but also make it difficult to compare the results obtained in our evaluation with those of previous NIST STD/OpenKWS/OpenSAT evaluations.