# The Domain Mismatch Problem in the Broadcast Speaker Attribution Task
## 3 Experimental Protocol  
### Albayzin Corpus and Allowed Data  
The dataset used for the evaluation of the systems is the Albayzin 2020 corpus (Evaluation plan available in [[http://catedrartve.unizar.es/reto2020/EvalPlan-SD-2020-v1.pdf](http://catedrartve.unizar.es/reto2020/EvalPlan-SD-2020-v1.pdf)]([http://catedrartve.unizar.es/reto2020/EvalPlan-SD-2020-v1.pdf](http://catedrartve.unizar.es/reto2020/EvalPlan-SD-2020-v1.pdf)), accessed on 9 September 2021). This dataset consists of approximately 750 h of broadcast content and is the accumulation of past Albayzin evaluations (2012, 2016, 2018 and 2020) proposed by Red Tematica de Tecnologias del Habla (RTTH, [[http://rhabla.es](http://rhabla.es)]([http://rhabla.es](http://rhabla.es)), accessed on 9 September 2021). The content of the dataset, focused on the broadcast domain, considers multiple sources (3/24 TV channel, Corporacion Aragonesa de Radio y Television (CARTV)and Radio Television Espanola (RTVE)), media (radio and TV) as well as languages (Spanish and Catalan). The corpus integrates a wide range of shows and genres, from talk-shows to fiction. Albayzin evaluations are not exclusively focused on speaker technologies. Thus, a wide variety of metadata is released: around 180 h of human-annotated content, including time-referenced speaker labels, 100 h of audio labeled with coarse acoustic segmentation and 570 h of audio with the released broadcast subtitles. For diarization and speaker attribution purposes, we only bear in mind the subset of audio with speaker labels, i.e., approximately 180 h of the total. In this analysis, we divide the corpus imitating the 2020 evaluation conditions: The audio from previous evaluations (2012, 2016 and 2018) is available for training purposes, consisting of approximately 140 h of content. The remaining audio, the data released in 2020, is divided into two subsets: development (4 h of audio) and test (37 h of content). The development set consists of two different shows: the documentary \"Aqui la Tierra\" as well as the teen drama \"Bajo la Red\". The test subset presents a larger heterogeneity with ten different shows that belong to four genres: Fiction series, such as the already mentioned \"Bajo la Red\", as well as \"Boca Norte\", \"Si Fueras Tu\" and \"Wake-up\"; two debate shows: \"Los Desayunos de TVE\" and \"Millenium\"; two comedy shows, namely \"Ese Programa del que Usted me Habla\" and \"Neverfilms\", are also evaluated. Finally, two magazines, \"Aqui la Tierra\" and \"Comando Actualidad\", complete the subset. (All involved shows are reproducible in RTVE Video on Demand (VoD) platform [[https://www.rtve.es/play](https://www.rtve.es/play)]([https://www.rtve.es/play](https://www.rtve.es/play)). accessed on 9 September 2021) The selection of shows for the evaluation involves large variability in terms of duration (from 3.5 min up to 96 min) or the number of speakers (from 6 to 74). Furthermore, although certain shows (\"Aqui la Tierra\", \"Bajo la Red\" and \"Millenium\") are not exclusive to the test subset, this information was deliberately hidden by the organizers during the evaluation to avoid particular setups in these specific audios. In order to perform the speaker attribution task, exclusive of the 2020 edition, extra audio for the enrolled speakers was also released. The provided data includes approximately an hour of audio involving 179 people (18 for development, 161 from test) in 193 recordings (25 for development and 168 in test). Hence, on average, each speaker is enrolled considering 20 s of audio, usually in a single condition. The released data for Albayzin 2020 are originally configured according to an open-set scenario, with non-enrolled speakers in the evaluation audios as well as unseen enrolled speakers. A closed-set version can be obtained by masking all the speech from non-enrolled speakers in both online processing and evaluation. In spite of the large amount of released data in the official Albayzin corpus, 2020 edition rules do not restrict the use of extra data if necessary.