# Speaker Diarization: A Review of Objectives and Methods
## 1 Introduction  
Processing of speech signals has had widespread application in recent years: _automatic speech recognition_ (ASR--translating spoken audio into text), _speaker verification_ (SV--identifying who is talking), language identification (which language is spoken), emotion recognition, and coding (e.g., _linear predictive coding_ (LPC) in cellular telephony) are the most common uses. Given the prominence of multi-speaker conversations in podcasts, broadcasts, entertainment, and meetings, another valuable speech application is _Speaker Diarization_ (SD), where a computer algorithm automatically estimates \"who spoke when?\" from a recorded audio signal [1]. As many recent papers have noted, SD still needs much research [2, 3], as performance remains far from ideal. One application for SD is to annotate a transcription derived from ASR, which would normally output an estimated text based on input speech, but one would not know who had said this text. Another motivation is to assist the ASR process, to allow application of specific speaker-dependent models to input speech for each portion labelled as spoken by a given person. SD can provide speaker identities for conversational artificial intelligence and help to retrieve audio/video recordings from desired target speakers. Applications for SD also include generating meeting/interview transcripts, medical notes, automated subtitling and dubbing, and downstream speaker analytics [4]. The SD field may include rich transcription, textual content retrieval, and indexing from speech; these applications utilize aspects beyond the scope of basic SD, e.g., determining syntactic boundaries and disfluencies (speech errors such as restarts and hesitations). Speech also often contains so-called _vocalizations_ (e.g., back-channels, filled pauses, singing, infant babbling, laughter, coughs, breaths, lip-snacks, and humming) that are not directly relevant for ASR. This paper will concentrate on acoustic analysis of speech, rather than on textual matters, and we include vocalizations as forms of speech. Incorporating textualinformation into SD would render it dependent on language, and general SD, as discussed here, can be developed in a language-independent fashion. Unlike most other speech applications, where objectives are usually straightforward, the task of SD can be phrased in different ways, depending on a priori knowledge of who may be participating in a multi-speaker audio signal. In comparison to SD, the objective for speech coding is to produce a reconstructed audio signal that is natural and intelligible with low transmission rate and latency. For ASR, the aim is the text intended by a speaker. For speaker, language, or emotion recognition, the desired output is simply the respective identity. However, for SD, the difficulty of the diarization task is greatly influenced by the turn-taking behaviour of speakers, in particular, how much people's speech is separated by non-speech or overlap in their speech [5]. Given an audio signal that contains speech, SD estimates an ordered set of times, called _time stamps_, which correspond to places in the audio each time that a speaker starts and stops talking. In addition, SD labels each time portion (_speaker turn_) by who is talking. For some applications, e.g., closed meetings, all participants are known to the system via prior training. However, when applying SD to arbitrary audio or broadcasts, the number and identities of speakers are often unknown. In these latter cases, SD still outputs time stamps, but notes speech portions (i.e., between each pair of successive time stamps) with labels of Speaker 1, 2, 3, etc. (including portions with multiple speakers). Audio often has overlapping speech, where two or more speakers talk simultaneously [6; 7]. Indeed, in the so-called _cocktail party_ situation [8; 9; 10], a microphone may record the speech of many people talking at the same time.