Intra-speaker variability effects on Speaker Verification performance

Abstract Speaker veriﬁcation systems have shown signiﬁcant progressand have reached a level of performance that make their usein practical applications possible. Nevertheless, large differ-ences in terms of performance are observed, depending on thespeaker or the speech excerpt used. This context emphasizesthe importance of a deeper analysis of the system’s performanceover average error rate. In this paper, the effect of the trainingexcerpt is investigated using ALIZE/SpkDet on two differentcorpora: NIST-SRE 08 (conversational speech) and BREF 120(controlled read speech). The results show that the SVS perfor-mance are highly dependent on the voice samples used to trainthe speaker model: the overall Equal Error Rate (EER) rangesfrom 4.1% to 29.1% on NIST-SRE 08 and from 1.0% to 33.0%on BREF 120. The hypothesis that such performance differ-ences are explained by phonetic contents of voice samples isstudied on BREF 120. 1. Introduction Over the last decade, automatic speaker veriﬁcation systems(SVS) have been assessed regularly by the National Institute ofStandards and Technology (NIST) [1]. The evaluation focuseson text-independent speaker detection and offers a common ex-perimental protocol and a stable set of evaluation rules.Although the task difﬁculty is changing through years, the NISTcampaigns clearly show a drastic progress in terms of perfor-mance during the last years. The level of performance reachedby the systems has become suitable for a large set of practical,commercial applications. Many applications are already avail-able or planned for the next future, including the forensic ones.This context underlines the importance of a deep analysis ofthe system’s performance, for instance on a per speaker basis,while the performance is usually assessed only through averageerror rate. In addition to the average performance information,performance variability also needs to be evaluated. Indeed, theidentiﬁcation of the performance variation factors is necessaryfor determining the contexts in which those systems may beused.The system performance is commonly measured using twokinds of errors. A false acceptance (FA) occurs when an impos-tor is accepted by the system. A false rejection (FR) consists ofrejecting a valid identity. Both error rates depend on the thresh-old used in the decision making process. Among the measuresused to compare system performances, detection error trade-off(DET) curve [2], Equal Error Rate (EER) and Decision CostFunction (DCF) are usually used. The DET curve is obtainedby plotting on a normal deviate curve the FA rate as a functionof the FR rate. The EER corresponds to the operating pointwhere FA rate = FR rate when the DCF corresponds to a spe-ciﬁc operating point, described by the weight tied to each error(FA and FR) and the prior probabilities of these errors. In NISTevaluation, each training excerpt is regarded as produced by adifferent speaker, but the same speaker may have been recordedin several extracts. No comparison between these different ex-tracts is conducted.Some studies have investigated the possible causes of perfor-mance variation. [3] showed that the performance of the systemmay be improved by increasing the length of the training andtesting signals. Indeed, the EER raised from 4.48% up to morethan 30% when the duration of the training signals are shortenedfrom 2.5 minutes to 10 seconds. Moreover, a short excerpt intraining is more disadvantageous than a short excerpt in testing(more than 14% of EER with short excerpts in testing vs. morethan 17% of EER with short excerpts in training)Inter-speaker variation has also been studied. Doddington et al.[4] studied the errors induced by different speakers in 12 auto-matic speaker veriﬁcation systems, and showed that the topol-ogy of the errors depends on speakers, consistently from onesystem to another. They distinguished 4 types of speakers, il-lustrated by a ’menagerie’. Sheeps correspond to the defaultspeaker type (low FA, low FR). Goats are speakers who gen-erate a disproportionate false rejection rate. Lambs correspondto speakers who generate a disproportionate false alarm rate.Wolves correspond to speakers that are likely to be mistaken foran other speaker.Finally, the inﬂuence of the phonetic content of test excerptswas evaluated by [5]. Results suggest that glides and liquidstogether, vowels and more particularly nasal vowels and nasalconsonants contain more speaker-speciﬁc information than pho-netically balanced test utterances, even though the training ex-cerpt were composed of 15 seconds of phonetically balancedspeech.This paper focuses on the variability due to the signal sampleused to represent the speaker voice. The information about thespeaker may differ among training excerpts. The aim of this pa-per is to quantify the effect of such a variability on SVS perfor-mance. The SVS scores for each training excerpt are comparedin order to selected the best and the worst training excerpts.Global performance is assessed using two different databases.Section 2 describes the system used. Section 3 and 4 investi-gate on the effect of training excerpt on NIST-08 and Bref 120database respectively. A preliminary phonetic analysis on theBREF 120 database is conducted in section 5 before conclud-ing in section 6.