Significance Test

in Speaker Recognition Data Analysis

with Data Dependency

Jin Chu Wu

Alvin F. Martin

Craig S. Greenberg

Raghu N. Kacker

Vincent M. Stanford

http://dx.doi.org/10.6028/NIST.IR.7884 

NISTIR 7884
 

NISTIR 7884

Significance Test

in Speaker Recognition Data Analysis

with Data Dependency 

Jin Chu Wu

Alvin F. Martin

Craig S. Greenberg

Information Access Division, Information Technology Laboratory 

Raghu N. Kacker

Applied and Computational Mathematics Division, Information Technology Laboratory 

Vincent M. Stanford

Information Access Division, Information Technology Laboratory 

http://dx.doi.org/10.6028/NIST.IR.7884 

October 2012 

U.S. Department of Commerce 

Rebecca Blank, Acting Secretary 

National Institute of Standards and Technology 

Patrick D. Gallagher, Under Secretary of Commerce for Standards and Technology and Director
1

Significance Test 

in Speaker Recognition Data Analysis with Data Dependency 

Jin Chu Wua, Alvin F. Martina, Craig S. Greenberga, Raghu N. Kackerb, and Vincent M. Stanforda

a

Information Access Division, bApplied and Computational Mathematics Division, 

Information Technology Laboratory, 

National Institute of Standards and Technology, Gaithersburg, MD 20899 

Abstract

To evaluate the performance of speaker recognition systems, a detection cost function defined as a 

weighted sum of the probabilities of type I and type II errors is employed. The speaker datasets may 

have data dependency due to multiple uses of the same subjects. Using the standard errors of the 

detection cost function computed by means of the two-layer nonparametric two-sample bootstrap 

method, a significance test is performed to determine whether the difference between the measured 

performance levels of two speaker recognition algorithms is statistically significant. While 

conducting the significance test, the correlation coefficient between two detection cost functions for 

two algorithms, respectively, is taken into account. Examples are provided. 

Keywords: Significance test; Speaker recognition; Data dependency; Standard error; Bootstrap. 
2

1 Introduction 

The Speaker Recognition Evaluation (SRE) is an ongoing project conducted by the National 

Institute of Standards and Technology (NIST) [1]. It has had a great impact on the research efforts 

and the development of technology in the community of the audio, speech, and language processing. 

Each trial in a speaker recognition test consists of a training model speaker and a test speech 

segment. The speaker recognition system must decide whether speech of the model speaker occurs 

in the test speech segment and generate a similarity score. A higher score indicates greater 

confidence that the test speech is spoken by the model speaker. Target (non-target) scores are 

generated by trials in which the test speech segment contains (does not contain) speech of the model 

speaker defined in the training data. 

To evaluate the performance of speaker recognition systems, a detection cost function defined as a 

weighted sum of the probabilities of type I error (miss) and type II error (false alarm) is employed as 

a metric [1]. These two error rates represent a tradeoff and are negatively correlated [2]. Further, the 

NIST speaker recognition data contain dependencies [3]. Data dependency in speaker recognition 

applications arises largely from multiple uses of the same subjects in order to provide more target 

and non-target scores due to limited resources. This data dependency is complicated, due in part to 

the way the data are collected. There are several ways to interpret the dependencies of the data, 

which can impact the bootstrap results. 

In our test, data dependency is determined based solely upon the multiple use of the training speaker 

identification (ID) number. Target scores and non-target scores generated using the same training 

speaker ID number are grouped into a target set and a non-target set, respectively, in order to 

preserve the data dependency. Then the speaker datasets are refined to a two-layer data structure: the 

first layer consists of target sets and non-target sets, and the second layer consists of target scores 

and non-target scores within sets. 

Based on our investigation of the probabilities for scores being selected, and to keep the numbers of 

scores resampled equal from iteration to iteration while using the bootstrap method, the datasets are 

adjusted in such a way that all target sets contain the same number of scores and likewise for the 

non-target sets [3]. The adjusted datasets had 132 target sets (130 non-target sets), each of which 

contained 96 target scores (244 non-target scores); and thus the total number of target (non-target) 

scores was 12,672 (31,720). Hence, there are still tens of thousands of scores in the datasets [3]. 

The sampling variability, including the data dependency, results in uncertainties of the detection cost 

function in the SRE. However, the covariance between the type I and II errors and the data 

dependency make the analytical computation of such uncertainties difficult. Hence, in our prior 

studies, the standard error (SE) of the detection cost function was estimated using the two-layer 

nonparametric two-sample bootstrap method, where the empirical distribution is assumed for each of 

the observed scores, based on our extensive bootstrap variability studies in ROC analysis on large 

datasets [2-9]. 

The two samples involved are referred to as a set of target scores and a set of non-target scores, 

which characterize the speaker recognition system that generates them and usually do not have well 
3

defined parametric forms [10, 11]. In the two-layer bootstrap, the nonparametric two-sample 

resampling takes place randomly with replacement (WR), not only on the first layer of the data, i.e., 

the target sets and non-target sets, but also subsequently on the second layer, i.e., the target scores 

and non-target scores within the sets. While resampling on the first layer, the bootstrap units are sets; 

on the second layer, the bootstrap units are scores within a set, where the similarity scores are 

conditionally independent. 

In evaluating and comparing the performances of speaker recognition systems, it is insufficient to 

only compute the uncertainty of the cost function [2, 12]. It may be of interest to determine whether 

the difference between the measured performance level of a specific speaker recognition algorithm 

and a hypothesized criterion value is real or by chance, or to determine whether the difference 

between the measured performance levels of two algorithms is statistically significant. The 

principles are the same. In SRE the latter is often of more interest, and thus is explored in this article. 

Comparison issues may be examined intuitively to some extent using the 95 % confidence intervals 

(CI) derived from the uncertainty. But it is difficult to reach any conclusion when the two 95 % CIs 

overlap. CIs alone are insufficient to provide quantitative information (such as p-values) on the 

statistical significance of the difference. Thus, statistical hypothesis testing is employed. 

By examining the relationship between the two types of 95 % CIs, it was found that the one 

computed using the quantile method matched very well with the one derived using the normality 

assumption for the distribution of the detection cost function. This suggests that the detection cost 

function be regarded as approximately normally distributed. Thereafter, the Z-test may be used to 

perform significance testing. 

The detection cost functions of the two speaker recognition systems might or might not be 

correlated, depending on how the test is designed and how the sets of similarity scores are generated. 

In our SRE tests, all the scores of the different systems were generated on a common set of speakers 

and speech segments and, therefore, are highly correlated. And thus, the resulting detection cost 

functions are also correlated. In this article, an algorithm is provided to find the correlated pairs of 

metrics from the correlated similarity scores, and then the correlation coefficient of the detection 

cost function can be computed explicitly [2, 12]. 

The notations of sets and scores are provided in Section 2. The formulas for computing the detection 

cost function are presented in Section 3. The general formulas of hypothesis testing for comparing 

two speaker recognition systems are shown in Section 4. The two-layer nonparametric two-sample 

bootstrap algorithm is provided in Section 5. An algorithm for computing the correlation coefficient 

of two detection cost functions is described in Section 6. The results of examples involving five 

speaker recognition systems1

 are presented in Section 7. The conclusions and discussion can be 

found in Section 8. 

 

1

 Specific hardware and software products identified in this paper were used in order to adequately support the 

development of technology to conduct the performance evaluations described in this document. In no case does such 

identification imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it 

imply that the products and equipment identified are necessarily the best available for the purpose. 
4

2 The notations of sets and scores 

target 

ST

sets ST1 ST 2 ...... mT ST 

scores 

T11 , T12 , ..., 

μT1 T1 

T 21 , T 2 2 , ..., 

μT 2 T 2 

...... TmT 1 , TmT 2 , ..., 

T TmT

Tm μ

Table 1 The target sets, the number of which is mT, and the target scores contained in each set. 

nontarget 

SN

sets SN1 SN 2 ...... N mN S

scores 

N11 , N12 , ..., 

μN1 N1 

N 21 , N 2 2 , ..., 

μN 2 N 2 

......  NmN 1 ,  NmN 2 , 

..., N NmN

 Nm μ

Table 2 The non-target sets, the number of which is mN, and the non-target scores contained in each set. 

There are several ways to group data into sets according to data dependency. As discussed in Section 

1, in this article the speaker recognition scores are grouped into sets based on the training speaker ID 

number. Target scores involving a given ID number of the training and test speakers are grouped 

into a target set, whereas non-target scores involving a given ID number of the training speaker but 

different ID numbers of the test speakers are grouped into a non-target set. 

Suppose that the number of the target sets is mT, and the number of the non-target sets is mN. Thus, 

the set ST of all target sets and the set SN of all non-target sets are expressed, respectively, as 

follows, 

Si = { Si j | j = 1, ..., mi }, i  {T, N}, (1) 

where ST j are target sets and SN j are non-target sets. 

In terms of its scores, each set can be expressed as 

Si j = { αi j k | k = 1, ..., μi j }, j = 1, ..., mi and i  {T, N}, (2) 

where αT j k are target scores, αN j k are non-target scores, and μi j stands for the number of scores in 

the corresponding set. 

Due to the reasons stated in Section 1, the datasets are adjusted in such a way that all target sets 

contain the same number of scores and likewise for the non-target sets. In other words, μi 1 = μi 2 = 

... = mi μi , where i  {T, N}. Moreover, since the scores are grouped into sets based on the training 

speaker ID numbers as discussed in Section 1, the two scores of any two speaker recognition 

systems with the same ordinal number of sets and the same ordinal number of scores in sets are 

generated by the speakers and speech segments with the same ID numbers, regardless of whether 

these two scores are target scores or non-target scores. Therefore, these two scores of two systems 

are correlated. 

Hence, the set of all target scores and the set of all non-target scores can be denoted, respectively, as 

T = { αT j k | k = 1, ..., μT j and j = 1, ..., mT }, (3) 
5

and 

N = { αN j k | k = 1, ..., μN j and j = 1, ..., mN }. (4) 

The sets Si j, T, and N are all in the sense of multiset, in which members are allowed to appear more 

than once. Indeed score can occur multiple times within a set. Finally, the total number of target 

scores NT and the total number of non-target scores NN are, respectively, 

 , 



mi

j 1

Ni μi j where i  {T, N}. (5) 

The target and non-target sets and scores contained in each set are explicitly listed in Table 1 and 

Table 2, respectively. There are mT target sets and mN non-target sets. The target sets ST1 , ST 2 , ..., 

mT ST contain μT1 , μT 2 , ..., mT μT target scores, respectively; and the non-target sets SN1 , SN 2 , 

..., N mN S have μ N1 , μN 2 , ..., mN μ N non-target scores, respectively. 

3 The detection cost function in speaker recognition evaluation

After converting scores to integer, without loss of generality, for a speaker recognition system, the 

scores are expressed inclusively using the integer score set {s} = {smin, smin+1, ..., smax}, running 

consecutively from the lowest score smin to the highest score smax. Let Ci (s), i  {T, N} denote the 

cumulative probabilities of target scores and non-target scores from the highest score smax down to 

an integer score s, respectively. 

The probability of type I error at a threshold t {s } for target scores, denoted by PI (t), is cumulated 

from the lowest score smin. The probability of type II error at a threshold t for non-target scores, 

denoted by PII (t), is cumulated from the highest score smax. For discrete probability distribution, 

while computing PI (t) and PII (t) at a threshold t, the probabilities of target scores and non-target 

scores at this threshold t must be taken into account [13]. 

Hence, at a threshold value t {s }, the estimators of the probabilities of type I error and type II 

error are expressed, respectively, as 

P (t) I

ˆ = 1 – CT (t + 1) for t {s } , (6) P (t) II

ˆ = CN (t) 

where CT (smax + 1) = 0 is assumed [2, 10]. Based on Eq. (8), in practice, the estimators P (t) I

ˆ and 

P (t) II

ˆ can be obtained by moving the score from the highest score smax down to the threshold t one 

score at a time to cumulate the probabilities of target scores and non-target scores, respectively. 

A number of metrics exist for measuring the performance level of a speaker recognition system [1]. 

In this article, the detection cost function at a threshold for the primary evaluation of speaker 

detection performance is employed as the metric of interest. However, the same method of 

computing the uncertainties of the detection cost functions can be used to compute uncertainties for 

other metrics in SRE as well. 
6

The detection cost function at a threshold t is defined as a weighted sum of the probabilities of type I 

error and of type II error at the threshold t [1] 

CDet (t) = CMiss × PI (t) × PTarget + CFalseAlarm × PII (t) × (1 – PTarget) . (7) 

Hence, it is a function of the threshold t. It was required that the thresholds be provided by speaker 

recognition systems in order to make an explicit speaker detection decision for each trial. The 

thresholds can also be determined in other ways. It is a challenging research problem to determine 

appropriate decision thresholds, which is outside the scope of this article. Therefore, the thresholds 

used in this article are those provided by the tested systems. 

The parameters CMiss and CFalseAlarm are the relative costs of detection errors, and the parameter PTarget

is the a priori probability of the specified model speaker. For the primary evaluation of speaker 

recognition performance for all speaker detection tests, the parameters CMiss, CFalseAlarm, and PTarget

were set to be 10, 1, and 0.01, respectively [1]. 

4 Two-algorithm hypothesis testing for comparisons 

As pointed out in Section 1, due to the specific applications in SRE, the comparison of two speaker 

recognition systems rather than the evaluation of a system with respect to a hypothesized criterion 

value for the cost function is of more interest and thus is explored in this article. Nonetheless, the 

principles stay the same. 

Let C1 and C2 denote the two detection cost functions for any two speaker recognition systems at 

respective thresholds. Then, the null and alternative hypotheses are 

Ho : C1 = C2 (8) Ha : C1 ≠ C2

If the statistic of interest is normally distributed, the general Z statistic for two-algorithm hypothesis 

testing is expressed as 

Z = 

C ) ˆ C ) SE ( ˆ C ) - 2 SE ( ˆ C ) SE ( ˆ SE (

Cˆ C - ˆ

2 1 2

2

1

2

1 2

 r

(9) 

where Cˆ 1 and Cˆ 2 are two estimators of the detection cost function, SE (Cˆ 1) and SE (Cˆ 2) stand for 

their SEs, respectively, and r is the correlation coefficient between Cˆ 1 and Cˆ 2. 

The Z statistic is distributed as the standard normal distribution with zero expectation and unit 

variance. The standard errors of the detection cost function with data dependency can be computed 

using the two-layer nonparametric two-sample bootstrap (see Section 5). If the two statistics of 

interest are positively correlated and the correlation coefficient r is not taken into account, it can 

leave the denominator of Eq. (9) larger and the Z score smaller; thereby reduce the chance of 

detecting a performance difference between two algorithms. 

There is no reason to believe a priori that the performance of one algorithm is likely to be better 

than the performance of the other algorithm. Further, the two-tailed test is generally more 
7

conservative than the one-tailed test in the sense that the former is more difficult to reject the null 

hypothesis for a given significance level [14]. Thus, the two-tailed Z-test is used in this article. 

5 An algorithm for the two-layer nonparametric two-sample bootstrap 

Because it is difficult to compute analytically the covariance term of the correlated probabilities of 

type I error PI (t) and type II error PII (t) at a threshold t in Eq. (7) and also because the data 

dependency is involved, the two-layer nonparametric two-sample bootstrap method is proposed to 

compute the estimate of the uncertainty of the detection cost function at a threshold t, based on our 

previous studies of bootstrap variability in ROC analysis on large datasets [2-9]. 

The two-layer resampling is carried out not only on the first layer of the new data structure where 

the resampling units are target sets and non-target sets, but also on the second layer of the data in 

which the resampling units are target scores and non-target scores in sets. From here on, the 

superscript indices are used for the numeration of the resampling iterations. The algorithm is shown 

as follows. 

Algorithm I (two-layer nonparametric two-sample bootstrap)

1: for i = 1 to B do 

2: WR_Random_Sampling_Set ( mT, ST, S 'T

i

 = { S 'T j

i

 | j = 1, ..., mT } ) 

3: for k = 1 to mT do

4: WR_Random_Sampling_Set ( μ'T k

i

, S 'T k

i

, S "T k

i

 ) 

5: end for 

6: WR_Random_Sampling_Set ( mN, SN, S 'N

i

 = { S 'N j

i

 | j = 1, ..., mN } ) 

7: for k = 1 to mN do

8: WR_Random_Sampling_Set ( μ'N k

i

, S 'N k

i

, S "N k

i

 ) 

9: end for

10: S "T

i

 = { S "T j

i

 | j = 1, ..., mT } and S "N

i

 = { S "N j

i

 | j = 1, ..., mN } => statistic i Cˆ

11: end for 

12: Q (1- / 2)) ˆ Q ( / 2), ˆ E and ( Sˆ C |i 1,...,B} ˆ { i    

13: end 

1.1: function WR_Random_Sampling_Set (L, Γ, Θ) 

1.2: for i = 1 to L do 

1.3: select randomly WR an index j  { 1, ..., L } 

1.4: θi = γj

1.5: end for 

1.6: end function 
8

The two-layer nonparametric two-sample bootstrap calls the function WR_Random_Sampling_Set. 

In this function, Γ stands for a set of sets or a set of scores, L is the cardinality of the set Γ, Θ

represents a new set of sets or scores accordingly with the same cardinality, and γj and θi are 

members of the sets Γ and Θ, respectively. Notice that this function can be applied to either a set of 

sets or a set of scores. It runs L iterations as shown from Step 1.2 to Step 1.5. In the i-th iteration, a 

member of the set Γ is randomly selected WR to become a member of a new set Θ, as indicated in 

Steps 1.3 and 1.4. As a result, L members (sets or scores) are randomly selected WR from the set Γ

to form a new set Θ. 

In Algorithm I, B is the number of the bootstrap replications, i.e., the number of iterations as shown 

from Step 1 to 11, ST is the set of all target sets and SN is the set of all non-target sets as expressed in 

Eq. (1), and mT and mN are the cardinalities of the set ST and the set SN, respectively. 

In the i-th iteration, as shown in Step 2 and Step 6, the function WR_Random_Sampling_Set is 

applied to the first layer of datasets, i.e., the target and non-target sets. That is, mT target sets are 

randomly selected WR from the set ST of all original target sets to form a new set S 'T

i

 = { S 'T j

i

 | j = 

1, ..., mT }, and mN non-target sets are randomly selected WR from the set SN of all original nontarget sets to constitute a new set S 'N

i

 = { S 'N j

i

 | j = 1, ..., mN }. 

Subsequently, the same function is applied to the second layer of datasets, i.e., the similarity scores 

in sets as well. As shown from Step 3 to 5, mT iterations take place after the first-layer resampling of 

the target sets in Step 2. In the k-th iteration, μ'T k

i

 target scores are randomly selected WR from the 

target set S 'T k

i

, which is the k-th new target set from the first-layer resampling, to form the k-th 

new target set S "T k

i

 of the second-layer resampling. The analogous interpretation can be applied to 

non-target scores in the non-target set S 'N k

i

 as shown from Step 7 to 9. 

As indicated in Step 10, all target scores in the new set S "T

i

 = { S "T j

i

 | j = 1, ..., mT } and all nontarget scores in the new set S "N

i

 = { S "N j

i

 | j = 1, ..., mN } are employed to calculate the estimators 

of the probabilities of type I and type II errors, i.e., P (t) I

ˆ and P (t) II

ˆ using Eq. (6) and then the i-th 

bootstrap replication of the estimated detection cost function at a given threshold, i.e., i Cˆ using Eq. 

(7). 

With the new data structure described in Section 1, not only does each target (non-target) score have 

the same probability to be selected, but also the same numbers of target scores and the same 

numbers of non-target scores, respectively, are resampled in Step 10 at different iterations of the 

two-layer nonparametric two-sample bootstrap. All these can reduce the variance of the 

computation. 

Finally, as shown in Step 12, from the set { i Cˆ | i = 1, ..., B }, the standard error SÊ of the detection 

cost function is estimated by the sample standard deviation of the B bootstrap replications, and the 

estimators of the /2 100 % and (1 - /2) 100 % quantiles of the bootstrap distribution, denoted by 

Q ( / 2) ˆ α and ) Q (1 / 2 ˆ α , at the significance level  can be calculated [5]. Definition 2 of quantile 

in Ref. [15] is adopted. That is, the sample quantile is obtained by inverting the empirical 
9

distribution function with averaging at discontinuities. Thus, )) Q (1 / 2 ˆ Q ( / 2), ˆ ( α α stands for the 

estimated bootstrap (1 - ) 100 % CÎ. If 95 % CÎ is of interest, then  is set to be 0.05. 

The remaining issue is to determine how many iterations the bootstrap algorithms need to run in 

order to reduce the bootstrap variance and ensure the accuracy of the computation. In our 

applications, such as biometrics and the evaluation of speaker recognition, etc., the sizes of datasets 

are tens to hundreds of thousands of similarity scores, which are much larger than those in some 

other applications of bootstrap methods like medical decision making, etc. [5]. Moreover, in ROC 

analysis our statistics of interest are mostly probabilities or a weighted sum of probabilities, etc. 

rather than a simple sample mean. And most importantly our data samples of similarity scores have 

no parametric model to fit. Therefore, the bootstrap variability was re-studied empirically, and the 

appropriate number of bootstrap replications B for our applications was determined to be 2,000 [2, 8, 

9]. 

6 An algorithm for computing the correlation coefficient 

The two detection cost functions for any two speaker recognition systems may or may not be 

correlated. However, they are correlated in our test set due to its structure. For example, consider 

two speaker recognition systems denoted by A and B. These two systems have the same two-layer 

data structures. As stated in Section 2, the two systems generate two scores with the same ordinal 

number of sets and the same ordinal number of scores in sets by matching the speakers and speech 

segments with the same ID numbers, for target scores as well as non-target scores. Therefore, these 

two scores corresponding to the two systems co-vary. Consequently the detection cost functions of 

any two systems, computed using similarity scores in Eqs. (6) and (7), are also correlated. 

An algorithm for computing the correlation coefficient of two detection cost functions is as follows. 

Algorithm II (Correlation coefficient)

1: for i = 1 to M do 

2: Synchronized_WR_Random_Sampling_Set 

 ( mT, S A

T, S A 'T

i

 = { S A 'T j

i

 | j = 1, ..., mT }, S B

T, S B 'T

i

 = { S B 'T j

i

 | j = 1, ..., mT } ) 

3: for k = 1 to mT do

4: Synchronized_WR_Random_Sampling_Set ( μ'T k

i

, S A 'T k

i

, S A "T k

i

, S B 'T k

i

, S B "T k

i

 ) 

5: end for 

6: Synchronized_WR_Random_Sampling_Set 

 ( mN, S A

N, S A 'N

i

 = { S A 'N j

i

 | j = 1, ..., mN }, S B

N, S B 'N

i

 = { S B 'N j

i

 | j = 1, ..., mN } ) 

7: for k = 1 to mN do

8: Synchronized_WR_Random_Sampling_Set ( μ'N k

i

, S A 'N k

i

, S A "N k

i

, S B 'N k

i

, S B "N k

i

 ) 

9: end for

10: S A "T

i

 = { S A "T j

i

 | j = 1, ..., mT } and S A "N

i

 = { S A "N j

i

 | j = 1, ..., mN } => statistic A i Cˆ

11: S B

 "T

i

 = { S B "T j

i

 | j = 1, ..., mT } and S B "N

i

 = { S B "N j

i

 | j = 1, ..., mN } => statistic Bi Cˆ
10

12: end for 

13: } {Cˆ |i 1, ..., M A i  and } {Cˆ |i 1, ..., M Bi  => the correlation coefficient ݎ̂

AB

C

14: end 

2.1: function Synchronized_WR_Random_Sampling_Set (L, ΓA, ΘA, ΓB

, ΘB) 

2.2: for j = 1 to L do 

2.3: select randomly WR an index k  { 1, ..., L } 

2.4: θA

j = γ

A

k

2.5: θB

j = γ

B

k

2.6: end for 

2.7: end function 

where ΓA stands for a set of sets or a set of scores generated by System A, and L is its cardinality. 

The analogous meanings are applied to ΘA, ΓB

, and ΘB. And γ

A

k, θAj, γ

B

k, and θBj are their members, 

respectively. Algorithm II is similar to Algorithm I, except that the Algorithm II is applied to two 

speaker recognition systems simultaneously. Based on our bootstrap variability studies, the number 

of iterations M is set to be 2000 [2, 8, 9]. 

The function Synchronized_WR_Random_Sampling_Set can be applied to either a set of sets or a 

set of scores. As stated above, the two sets or scores γ

A

k and γ

B

k of two systems A and B with the 

same ordinal number k co-vary. Therefore, this function synchronizes the selection in a set ΓA

created by System A and the selection in a set ΓB

 generated by System B so that the sets or the 

scores with the same ordinal number k are chosen to form two new sets ΘA and ΘB

, respectively. In 

other words, the correlated similarity scores generated by these two systems are selected. 

From Step 1 to 12, Algorithm II runs M iterations. In the i-th iteration, in Step 2, the function is 

applied simultaneously to the first layer of the datasets, i.e., the set S A

T of target sets generated by 

System A and the set S B

T of target sets created by System B so that the two target sets with the same 

ordinal number in the two systems’ datasets are randomly selected WR and form two new sets S A 'T

i

and S B 'T

i

 of target sets. 

Then, from Step 3 to 5, this function is applied simultaneously to the second layer of the datasets mT

times created by the two systems. Hence, the target scores in set S A 'T k

i

 generated by System A and 

the target scores in set S B 'T k

i

 created by System B with the same ordinal number in the two 

systems’ datasets are randomly chosen WR. These correlated similarity scores constitute two new 

sets of target scores S A "T k

i

 and S B "T k

i

, respectively. 

The analogous interpretation can be applied to non-target sets and non-target scores in sets from 

Step 6 to 9. In Step 10, the target scores in set S A "T

i

 and the non-target scores in set S A "N

i

 for 

System A produce the i-th bootstrap replication of the estimated detection cost function A i Cˆ for 

System A. In Step 11, the correlated target scores in set S B

 "T

i

 and the correlated non-target scores 

in set S B

 "N

i

 for System B produce the i-th bootstrap replication Bi Cˆ for System B. Thus, the 

correlated pairs of bootstrap replications of estimated cost functions are calculated from the 
11

correlated similarity scores. Finally, in Step 13, the estimated correlation coefficient of the detection 

cost functions, ݎ̂

AB

C, is computed from these two sets of correlated bootstrap replications of 

estimated cost functions [13]. 

7 Results 

Systems Cost functions SÊs 95% CÎs 

EL 0.022199 0.001952 (0.018384, 0.026084) 

UJ 0.028996 0.002026 (0.025082, 0.033150) 

BK 0.031588 0.001883 (0.028046, 0.035311) 

LZ 0.040098 0.002897 (0.034641, 0.045880) 

DL 0.040880 0.001841 (0.037185, 0.044511) 

Table 3 The estimated detection cost functions, SÊs, and 95 % CÎs of five speaker recognition systems. 

Figure 1 The estimated detection cost functions, and 95 % CÎs of five speaker recognition systems. 

Five speaker recognition systems, labeled as EL, UJ, BK, LZ and DL2

, are used for illustration. 

Their estimated detection cost functions, the estimated SÊs, and 95 % CÎs are shown in Table 3. In 

this table, the systems are listed in ascending order of the estimated detection cost functions. The 

estimated cost functions were derived using Eq. (7), in which all parameters were set as in Section 3 

and the thresholds were all provided by speaker recognition systems. The estimated SÊs, and 95 % 

CÎs were all computed using the two-layer nonparametric two-sample bootstrap method as described 

in Section 5 taking account of the data dependency in the speaker datasets. 

 

2

 It is the policy of NIST and the evaluation sponsors not to publicly associate specific SRE participants with their 

evaluation performance results, and therefore system names are encoded in this article. 
12

The estimated 95 % CÎs shown in Table 3 were all calculated using the quantile method as described 

in Section 5. They can also be computed by multiplying 1.96 by the estimated SÊ, assuming that the 

distribution of 2,000 bootstrap replications of the detection cost function is normal. These two types 

of 95 % CÎs are matched up to the third or fourth decimal place for all five systems shown in Table 

3. For instance, for system EL, the 95 % CÎ derived from the quantile method is (0.018384, 

0.026084) as shown in Table 3, while it is (0.018374, 0.026024) based on the assumption of 

normality. This suggests that the detection cost function may be normally distributed. 

Figure 1 depicts the estimated detection cost functions, and their estimated 95 % CÎs, for the five 

speaker recognition systems. The estimated 95 % CÎs overlap in some cases. For instance, the 95 % 

CÎ of System EL somewhat overlaps the one of System UJ; and the latter considerably overlaps that 

of System BK. If the speaker recognition system pair needs to be compared to assess which may be 

more accurate than the other, no conclusion can be reached in these cases. In other words, useful 

comparisons cannot be made in these cases merely using the confidence interval approach. 

Systems EL UJ BK LZ DL 

EL 1.000000 0.233958 0.433872 0.620300 0.388808 

UJ 1.000000 0.347396 0.196418 0.425286 

BK 1.000000 0.437193 0.640776 

LZ 1.000000 0.426599 

DL 1.000000 

Table 4 The average correlation coefficients of two detection cost functions out of 20 runs of five speaker 

recognition systems. 

Systems EL UJ BK LZ DL 

EL 1.0000 0.0058 0.0000 0.0000 0.0000 

UJ 1.0000 0.2463 0.0005 0.0000 

BK 1.0000 0.0015 0.0000 

LZ 1.0000 0.7713 

DL 1.0000 

Table 5 The two-tailed p-values of two speaker recognition systems, where the correlation coefficients were taken 

into account. 

To determine whether the difference between the performances of two speaker recognition systems 

is statistically significant, hypothesis testing is carried out. Based on the normality assumption for 

the distribution of the cost function as stated above, the two-algorithm hypothesis testing provided in 

Section 4 can be employed. 

The correlation coefficient of the detection cost functions of two systems, which appears in Eq. (9), 

can be estimated using Algorithm II as presented in Section 6. This algorithm involves a 

synchronized random resampling. Due to the stochastic nature of resampling, for this article, the 

algorithm was run 20 times, and the average out of these runs was taken to be the resultant 

correlation coefficient for significance testing, in order to reduce the computational fluctuation. In 
13

our testing, the algorithm was also run 50 times, and the results did not alter the qualitative 

substance shown here. In practice, if the p-value is considerably different from the critical value of 

interest such as 5 %, 1 %, etc., then this algorithm only needs to run once. 

All the correlation coefficients are shown in Table 4. Note that all of them are positive. This 

indicates as expected that all systems tend to assign higher or lower similarity scores to particular 

trials. Thus, these results provide evidence that the synchronized algorithm for computing the 

correlation coefficient is quite reasonable. 

All two-tailed p-values of system pair among the five speaker recognition systems are presented in 

Table 5, where the correlation coefficients were taken into account. In this table, only two p-values 

are greater than 5 %. They are 24.63 % for Systems UJ and BK and 77.13 % for Systems LZ and 

DL. This indicates that the null hypothesis cannot be rejected, i.e., the performance differences 

between UJ and BK and between LZ and DL are not significant, even though the estimated detection 

cost functions of Systems UJ and LZ are smaller than those of Systems BK and DL, respectively. 

This conclusion is consistent with the observation in Figure 1, where the estimated 95 % CÎs of the 

detection cost functions for Systems UJ and LZ overlap considerably those for Systems BK and DL, 

respectively. 

All other p-values in Table 5 are considerably less than 5 %. This suggests that the null hypothesis 

be strongly rejected. That is, the performance difference between the corresponding two systems is 

real. For instance, comparing Systems EL and UJ, the two-tailed p-value is 0.58 %. Thus, the 

performance of System EL is significantly better than the performance of System UJ, although their 

estimated 95 % CÎs slightly overlap as shown in Table 3 and Figure 1. 

In addition, the magnitudes of the p-values in Table 5 suggest, to some extent, how much the 

corresponding 95 % CÎs overlap. Thus, they describe quantitatively how significant the differences 

are between the performances of the two systems. In other words, the statistical hypothesis testing 

provides quantitative information (such as p-values) regarding the statistical significance of 

differences. 

8 Conclusions and discussion 

SRE involves the evaluation and comparison of speaker recognition systems. It can be important to 

determine whether the difference between the performance level of one speaker recognition system 

and a performance criterion value, or the difference between the performance levels of two systems 

is statistically significant. In this article, the latter case was investigated, but the principle involved in 

the former case is similar. 

To evaluate the performance of speaker recognition systems, a detection cost function defined as a 

weighted sum of the probabilities of type I error (miss) and type II error (false alarm) is employed as 

a metric. The NIST speaker recognition data contain dependencies due to multiple uses of the same 

subjects. Thus, the similarity scores are grouped into sets to preserve the data dependency, and the 

speaker datasets are refined into a two-layer data structure. 
14

The sampling variability, including this data dependency, results in uncertainty for the value of the 

detection cost function. The uncertainties of the detection cost function in terms of SE and 95 % CI 

were computed using the two-layer nonparametric two-sample bootstrap method with 2000 

bootstrap replications based on our variability study of bootstraps in ROC analysis. 

The detection cost function may be approximately normally distributed regardless of the 

distributions of target scores and non-target scores. This assumption is supported by the matches 

between two types of 95 % CIs. One is computed using the definition of quantile, while the other is 

calculated based on the assumption that the distribution of 2000 bootstrap replications of the statistic 

of interest is normal. As a consequence, it seems reasonable to apply the Z-test. 

In SRE, the similarity scores of any two speaker recognition systems are correlated. Therefore, the 

detection cost functions of two systems are also correlated. If the two statistics of interest are indeed 

positively correlated and the correlation coefficient is not taken into account, the likelihood of 

detecting a difference between the performance levels of two systems will be reduced. In this article, 

a synchronized algorithm is provided to calculate such correlation coefficients. 

This algorithm is a stochastic process, since it involves a synchronized sampling. In practice, if the 

p-value is not considerably different from the critical value of interest, such as 5 %, 1 %, etc., then 

this algorithm needs to run several times (20 in our case) in order to reduce the computational 

fluctuation. The average correlation coefficient from these is taken to be the resultant correlation 

coefficient for the significance test. 

When conducting comparisons, the 95% CIs can be examined intuitively. It is hard, however, to 

reach any conclusion when the two 95 % CIs overlap. Determining whether the difference is real or 

by chance may be addressed using a significance test. As presented in Section 7, although the 95 % 

CIs of Systems EL and UJ did slightly overlap, hypothesis testing showed that the difference in 

performance levels between these two algorithms was statistically significant. 

The pairwise comparison conducted after obtaining a priori knowledge from the relationship among 

the 95% CIs as described in Section 7 is to show how crucial the significance test is if the two 95% 

CIs overlap while determining whether the difference between the performances of the two 

algorithms is statistically significant. If the confidence intervals for any combinations of algorithms 

are of interest, for instance, then some multiple comparison procedures, such as Tukey’s method, 

Scheffe’s method, Bonferroni’s method and so on, might need to be employed [16, 17, and 

references therein]. 

Conventionally, if the two-tailed p-value is greater than or equal to 5 %, the null hypothesis is not 

rejected; if it is less than 5 %, the null hypothesis is rejected in favor of the alternative hypothesis. In 

the literature [5], it is alternatively suggested: If the p-value is less than 0.10, borderline evidence is 

against Ho; if the p-value is less than 0.05, reasonably strong evidence is against Ho; if the p-value is 

less than 0.025, strong evidence is against Ho; if the p-value is less than 0.01, very strong evidence is 

against Ho. 
15

References 

1. “The NIST Speaker Recognition Evaluation”, the URL of the website is at 

http://www.itl.nist.gov/iad/mig/tests/spk/ (2012). 

2. J.C. Wu, A.F. Martin and R.N. Kacker, Measures, uncertainties, and significance test in 

operational ROC analysis, J. Res. Natl. Inst. Stand. Technol. 116 (1), 517-537 (2011). 

3. J.C. Wu, A.F. Martin, C.S. Greenberg and R.N. Kacker, Data dependency on measurement 

uncertainties in speaker recognition evaluation, in Active and Passive Signatures III, Proc. 

SPIE 8382, 83820D (2012). 

4. B. Efron, Bootstrap methods: Another look at the Jackknife, Ann. Statistics 7, 1-26 (1979). 

5. B. Efron, and R.J. Tibshirani, An Introduction to the Bootstrap, Chapman & Hall, New York, 

(1993). 

6. R.Y. Liu, and K. Singh, Moving blocks jackknife and bootstrap capture weak dependence, in 

Exploring the limits of bootstrap, ed. by LePage and Billard. John Wiley, New York, (1992). 

7. R.M. Bolle, J.H. Connell, S. Pankanti, N.K. Ratha and A.W. Senior, Guide to Biometrics, 

Springer, New York, 269-292 (2003). 

8. J.C. Wu, Studies of operational measurement of ROC curve on large fingerprint data sets 

using two-sample bootstrap, NISTIR 7449, National Institute of Standards and Technology, 

September, (2007). 

9. J.C. Wu, A.F. Martin and R.N. Kacker, Bootstrap variability studies in ROC analysis on 

large datasets, Communications in Statistics – Simulation and Computation, in press (2012). 

10. J.C. Wu, and C.L. Wilson, Nonparametric analysis of fingerprint data on large data sets, 

Pattern Recognition 40 (9), 2574-2584 (2007). 

11. J.C. Wu, A.F. Martin, C.S. Greenberg and R.N. Kacker, Uncertainties of measures in speaker 

recognition evaluation, in Active and Passive Signatures II, Proc. SPIE 8040, 804008 (2011). 

12. J.C. Wu, A.F. Martin, R.N. Kacker and C.R. Hagwood, Significance test in operational ROC 

analysis, in Biometric Technology for Human Identification VII, Proc. SPIE 7667, 76670I 

(2010). 

13. B. Ostle, and L.C. Malone, Statistics in Research: Basic Concepts and Techniques for 

Research Workers, fourth ed., Iowa State University Press, Ames, (1988). 

14. Box, G. E. P., Hunter, J. S. and Hunter, W. G., Statistics for experimenters: design, 

innovation, and discovery, second ed., John Wiley & Sons, Inc., New York, (2005). 

15. R.J. Hyndman, and Y. Fan, Sample quantiles in statistical packages, American Statistician 

50, 361-365 (1996). 

16. H. Abdi, The Bonferroni and Sidak corrections for multiple comparisons, in Encyclopedia of 

Measurement and Statistics, ed. by N. Salkind, Thousand Oaks (CA): Sage, (2007). 

17. Engineering Statistics Handbook, by the National Institute of Standards and Technology, at 

http://www.itl.nist.gov/div898/handbook/ (2012).