Sequential Algorithms for Testing Closeness of Distributions

What advantage do sequential procedures provide over batch algorithms for testing properties of unknown distributions ? Focusing on the problem of testing whether two distributions D1 and D2 on { 1 , . . . , n } are equal or ε-far , we give several answers to this question . We show that for a small alphabet size n , there is a sequential algorithm that outperforms any batch algorithm by a factor of at least 4 in terms sample complexity . For a general alphabet size n , we give a sequential algorithm that uses no more samples than its batch counterpart , and possibly fewer if the actual distance TV ( D1 , D2 ) between D1 and D2 is larger than ε . As a corollary , letting ε go to 0 , we obtain a sequential algorithm for testing closeness when no a priori bound on TV ( D1 , D2 ) is given that has a sample complexity Õ ( n 2/3 TV ( D1 , D2 ) 4/3 ) : this improves over the Õ ( n/ lognTV ( D1 , D2 ) 2 ) tester of Daskalakis and Kawase [ 2017 ] and is optimal up to multiplicative constants . We also establish limitations of sequential algorithms for the problem of testing closeness : they can improve the worst case number of samples by at most a constant factor . 1 Introduction . How to test if two discrete sources of randomness are similar or distinct ? This basic and ubiquitous question is surprisingly not closed if frugality matters , that is if one wants to take the right decision using as few samples as possible . To state the problem more precisely , one first needs to define what “ distinct '' means . In this paper , we endow the set of probability distributions on { 1 , . . . , n } with the total variation distance TV , and we fix a tolerance parameter ε ∈ [ 0 , 1 ] . We consider two distributions D1 and D2 , and we assume that either D1 = D2 or TV ( D1 , D2 ) > ε . Whenever 0 < TV ( D1 , D2 ) ≤ ε , we do not expect any determined behaviour from our test . When both distributions are unknown we are testing closeness , based on an equal number of independent samples of both distributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We also need to specify what kind of “ test '' is considered . Here we treat the two hypotheses symmetrically ( there is no “ null hypothesis '' ) : given a fixed risk δ ∈ ( 0 , 1 ) , we expect our procedure to find the true one with probability 1− δ , whichever it is . We call such a procedure δ-correct . Finally , we consider and compare two notions of “ frugality '' : in the batch setting , the agent specifies in advance the number of samples needed for the test : she takes her decision just after observing the data all at once , and the sample complexity of the test is the smallest sample size of a δ−correct procedure . In the sequential setting , the agent observes the samples one by one , and decides accordingly whether she takes her decision or requests to see more samples before making a decision . Then , the sample complexity of the test is the smallest expected number of samples needed before a δ-correct procedure takes a decision . Note that this expected number can depend on the unknown distributions D1 and D2 , which can lead to important advantages of sequential procedures . Contributions When n ≥ 2 is small , we show that the optimal sample complexities can be precisely characterized ( up to lower order terms in δ ) in both the batch and sequential setting as shown in Table 1 . This establishes a provable advantage for sequential strategies over batch strategies when n log ( 1/δ ) : sequential algorithms reduce the sample complexity by a factor of at least 4 for any pair of distributions D1 and D2 . In addition , the sequential algorithms stop even more rapidly if the tested distributions are far ( i.e. , TV ( D1 , D2 ) ε ) . The improvements of the sequential algorithm are illustrated in Fig . 1 . The sequential algorithms use stopping rules inspired from time uniform concentration inequalities . The problem of testing closeness for small n is studied in Sec . 3 . For general n ≥ 2 , we improve the dependence on ε to ε ∨ TV ( D1 , D2 ) in the best batch algorithm due to Diakonikolas et al . [ 2020 ] , which is known to be optimal up to multiplicative constants . Namely we obtain a sequential closeness testing algorithm using a number of samples given by O ( max ( n2/3 log1/3 ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 4/3 , n1/2 log1/2 ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 2 , log ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 2 ) ) . ( 1 ) As a special case , when ε = 0 ( the algorithm should not stop when D1 = D2 in this case ) we show that there is an algorithm that stops after O ( max ( log log ( 1/d ) d2 , n2/3 log log ( 1/d ) 1/3 d4/3 , n1/2 log log ( 1/d ) 1/2 d2 ) ) ( 2 ) samples where d = TV ( D1 , D2 ) > 0 . This is an improvement over the sequential algorithm of Daskalakis and Kawase [ 2017 ] which uses Θ ( n/ lognd2 log log ( 1/d ) ) samples . We design the stopping rules according to a time uniform concentration inequality deduced from McDiarmid ’ s inequality , where we use the ideas of Howard et al . [ 2018 , 2020 ] in order to obtain powers of log log ( 1/d ) instead of log ( 1/d ) . We show that the sample complexity for the testing closeness problem given by Eq . ( 1 ) is optimal up to multiplicative constants in the worst case setting ( i.e. , when looking for a bound independent of the distributions D1 and D2 ) . To do so , we construct two families of distributions whose cross TV distance is exactly d ≥ ε and hard to distinguish unless we have a number of samples given by Eq . ( 1 ) . This latter lower bound is based on properties of KL divergence along with Wald ’ s Lemma . We establish a lower bound on the number of queries that matches Eq . ( 2 ) up to multiplicative constants . The proof is inspired by Karp and Kleinberg [ 2007 ] who proved lower bounds for testing whether the mean of a sequence of i.i.d . Bernoulli variables is smaller or larger than 1/2 . We construct well-chosen distributions Dk ( for k integer ) that are at distance εk ( εk decreasing to 0 ) from uniform and then use properties of the Kullback-Leibler ’ s divergence to show that no algorithm can distinguish between Dk and uniform using fewer samples than in Eq . ( 2 ) . Discussion on the setting and related work It is clearly impossible to test D1 = D2 versus D1 6= D2 in finite time : this is why we introduce the slack parameter ε . Other authors like Daskalakis and Kawase [ 2017 ] make a different choice : they fix no ε , but only require that the test decides for D1 6= D2 as soon as it can , and never stops with high probability when D1 = D2 . We focus on the TV distance in testing closeness problems because it characterises the probability of error for the problem of distributions discrimination ; as noted by Daskalakis et al . [ 2018 ] , using other distances such as KL and χ2 is in general impossible . For an overview of testing discrete distributions we recommend the survey of Canonne [ 2020 ] . Testing closeness was solved by Chan et al . [ 2014 ] , and a distribution dependent complexity was found by Diakonikolas and Kane [ 2016 ] and finally the high probability version by Diakonikolas et al . [ 2020 ] . Moreover , the problem of testing D1 = D2 vs D1 6= D2 was solved by Daskalakis and Kawase [ 2017 ] for n = 2 , however the constants are not optimal . They also propose algorithms for the general case using black-box reduction from non-sequential hypothesis testers . Sequential and adaptive procedures have also been explored in active hypothesis setting [ Naghshvar and Javidi , 2013 ] and channels ’ discrimination [ Hayashi , 2009 ] . Sequential strategies have been also considered for testing continuous distributions by Zhao et al . [ 2016 ] and Balsubramani and Ramdas [ 2015 ] . In the latter , the authors design sequential algorithms whose stopping time adapts to the unknown difficulty of the problem . The techniques used are time uniform concentration inequalities which are surveyed by Howard et al . [ 2020 ] . In contrast to the present work , however , they test on the means of the distributions . 2 Preliminaries . We follow mostly Daskalakis and Kawase [ 2017 ] in the notation . Given two distributions D1 and D2 on { 1 , . . . , n } we want to distinguish between two hypothesis H1 : D1 = D2 and H2 : TV ( D1 , D2 ) > ε . We call a stopping rule a function T : ⋃ k∈N [ n ] k × [ n ] k → { 0 , 1 , 2 } such that if T ( x , y ) 6= 0 then T ( xz , yt ) = T ( x , y ) for all strings x , y , z , t with |x| = |y| and |z| = |t| . T ( x , y ) = 1 ( resp . T ( x , y ) = 2 ) means that the rule accepts H1 ( resp . H2 ) after seeing the sequences x and y while T ( x , y ) = 0 means the rule doesn ’ t make a choice and continue sampling . We define two different stopping times , the first τ1 ( T , D1 , D2 ) = inf { t , T ( x1 · · ·xt , y1 . . . yt ) = 1 } and the second τ2 ( T , D1 , D2 ) = inf { t , T ( x1 · · ·xt , y1 . . . yt ) = 2 } where x1 , . . . are i.i.d . samples from D1 and y1 , . . . samples from D2 . We want to find stopping rules satisfying P ( τ2 ( T , D1 , D2 ) ≤ τ1 ( T , D1 , D2 ) ) ≤ δ if D1 = D2 and P ( τ1 ( T , D1 , D2 ) ≤ τ2 ( T , D1 , D2 ) ) ≤ δ whenever TV ( D1 , D2 ) > ε . We call such stopping rules δ-correct . Our goal is to minimize the expected sample complexityE ( τ1 ( T , D1 , D2 ) ) in case of the input is fromD1 , D2 such thatD1 = D2 and E ( τ2 ( T , D1 , D2 ) ) in case of the input is from D1 , D2 such that TV ( D1 , D2 ) > ε .

Summary: The authors tackle (a version of) the two-sample testing problem, where the distributions in question are known to be discrete.  This is clearly a fundamental problem across both theoretical and applied machine learning / statistics.  Letting $\mathcal D_1, \mathcal D_2$ be two distributions on $\{1,\ldots,n\}$, and $\varepsilon > 0$ be some fixed tolerance level, the authors consider testing: $$ H_0: \mathcal D_1 = \mathcal D_2 \quad \textrm{vs.} \quad H_1: \textrm{TV}(\mathcal D_1, \mathcal D_2) > \varepsilon, $$  where $\textrm{TV}$ denotes the usual total variation operator.  The authors study the above problem in both (i) the batch setting, where $\tau > 0$ samples from both $\mathcal D_1, \mathcal D_2$ are available, a priori; and (ii) the sequential setting, where (pairs of) draws from the two distributions stream in, one at a time.  The authors give minimax lower and upper bounds on the testing risk (phrased in the paper in terms of sample complexity, i.e., in terms of $\tau$); actually, the authors give pairs of bounds (one pair for each problem setting): one bound for $n \geq 2$, and another for $n \geq 3$.  A single simulation study is presented.