Sequential Algorithms for Testing Closeness of Distributions

What advantage do sequential procedures provide over batch algorithms for testing properties of unknown distributions ? Focusing on the problem of testing whether two distributions D1 and D2 on { 1 , . . . , n } are equal or ε-far , we give several answers to this question . We show that for a small alphabet size n , there is a sequential algorithm that outperforms any batch algorithm by a factor of at least 4 in terms sample complexity . For a general alphabet size n , we give a sequential algorithm that uses no more samples than its batch counterpart , and possibly fewer if the actual distance TV ( D1 , D2 ) between D1 and D2 is larger than ε . As a corollary , letting ε go to 0 , we obtain a sequential algorithm for testing closeness when no a priori bound on TV ( D1 , D2 ) is given that has a sample complexity Õ ( n 2/3 TV ( D1 , D2 ) 4/3 ) : this improves over the Õ ( n/ lognTV ( D1 , D2 ) 2 ) tester of Daskalakis and Kawase [ 2017 ] and is optimal up to multiplicative constants . We also establish limitations of sequential algorithms for the problem of testing closeness : they can improve the worst case number of samples by at most a constant factor . 1 Introduction . How to test if two discrete sources of randomness are similar or distinct ? This basic and ubiquitous question is surprisingly not closed if frugality matters , that is if one wants to take the right decision using as few samples as possible . To state the problem more precisely , one first needs to define what “ distinct '' means . In this paper , we endow the set of probability distributions on { 1 , . . . , n } with the total variation distance TV , and we fix a tolerance parameter ε ∈ [ 0 , 1 ] . We consider two distributions D1 and D2 , and we assume that either D1 = D2 or TV ( D1 , D2 ) > ε . Whenever 0 < TV ( D1 , D2 ) ≤ ε , we do not expect any determined behaviour from our test . When both distributions are unknown we are testing closeness , based on an equal number of independent samples of both distributions . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . We also need to specify what kind of “ test '' is considered . Here we treat the two hypotheses symmetrically ( there is no “ null hypothesis '' ) : given a fixed risk δ ∈ ( 0 , 1 ) , we expect our procedure to find the true one with probability 1− δ , whichever it is . We call such a procedure δ-correct . Finally , we consider and compare two notions of “ frugality '' : in the batch setting , the agent specifies in advance the number of samples needed for the test : she takes her decision just after observing the data all at once , and the sample complexity of the test is the smallest sample size of a δ−correct procedure . In the sequential setting , the agent observes the samples one by one , and decides accordingly whether she takes her decision or requests to see more samples before making a decision . Then , the sample complexity of the test is the smallest expected number of samples needed before a δ-correct procedure takes a decision . Note that this expected number can depend on the unknown distributions D1 and D2 , which can lead to important advantages of sequential procedures . Contributions When n ≥ 2 is small , we show that the optimal sample complexities can be precisely characterized ( up to lower order terms in δ ) in both the batch and sequential setting as shown in Table 1 . This establishes a provable advantage for sequential strategies over batch strategies when n log ( 1/δ ) : sequential algorithms reduce the sample complexity by a factor of at least 4 for any pair of distributions D1 and D2 . In addition , the sequential algorithms stop even more rapidly if the tested distributions are far ( i.e. , TV ( D1 , D2 ) ε ) . The improvements of the sequential algorithm are illustrated in Fig . 1 . The sequential algorithms use stopping rules inspired from time uniform concentration inequalities . The problem of testing closeness for small n is studied in Sec . 3 . For general n ≥ 2 , we improve the dependence on ε to ε ∨ TV ( D1 , D2 ) in the best batch algorithm due to Diakonikolas et al . [ 2020 ] , which is known to be optimal up to multiplicative constants . Namely we obtain a sequential closeness testing algorithm using a number of samples given by O ( max ( n2/3 log1/3 ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 4/3 , n1/2 log1/2 ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 2 , log ( 1/δ ) ( ε ∨ TV ( D1 , D2 ) ) 2 ) ) . ( 1 ) As a special case , when ε = 0 ( the algorithm should not stop when D1 = D2 in this case ) we show that there is an algorithm that stops after O ( max ( log log ( 1/d ) d2 , n2/3 log log ( 1/d ) 1/3 d4/3 , n1/2 log log ( 1/d ) 1/2 d2 ) ) ( 2 ) samples where d = TV ( D1 , D2 ) > 0 . This is an improvement over the sequential algorithm of Daskalakis and Kawase [ 2017 ] which uses Θ ( n/ lognd2 log log ( 1/d ) ) samples . We design the stopping rules according to a time uniform concentration inequality deduced from McDiarmid ’ s inequality , where we use the ideas of Howard et al . [ 2018 , 2020 ] in order to obtain powers of log log ( 1/d ) instead of log ( 1/d ) . We show that the sample complexity for the testing closeness problem given by Eq . ( 1 ) is optimal up to multiplicative constants in the worst case setting ( i.e. , when looking for a bound independent of the distributions D1 and D2 ) . To do so , we construct two families of distributions whose cross TV distance is exactly d ≥ ε and hard to distinguish unless we have a number of samples given by Eq . ( 1 ) . This latter lower bound is based on properties of KL divergence along with Wald ’ s Lemma . We establish a lower bound on the number of queries that matches Eq . ( 2 ) up to multiplicative constants . The proof is inspired by Karp and Kleinberg [ 2007 ] who proved lower bounds for testing whether the mean of a sequence of i.i.d . Bernoulli variables is smaller or larger than 1/2 . We construct well-chosen distributions Dk ( for k integer ) that are at distance εk ( εk decreasing to 0 ) from uniform and then use properties of the Kullback-Leibler ’ s divergence to show that no algorithm can distinguish between Dk and uniform using fewer samples than in Eq . ( 2 ) . Discussion on the setting and related work It is clearly impossible to test D1 = D2 versus D1 6= D2 in finite time : this is why we introduce the slack parameter ε . Other authors like Daskalakis and Kawase [ 2017 ] make a different choice : they fix no ε , but only require that the test decides for D1 6= D2 as soon as it can , and never stops with high probability when D1 = D2 . We focus on the TV distance in testing closeness problems because it characterises the probability of error for the problem of distributions discrimination ; as noted by Daskalakis et al . [ 2018 ] , using other distances such as KL and χ2 is in general impossible . For an overview of testing discrete distributions we recommend the survey of Canonne [ 2020 ] . Testing closeness was solved by Chan et al . [ 2014 ] , and a distribution dependent complexity was found by Diakonikolas and Kane [ 2016 ] and finally the high probability version by Diakonikolas et al . [ 2020 ] . Moreover , the problem of testing D1 = D2 vs D1 6= D2 was solved by Daskalakis and Kawase [ 2017 ] for n = 2 , however the constants are not optimal . They also propose algorithms for the general case using black-box reduction from non-sequential hypothesis testers . Sequential and adaptive procedures have also been explored in active hypothesis setting [ Naghshvar and Javidi , 2013 ] and channels ’ discrimination [ Hayashi , 2009 ] . Sequential strategies have been also considered for testing continuous distributions by Zhao et al . [ 2016 ] and Balsubramani and Ramdas [ 2015 ] . In the latter , the authors design sequential algorithms whose stopping time adapts to the unknown difficulty of the problem . The techniques used are time uniform concentration inequalities which are surveyed by Howard et al . [ 2020 ] . In contrast to the present work , however , they test on the means of the distributions . 2 Preliminaries . We follow mostly Daskalakis and Kawase [ 2017 ] in the notation . Given two distributions D1 and D2 on { 1 , . . . , n } we want to distinguish between two hypothesis H1 : D1 = D2 and H2 : TV ( D1 , D2 ) > ε . We call a stopping rule a function T : ⋃ k∈N [ n ] k × [ n ] k → { 0 , 1 , 2 } such that if T ( x , y ) 6= 0 then T ( xz , yt ) = T ( x , y ) for all strings x , y , z , t with |x| = |y| and |z| = |t| . T ( x , y ) = 1 ( resp . T ( x , y ) = 2 ) means that the rule accepts H1 ( resp . H2 ) after seeing the sequences x and y while T ( x , y ) = 0 means the rule doesn ’ t make a choice and continue sampling . We define two different stopping times , the first τ1 ( T , D1 , D2 ) = inf { t , T ( x1 · · ·xt , y1 . . . yt ) = 1 } and the second τ2 ( T , D1 , D2 ) = inf { t , T ( x1 · · ·xt , y1 . . . yt ) = 2 } where x1 , . . . are i.i.d . samples from D1 and y1 , . . . samples from D2 . We want to find stopping rules satisfying P ( τ2 ( T , D1 , D2 ) ≤ τ1 ( T , D1 , D2 ) ) ≤ δ if D1 = D2 and P ( τ1 ( T , D1 , D2 ) ≤ τ2 ( T , D1 , D2 ) ) ≤ δ whenever TV ( D1 , D2 ) > ε . We call such stopping rules δ-correct . Our goal is to minimize the expected sample complexityE ( τ1 ( T , D1 , D2 ) ) in case of the input is fromD1 , D2 such thatD1 = D2 and E ( τ2 ( T , D1 , D2 ) ) in case of the input is from D1 , D2 such that TV ( D1 , D2 ) > ε .

Summary: In this paper, the authors look at the problem of the goodness of fit testing in the sequential model. There has been a plethora of results and techniques in the last 20 years for the batch model. Here the authors allow the algorithm to stop at any point in time or ask for more samples. They show that the sequential setting beats the batch setting by at least a factor of 4 in the sample complexity. In fact, for a Boolean sample space, they have a more precise characterization up to the lower order terms in the batch setting. For the n>2 cases, they improve the dependence on the farness parameter from $\epsilon$ to $max(\epsilon, TV(D_1,D_2))$, which in particular improves upon a result by Daskalakis and Kawase for the case when $epsilon=0$.