Title: Sequential Predictive Two-Sample and Independence Testing
Venue: NeurIPS 2023
Accepted: yes
Keywords: two-sample testing

independence testing

testing by betting

sequential testing
Abstract: We study the problems of sequential nonparametric two-sample and independence testing. Sequential tests process data online and allow using observed data to decide whether to stop and reject the null hypothesis or to collect more data, while maintaining type I error control. We build upon the principle of (nonparametric) testing by betting, where a gambler places bets on future observations and their wealth measures evidence against the null hypothesis. While recently developed kernel-based betting strategies often work well on simple distributions, selecting a suitable kernel for high-dimensional or structured data, such as images, is often nontrivial. To address this drawback, we design prediction-based betting strategies that rely on the following fact: if a sequentially updated predictor starts to consistently determine (a) which distribution an instance is drawn from, or (b) whether an instance is drawn from the joint distribution or the product of the marginal distributions (the latter produced by external randomization), it provides evidence against the two-sample or independence nulls respectively. We empirically demonstrate the superiority of our tests over kernel-based approaches under structured settings. Our tests can be applied beyond the case of independent and identically distributed data, remaining valid and powerful even when the data distribution drifts over time.

Sequential Predictive Two-Sample and Independence Testing

Abstract

We study the problems of sequential nonparametric two-sample and independence testing. Sequential tests process data online and allow using observed data to decide whether to stop and reject the null hypothesis or to collect more data, while maintaining type I error control. We build upon the principle of (nonparametric) testing by betting, where a gambler places bets on future observations and their wealth measures evidence against the null hypothesis. While recently developed kernel-based betting strategies often work well on simple distributions, selecting a suitable kernel for high-dimensional or structured data, such as images, is often nontrivial. To address this drawback, we design prediction-based betting strategies that rely on the following fact: if a sequentially updated predictor starts to consistently determine (a) which distribution an instance is drawn from, or (b) whether an instance is drawn from the joint distribution or the product of the marginal distributions (the latter produced by external randomization), it provides evidence against the two-sample or independence nulls respectively. We empirically demonstrate the superiority of our tests over kernel-based approaches under structured settings. Our tests can be applied beyond the case of independent and identically distributed data, remaining valid and powerful even when the data distribution drifts over time.

1 Introduction

̸

We consider two closely-related problems of nonparametric two-sample and independence testing. In the former, given observations from two distributions P and Q , the goal is to test the null hypothesis that the distributions are the same: H 0 : P = Q , against the alternative that they are not: H 1 : P = Q . In the latter, given observations from some joint distribution P XY , the goal is to test the null hypothesis that the random variables are independent: H 0 : P XY = P X × P Y , against the alternative that they are not: H 1 : P XY = P X × P Y . Kernel tests, such as kernel-MMD [Gretton et al., 2012] for two-sample and HSIC [Gretton et al., 2005] for independence testing, are amongst the most popular methods for solving these tasks which work well on data from simple distributions. However, their performance is sensitive to the choice of a kernel and respective parameters, like bandwidth, and applying such tests requires additional effort. Further, selecting kernels for structured data, like images, is a nontrivial task. Lastly, kernel tests suffer from decaying power in high dimensions [Ramdas et al., 2015].

Predictive two-sample and independence tests (2STs and ITs respectively) aim to address such limitations of kernelized approaches. The idea of using classifiers for two-sample testing dates back to Friedman [2004] who proposed using the output scores as a dimension reduction method. More recent works focused on the direct evaluation of a learned model for testing. In an initial arXiv 2016 preprint, Kim et al. [2021] proposed and analyzed predictive 2STs based on sample-splitting, namely testing whether the accuracy of a model trained on the first split of data and estimated on the second split is significantly better than chance. The authors established the consistency of asymptotic and exact tests in high-dimensional settings and provided rates for the case of Gaussian

Aaditya Ramdas

̸

distributions. Inspired by this work, Lopez-Paz and Oquab [2017] soon after demonstrated that empirically predictive 2STs often outperform state-of-the-art 2STs, such as kernel-MMD. Recently, Hediger et al. [2022] proposed a related test that utilizes out-of-bag predictions for bagging-based classifiers, such as random forests. To incorporate measures of model confidence, many authors have also explored using test statistics that are based on the output scores instead of the binary class predictions [Kim et al., 2019, Liu et al., 2020, Cheng and Cloninger, 2022, Kübler et al., 2022].

The focus of the above works is on batch tests which are calibrated to have a fixed false positive rate (say, 5% ) if the sample size is specified in advance. In contrast, we focus on the setting of sequentially released data. Our tests allow on-the-fly decision-making: an analyst can use observed data to decide whether to stop and reject the null or to collect more data, without inflating the false alarm rate.

Problem Setup. First, we define the problems of sequential two-sample and independence testing.

Definition 1 (Sequential two-sample testing) . Suppose that we observe a stream of i.i.d. observations (( Z t , W t )) t ≥ 1 , where W t ∼ Rademacher(1 / 2) , the distribution of Z t | W t = +1 is denoted P , and that of Z t | W t = -1 is denoted Q . The goal is to design a sequential test for

̸

Definition 2 (Sequential independence testing) . Suppose that we observe that a stream of observations: (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) ∼ P XY for t ≥ 1 . The goal is to design a sequential test for

̸

We operate in the framework of 'power-one tests' [Darling and Robbins, 1968] and define a levelα sequential test as a mapping Φ : ∪ ∞ t =1 Z t → { 0 , 1 } that satisfies: P H 0 ( ∃ t ≥ 1 : Φ( Z 1 , . . . , Z t ) = 1) ≤ α . We refer to such notion of type I error control as time-uniform . Here, 0 stands for 'do not reject the null yet' and 1 stands for 'reject the null and stop'. Defining the stopping time as the first time that the test outputs 1 : τ := inf { t ≥ 1 : Φ( Z 1 , . . . , Z t ) = 1 } , a sequential test must satisfy

We aim to design consistent tests which are guaranteed to stop if the alternative happens to be true:

Related Works. Our construction follows the principle of testing by betting [Shafer, 2021]. The most closely related work is that of 'nonparametric 2ST by betting' of Shekhar and Ramdas [2023], which later inspired several follow-up works, including sequential (marginal) kernelized independence tests of Podkopaev et al. [2023], and the sequential conditional independence tests under the model-X assumption of Grünwald et al. [2023] and Shaer et al. [2023]. We extend the line of work of Shekhar and Ramdas [2023] and of Podkopaev et al. [2023], studying predictive approaches in detail.

Sequential predictive 2STs were studied by Lhéritier and Cazals [2018, 2019], but in practice, those tests were found to be inferior to the ones developed by Shekhar and Ramdas [2023]. Recently, Pandeva et al. [2022] proposed a predictive 2ST that handles unknown class proportions using ideas from [Wasserman et al., 2020]. In Section 2, we show that our tests are closely related to [Lhéritier and Cazals, 2018, 2019, Pandeva et al., 2022], but are consistent under much milder assumptions.

Sequential Nonparametric Two-Sample and Independence Testing by Betting. Suppose that one observes a sequence of random variables ( Z t ) t ≥ 1 , where Z t ∈ Z . The principle of testing by betting [Shafer and Vovk, 2019, Shafer, 2021] can be described as follows. A player starts the game with initial capital K 0 = 1 . At round t , she selects a payoff function f t : Z → [ -1 , ∞ ) that satisfies E Z ∼ P Z [ f t ( Z ) | F t -1 ] = 0 for all P Z ∈ H 0 , where F t -1 = σ ( Z 1 , . . . , Z t -1 ) denotes the sigma-field generated by Z 1 , . . . , Z t -1 with F 0 being the trivial sigma-field, and bets a fraction of her wealth λ t K t -1 for an F t -1 -measurable λ t ∈ [0 , 1] . Once Z t is revealed, her wealth is updated as

The wealth is used to measure the evidence against H 0 : if a player can make money in such game, the null is rejected. Formally, for testing H 0 at level α ∈ (0 , 1) , we use the stopping rule:

The validity of the test follows from Ville's inequality [Ville, 1939], a time-uniform generalization of Markov's inequality, since ( K t ) t ≥ 0 is a nonnegative martingale starting at 1 under any P Z ∈ H 0 . To ensure high power, one has to choose ( f t ) t ≥ 1 and ( λ t ) t ≥ 1 to guarantee the growth of the wealth if the alternative is true. In the context of two-sample and independence testing, Shekhar and Ramdas [2023] and Podkopaev et al. [2023] recently proposed effective betting strategies based on kernelized measures of statistical distance and dependence respectively which admit a variational representation. In a nutshell, datapoints observed prior to a given round are used to estimate the witness function -one that best highlights the discrepancy between P and Q for two-sample (or between P XY and P X × P Y for independence) testing - and a bet is formed as an estimator of a chosen measure of distance (or dependence). In contrast, our bets are based on evaluating the performance of a sequentially learned predictor that distinguishes between instances from distributions of interest.

Remark 1 . In practical settings, an analyst may not be able to continue collecting data forever and may adaptively stop the experiment before the wealth exceeds 1 /α . In such case, one may use a different threshold for rejecting the null at a stopping time τ , namely U/α , where U is a (stochastically larger than) uniform random variable on [0 , 1] drawn independently from ( F t ) t ≥ 0 . This choice strictly improves the power of the test without violating the validity; see [Ramdas and Manole, 2023].

Contributions. In Section 2, we develop sequential predictive two-sample and independence tests. We establish sufficient conditions for consistency of our tests and relate those to evaluation metrics of the underlying models. In Section 3, we conduct an extensive empirical study on synthetic and real data, justifying the superiority of our tests over the kernelized ones on structured data.

2 Classification-based Two-Sample Testing

We begin with the two-sample testing setting outlined in Definition 5. Let G : Z → [ -1 , 1] denote a class of predictors used to distinguish between instances from P (labeled as +1 ) and Q (labeled as -1 ) 1 . We assume that: (a) if g ∈ G , then -g ∈ G , (b) if g ∈ G and s ∈ [0 , 1] , then sg ∈ G , and (c) predictions are based on sign[ g ( · )] , and if g ( z ) = 0 , then z is assigned to the positive class. Two natural evaluation metrics of a predictor g ∈ G include the misclassification and the squared risks:

which give rise to the following measures of distance between P and Q , namely

It is easy to check that d m ( P, Q ) ∈ [0 , 1 / 2] and d m ( P, Q ) ∈ [0 , 1] . The upper bounds hold due to the non-negativity of the risks and the lower bounds follow by considering g : g ( z ) = 0 , ∀ z ∈ Z . Note that the misclassification risk is invariant to rescaling ( R m ( sg ) = R m ( g ) , ∀ s ∈ (0 , 1] ), whereas the squared risk is not, and rescaling any g to optimize the squared risk provides better contrast between P and Q . In the next result, whose proof is deferred to Appendix D.3, we present an important relationship between the squared risk of a rescaled predictor and its expected margin: E [ W · g ( Z )] .

Proposition 1. Fix an arbitrary predictor g ∈ G . The following claims hold 2 :

For the misclassification risk, we have that:

For the squared risk, we have that:

Further, d s ( P, Q ) > 0 if and only if there exists g ∈ G such that E [ W · g ( Z )] > 0 .

Consider an arbitrary predictor g ∈ G . Note that under the null H 0 in (1a), the misclassification risk R m ( g ) does not depend on g , being equal to 1/2, whereas the squared risk R s ( g ) does. In contrast, the lower bound (10) no longer depends on g under the null H 0 , being equal to 0.

Oracle Test. It is a known fact that the minimizer of either the misclassification or the squared risk is g Bayes ( z ) = 2 η ( z ) -1 , where η ( z ) = P ( W = +1 | Z = z ) . Since g Bayes may not belong to G , we consider g ⋆ ∈ G , which minimizes either the misclassification or the squared risk over predictors in G , and omit superscripts for brevity. To design payoff functions, we follow Proposition 1 and consider

Let the oracle wealth processes based on misclassification and squared risks ( K m ,⋆ t ) t ≥ 0 and ( K s ,⋆ t ) t ≥ 0 be defined by using the payoff functions (11a) and (11b) respectively, along with a predictable sequence of betting fractions ( λ t ) t ≥ 1 selected via online Newton step (ONS) strategy [Hazan et al., 2007] (Algorithm 1), which has been studied in the context of coin-betting by Cutkosky and Orabona [2018]. If a constant betting fraction is used throughout: λ t = λ , ∀ t , then

To illustrate the tightness of our results, we consider the optimal constant betting fractions which maximize the log-wealth (12) and are constrained to lie in [ -0 . 5 , 0 . 5] , like ONS bets:

Algorithm 1 Online Newton step (ONS) strategy for selecting betting fractions

.

Assuming that the 2ST null is false and a predictor with non-trivial prediction risk is used, it is easy to see that the optimal constant betting fraction has to be positive, which is why in Algorithm 1 we truncate λ ONS t at 0 instead of -1 / 2 (the latter option is used in [Cutkosky and Orabona, 2018]). We note that this is allowed and does not affect any theoretical claims of the paper. In early simulations, we also did not observe any major differences between the two truncating options. We conclude with the following result for the oracle tests, whose proof is deferred to Appendix D.3.

Theorem 1. The following claims hold:

Suppose that H 0 in (1a) is true. Then the oracle sequential test based on either ( K m ,⋆ t ) t ≥ 0 or ( K s ,⋆ t ) t ≥ 0 ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (1b) is true. Then:

(a) The growth rate of the oracle wealth process ( K m ,⋆ t ) t ≥ 0 satisfies:

If R m ( g ⋆ ) < 1 / 2 , then the test based on ( K m ,⋆ t ) t ≥ 0 is consistent: P H 1 ( τ < ∞ ) = 1 . Further, the optimal growth rate achieved by λ m ⋆ in (13) satisfies:

(b) The growth rate of the oracle wealth process ( K s ,⋆ t ) t ≥ 0 satisfies:

If E [ W · g ⋆ ( Z )] > 0 , then the test based on ( K s ,⋆ t ) t ≥ 0 is consistent: P H 1 ( τ < ∞ ) = 1 . Further, the optimal growth rate achieved by λ s ⋆ in (13) satisfies:

Theorem 1 precisely characterizes the properties of the oracle wealth processes and relates those to interpretable metrics of predictive performance. Further, the proof of Theorem 1 highlights a direct impact of the variance of the payoffs on the wealth growth rate, and hence the power of the resulting sequential tests (as the null is rejected once the wealth exceeds 1 /α ).

The second moment of the payoffs based on the misclassification risk (11a) is equal to one, resulting in a slow growth: the bound (14) is proportional to squared deviation of the misclassification risk from one half. The bound (15) shows that the growth rate with the ONS strategy matches, up to constants, that of the oracle betting fraction. Note that the second term in (15) characterizes the growth rate if R m ( g ⋆ ) < 5 / 16 (low Bayes risk). In this regime, the growth rate of our test is at least (3 / 16) · (1 / 2 -R m ( g ⋆ )) which is close to the optimal rate. The second moment of the payoffs based on the squared risk is more insightful. First, we present a result for the case when the oracle predictor g ⋆ in (11b) is replaced by an arbitrary g ∈ G . The proof is deferred to Appendix D.3.

Corollary 1. Consider an arbitrary g ∈ G with nonnegative expected margin: E [ W · g ( Z )] ≥ 0 . Then the growth rate of the corresponding wealth process ( K s t ) t ≥ 0 satisfies:

and the optimal growth rate achieved by λ s ⋆ in (13) satisfies:

Corollary 1 states that for an arbitrary g ∈ G , the growth rate is lower bounded by the minimum of the expected margin and the (optimized) squared risk of such predictor. While the latter term is always smaller for the optimal g ⋆ , this may not hold for an arbitrary g ∈ G . The lower bound (18b), which follows from Proposition 1, is always worse than that for g ⋆ (the expected margin is squared). The upper bound (19) shows that the growth rate with the ONS strategy matches, up to constants, that of the optimal constant betting fraction. Before presenting a practical sequential 2ST, we provide two important remarks that further contextualize the current work in the literature.

Remark 2 . In practice, we learn a predictor sequentially and have to choose a learning algorithm. Note that (18a) suggests that direct margin maximization may hurt the power of the resulting 2ST: the squared risk is sensitive to miscalibrated and overconfident predictors. Kübler et al. [2022] made a similar conjecture in the context of batch two-sample testing. To optimize the power, the authors suggested minimizing the cross-entropy or the squared loss and related such approach to maximizing the signal-to-noise ratio, a heuristic approach that was proposed earlier by Sutherland et al. [2017] 3 . Remark 3 . Suppose that g Bayes ∈ G and consider the payoff function based on the squared risk (11b). At round t , the wealth of a player K t -1 is multiplied by

and hence, the betting fractions interpolate between the regimes of not betting and betting using a likelihood ratio. From this standpoint, 2STs of Lhéritier and Cazals [2018, 2019], Pandeva et al. [2022] set λ t = 1 , ∀ t , and use only the second term for updating the wealth despite the fact that the true likelihood ratio is unknown. An argument about the consistency of such test hence requires imposing strong assumptions about a sequence of predictors ( g t ) t ≥ 1 [Lhéritier and Cazals, 2018, 2019]. Our test differs in a critical way: we use a sequence of betting fractions, ( λ t ) t ≥ 1 , which adapts to the quality of the underlying predictors, yielding a consistent test under much weaker assumptions.

Example 1. Consider P = N (0 , 1) and Q = N ( δ, 1) for 20 values of δ , equally spaced in [0 , 0 . 5] . For a given δ , the Bayes-optimal predictor is

where φ ( z ; µ, σ 2 ) denotes the density of N ( µ, σ 2 ) evaluated at z . In Figure 1a, we compare tests that use (a) the Bayes-optimal predictor, (b) a predictor constructed with the plug-in estimates of the means and variances. While in the former case betting using a likelihood ratio ( λ t = 1 , ∀ t ) is indeed optimal, our test with an adaptive sequence ( λ t ) t ≥ 1 is superior when a predictor is learned. The difference becomes even more drastic in Figure 1b where a (regularized) k -NN predictor is used.

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

δ

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

T

= 5000

Oracle predictor:

λ

t

= 1

,

∀

t

Our test

Learned predictor:

λ

t

= 1

,

∀

t

Our test

(a) Parametric model.

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

δ

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

T

= 5000

λ

t

= 1

,

∀

t

Our Test

α

= 0

.

05

(b) Nonparametric model (

k

-NN).

Sequential Classification-based 2ST (Seq-C-2ST). Let A c : ( ∪ t ≥ 1 ( Z × {1 , +1 } ) t ) ×G → G denote a learning algorithm which maps a training dataset of any size and previously used classifier, to an updated predictor. For example, A c may apply a single gradient descent step using the most recent observation to update a model. We start with D 0 = ∅ and g 1 ∈ G : g 1 ( z ) = 0 , for any z ∈ Z . At round t , we use one of the payoffs:

After ( Z t , W t ) is used for betting, we update a training dataset: D t = D t -1 ∪ { ( Z t , W t ) } , and an existing predictor: g t +1 = A c ( D t , g t ) . We summarize our sequential classification-based 2ST (SeqC-2ST) in Algorithm 2. We note that using pre-trained models (e.g., for image data) as initialization may definitely improve the performance during the early stages of testing in practice.

Algorithm 2 Sequential classification-based 2ST (Seq-C-2ST)

While we do not need any assumptions to confirm the type I error control, we place some mild assumptions on the learning algorithm A c to argue about the consistency.

Assumption 1 ( R m -learnability) . Suppose that H 1 in (1b) is true. An algorithm A c is such that the resulting sequence ( g t ) t ≥ 1 satisfies: lim sup t →∞ 1 t ∑ t i =1 { W i · sign [ g i ( Z i )] < 0 } a . s . < 1 / 2 .

✶ Assumption 2 ( R s -learnability) . Suppose that H 1 in (1b) is true. An algorithm A c is such that the resulting sequence ( g t ) t ≥ 1 satisfies: lim sup t →∞ 1 t ∑ t i =1 ( g i ( Z i ) -W i ) 2 a . s < 1 .

In words, the above assumptions state that a sequence of predictors ( g t ) t ≥ 1 is better than a chance predictor on average. We conclude with the following result, whose proof is deferred to Appendix D.3.

Theorem 2. The following claims hold for Seq-C-2ST (Algorithm 2):

If H 0 in (1a) is true, the test ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (1b) is true. Then:

(a) Under Assumption 1, the test with the payoff (22a) is consistent: P H 1 ( τ < ∞ ) = 1 .

Remark 4 . Suppose that we perform two-sample testing for d -dimensional data. At time T , the total accumulated computation for the kernelized test of Shekhar and Ramdas [2023] is O ( dT 2 ) . For our test, the answer depends on the chosen classifier and learning algorithm. Using logistic regression in combination with gradient descent for updating model parameters results in cheap updates and payoff evaluation (both are O ( d ) at each round, and hence the total accumulated computation at time T is O ( dT ) ). For k -NN classifier, no parameters have to be updated, yet evaluating payoffs becomes more expensive with a growing sample size, resulting in the total accumulated computation of O ( kdT 2 ) at time T . For more complex models like neural nets, runtime depends on the chosen architecture: the total accumulated computation at time T is O (( cB + F ) T ) , where F and B are the costs of forward-propagation and back-propagation steps respectively and c is the number of back-propagation steps applied after processing the next point (the exact cost depends on the architecture).

(b) Under Assumption 2, the test with the payoff (22b) is consistent: P H 1 ( τ < ∞ ) = 1 .

Sequential Classification-based Independence Test (Seq-C-IT). Under the setting of Definition 2, a single point from P XY is revealed at each round. Following [Podkopaev et al., 2023], we bet on two points from P XY (labeled as +1 ) and utilize external randomization to produce instances from P X × P Y (labeled as -1 ). Let A IT c : ( ∪ t ≥ 1 (( X × Y ) ×{-1 , +1 } ) t ) ×G → G denote a learning algorithm which maps a training dataset of any size and previously used classifier, to an updated predictor. We start with D 0 = ∅ and g 1 : g 1 ( x, y ) = 0 , ∀ ( x, y ) ∈ X × Y . We use derandomized versions of the payoffs (22), e.g., instead of (22b), we use

After ( X 2 t -1 , Y 2 t -1 ) , ( X 2 t , Y 2 t ) have been used for betting, we update a training dataset:

and an existing predictor: g t +1 = A IT c ( D t , g t ) . Seq-C-IT inherits the time-uniform type I error control and the consistency guarantees of Theorem 2, and we omit details for brevity.

3 Experiments

Synthetic Experiments. In our evaluation, we first consider synthetic datasets where the complexity of the independence testing setup is characterized by a single univariate parameter. We set the monitoring horizon to T = 5000 points from P XY , and for each parameter value, we aggregate the results over 200 runs. In particular, we use the following synthetic settings:

Spherical model. Let ( U t ) t ≥ 1 be a sequence of random vectors on a unit sphere in R d : U t iid ∼ Unif( S d ) , and let u ( i ) denote the i -th coordinate of u . For t ≥ 1 , we take

We consider d ∈ { 3 , . . . , 10 } , where larger d defines a harder setup.

Hard-to-detect-dependence (HTDD) model. We sample (( X t , Y t )) t ≥ 1 from

p ( x, y ) = 1 4 π 2 (1 + sin( wx ) sin( wy )) · ✶ { ( x, y ) ∈ [ -π, π ] 2 } . (24) We consider w ∈ { 0 , . . . , 6 } , where H 0 is true (random variables are independent) if and only if w = 0 . For w > 0 , Corr ( X,Y ) ≈ 1 /w 2 , and the setup is harder for larger w .

For the comparison, we use two predictive models to construct Seq-C-ITs:

Let N t ( z ) := N ( z, D t -1 , k t ) define the set of k t closest points in D t -1 to a query point z := ( x, y ) . We consider a regularized k -NN predictor: ˆ g t ( z ) = 1 k t +1 ∑ ( Z,W ) ∈N t ( z ) W. We select the number of neighbors using the square-root rule: k t = √ |D t -1 | = √ 4( t -1) .

We use a multilayer perceptron (MLP) with three hidden layers and 128, 64 and 32 neurons respectively and the parameters learned using an incremental training scheme.

We use the HSIC-based sequential kernelized independence test (SKIT) [Podkopaev et al., 2023] as a reference test and defer details, such as MLP training scheme and SKIT hyperparameters, to Appendix E.1. In Figure 2, we observe that SKIT outperforms Seq-C-ITs under the spherical model (with no localized dependence structure), whereas, under the structured HTDD model, Seq-C-ITs, is superior. Further, inspecting Figure 2b at w = 0 confirms that all tests control the type I error. We refer the reader to Appendix E.2 for additional experiments on synthetic data with localized dependence where Seq-C-ITs are superior. In Appendix E.2, we also provide the results for the average stopping times of our tests: we empirically confirm that our tests are adaptive to the complexity of a problem at hand: they stop earlier on easy tasks and later on harder ones.

3

4

5

6

7

8

9

10

d

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Power at Time

T

= 5000

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

(a) Spherical model.

0

1

2

3

4

5

6

w

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

T

= 5000

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

α

= 0

.

05

(b) HTDD model.

Real Data Experiments. First, we compare sequential classification-based and kernelized 2STs using Karolinska Directed Emotional Faces dataset (KDEF) [Lundqvist et al., 1998] which contains images of actors and actresses expressing different emotions: afraid (AF), angry (AN), disgusted (DI), happy (HA), neutral (HE), sad (SA), and surprised (SU). Following earlier works [Lopez-Paz and Oquab, 2017, Jitkrittum et al., 2016], we focus on straight profile only and assign HA, NE, SU emotions to the positive class (instances from P ), and AF, AN, DI emotions to the negative class (instances from Q ); see Figure 3a. We remove corrupted images and obtain a dataset containing 802 images with six different emotions. The original images ( 562 × 762 pixels) are cropped to exclude the background, resized to 64 × 64 pixels and converted to grayscale.

For Seq-C-2ST, we use a small CNN as an underlying model and defer details about the architecture and training to Appendix E.1. As a reference kernel-based 2ST, we use the sequential MMD test of Shekhar and Ramdas [2023] and adapt it to the setting where at each round either an observation from P or that from Q is revealed; see Appendix E.1 for details. We omit the comparison to Lhéritier and Cazals [2019] since their test has been shown to be inferior to the one developed by Shekhar and Ramdas [2023]. In Figure 3b, we illustrate that while both tests achieve perfect power after processing sufficiently many observations, our Seq-C-2ST requires fewer observations to do so.

(a)

0

100

200

300

400

500

600

700

800

Sample size

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Rejection Rate

Seq-C-2ST

Kernel 2ST

(b)

香

Next, we compare two independence tests using MNIST image dataset [LeCun et al., 1998]. To simulate the null setting, we sample pairs of random images from the entire dataset, and to simulate the alternative, we sample pairs of random images depicting the same digit (Figure 4a). For Seq-C-IT, we use MLP with the same architecture as for simulations on synthetic data. For SKIT, we use the median heuristic with 20 points from P XY to compute kernel hyperparameters. In Figure 4b, we show that while both tests control the type I error under H 0 , SKIT is inferior to Seq-C-IT under H 1 , requiring twice as much data to achieve perfect power.

(a)

0

100

200

300

400

500

Number of Pairs

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Rejection Rate

Seq-C-IT:

H

1

is true

H

0

is true

SKIT:

H

1

is true

H

0

is true

(b)

33

3

4 Conclusion

While kernel methods are state-of-the-art for nonparametric two-sample and independence testing, their performance often deteriorates on complex data, e.g., high-dimensional data with localized dependence. In such settings, prediction-based tests are often much more effective. In this work, we developed sequential predictive two-sample and independence tests following the principle of testing by betting. Our tests control the type I error despite continuously monitoring the data and are consistent under weak and tractable assumptions. Further, our tests provably adapt to the complexity of a problem at hand: they stop earlier on easy tasks and later on harder ones. An additional advantage of our tests is that an analyst may modify the design choices, e.g., model architecture, on-the-fly. Through experiments on synthetic and real data, we confirm that our tests are competitive to kernel-based ones overall and outperform those under structured settings.

We refer the reader to the Appendix for additional results that were not included in the main paper:

In Appendix A, we complement classification-based ITs with a regression-based approach. Regression-based ITs represent an alternative to the classification-based approach in settings where a data stream (( X t , Y t )) t ≥ 1 may be processed directly as feature-response pairs.

In Section 2, we considered the case of balanced classes, meaning that at each round, an instance from either P or Q is observed with equal chance. In Appendix B, we extend the methodology to a more general case of two-sample testing with unknown class proportions.

Batch two-sample and independence tests rely on either a cutoff computed using the asymptotic null distribution of a chosen test statistic (when it is tractable) or a permutation p-value, and if the distribution drifts, both approaches fail to provide the type I error control. In contrast, Seq-C-2ST and Seq-C-IT remain valid beyond the i.i.d. setting by construction (analogous to tests developed by Shekhar and Ramdas [2023], Podkopaev et al. [2023]), and we refer the reader to Appendix C for more details.

Acknowledgements. The authors thank Ian Waudby-Smith and Tudor Manole for fruitful discussions. AR acknowledges support from NSF grants IIS-2229881 and DMS-2310718.

References

Thomas B. Berrett and Richard J. Samworth. Nonparametric independence testing via mutual information. Biometrika , 2019.

Xiuyuan Cheng and Alexander Cloninger. Classification logit two-sample testing by neural networks for differentiating near manifold densities. IEEE Transactions on Information Theory , 2022.

Ashok Cutkosky and Francesco Orabona. Black-box reductions for parameter-free online learning in banach spaces. In Conference on Learning Theory , 2018.

Donald A. Darling and Herbert E. Robbins. Some nonparametric sequential tests with power one. Proceedings of the National Academy of Sciences , 1968.

Jerome H. Friedman. On multivariate goodness-of-fit and two-sample testing. Technical report, Stanford University, 2004.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In Algorithmic Learning Theory , 2005.

Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. Journal of Machine Learning Research , 2012.

Peter Grünwald, Alexander Henzi, and Tyron Lardy. Anytime-valid tests of conditional independence under model-X. Journal of the American Statistical Associatio , 2023.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 2007.

Simon Hediger, Loris Michel, and Jeffrey Näf. On the use of random forest for two-sample testing. Computational Statistics & Data Analysis , 2022.

Wittawat Jitkrittum, Zoltán Szabó, Kacper P Chwialkowski, and Arthur Gretton. Interpretable distribution features with maximum testing power. In Advances in Neural Information Processing Systems , 2016.

Ilmun Kim and Aaditya Ramdas. Dimension-agnostic inference using cross U-statistics. Bernoulli , 2020.

Ilmun Kim, Ann B. Lee, and Jing Lei. Global and local two-sample tests via regression. Electronic Journal of Statistics , 2019.

Ilmun Kim, Aaditya Ramdas, Aarti Singh, and Larry Wasserman. Classification accuracy as a proxy for two-sample testing. The Annals of Statistics , 2021.

Jonas M. Kübler, Vincent Stimper, Simon Buchholz, Krikamol Muandet, and Bernhard Schölkopf. AutoML two-sample test. In Advances in Neural Information Processing Systems , 2022.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 1998.

Alix Lhéritier and Frédéric Cazals. A sequential non-parametric multivariate two-sample test. IEEE Transactions on Information Theory , 2018.

Alix Lhéritier and Frédéric Cazals. Low-complexity nonparametric bayesian online prediction with universal guarantees. In Advances in Neural Information Processing Systems , 2019.

Feng Liu, Wenkai Xu, Jie Lu, Guangquan Zhang, Arthur Gretton, and Danica J. Sutherland. Learning deep kernels for non-parametric two-sample tests. In International Conference on Machine Learning , 2020.

David Lopez-Paz and Maxime Oquab. Revisiting classifier two-sample tests. In International Conference on Learning Representations , 2017.

Daniel Lundqvist, Anders Flykt, and Arne Öhman. The Karolinska Directed Emotional Faces KDEF. Technical report, Karolinska Institutet, 1998.

Teodora Pandeva, Tim Bakker, Christian A Naesseth, and Patrick Forré. E-valuating classifier two-sample tests. arXiv preprint: 2210.13027 , 2022.

Aleksandr Podkopaev, Patrick Blöbaum, Shiva Prasad Kasiviswanathan, and Aaditya Ramdas. Sequential kernelized independence testing. In International Conference on Machine Learning , 2023.

Aaditya Ramdas and Tudor Manole. Randomized and exchangeable improvements of Markov's, Chebyshev's and Chernoff's inequalities. arXiv preprint: 2304.02611 , 2023.

Aaditya Ramdas, Sashank Jakkam Reddi, Barnabas Poczos, Aarti Singh, and Larry Wasserman. On the decreasing power of kernel and distance based nonparametric hypothesis tests in high dimensions. AAAI Conference on Artificial Intelligence , 2015.

Aaditya Ramdas, Johannes Ruf, Martin Larsson, and Wouter Koolen. Admissible anytime-valid sequential inference must rely on nonnegative martingales. arXiv preprint: 2009.03167 , 2020.

Shalev Shaer, Gal Maman, and Yaniv Romano. Model-free sequential testing for conditional independence via testing by betting. In Conference on Artificial Intelligence and Statistics , 2023.

Glenn Shafer. Testing by betting: a strategy for statistical and scientific communication. Journal of the Royal Statistical Society: Series A (Statistics in Society) , 2021.

Glenn Shafer and Vladimir Vovk. Game-Theoretic Foundations for Probability and Finance . Wiley Series in Probability and Statistics. Wiley, 2019.

Shubhanshu Shekhar and Aaditya Ramdas. Nonparametric two-sample testing by betting. IEEE Transactions on Information Theory , 2023.

Danica J. Sutherland, Hsiao-Yu Tung, Heiko Strathmann, Soumyajit De, Aaditya Ramdas, Alex Smola, and Arthur Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In International Conference on Learning Representations , 2017.

Jean Ville. Étude critique de la notion de collectif . Thèses de l'entre-deux-guerres. Gauthier-Villars, 1939.

Larry Wasserman, Aaditya Ramdas, and Sivaraman Balakrishnan. Universal inference. Proceedings of the National Academy of Sciences , 2020.

Appendix

A Regression-based Independence Testing

Regression-based independence tests represent an alternative to classification-based approaches in settings where a data stream (( X t , Y t )) t ≥ 1 may be processed directly as feature-response pairs. Suppose that one selects a functional class G : X → Y for performing such prediction task, and let ℓ denote a loss function that evaluates the quality of predictions. For example, if ( Y t ) t ≥ 1 is a sequence of univariate random variables, one can use the squared loss: ℓ ( g ( x ) , y ) = ( g ( x ) -y ) 2 , or the absolute loss: ℓ ( g ( x ) , y ) = | g ( x ) -y | .

Such tests rely on the following idea: if the alternative H 1 in (2b) is true and a sequence of sequentially updated predictors ( g t ) t ≥ 1 has nontrivial predictive power, then the losses on random instances drawn from the joint distribution P XY are expected to be less on average than the losses on random instances from P X × P Y . For the t -th pair of points from P XY , we can label the losses of g t on all possible ( X,Y ) -pairs as

One can view this problem as sequential two-sample testing under distribution drift (due to incremental learning of ( g t ) t ≥ 1 ). Hence, one may use either Seq-C-2ST from Section 2 or sequential kernelized 2ST of Shekhar and Ramdas [2023] on the resulting sequence of the losses on observations from P XY and P X × P Y . In what follows, we analyze a direct approach where testing is performed by comparing the losses on instances drawn from the two distributions. A critical difference with a construction of Seq-C-2ST is that to design a valid betting strategy one has to ensure that the payoff functions are lower bounded by negative one.

A.1 Proxy Regression-based Independence Test

To avoid cases when some expected values are not well-defined, we assume for simplicity that X is a bounded subset of R d for som d ≥ 1 : X = { x ∈ R d : ∥ x ∥ 2 ≤ B 1 } for some B 1 > 0 . Similarly, we assume that Y is a bounded subset of R : Y = { y ∈ R : | y | ≤ B 2 } for some B 2 > 0 . We note that the construction of the regression-based IT will not require explicit knowledge of constants B 1 and B 2 . First, we consider a setting where an instance either from the joint distribution or an instance from the product of the marginal distributions is observed at each round.

Definition 3 (Proxy Setting) . Suppose that we observe a stream of i.i.d. observations (( X t , Y t , W t )) t ≥ 1 , where W t ∼ Rademacher(1 / 2) , the distribution of ( X t , Y t ) | W t = +1 is P X × P Y , and that of ( X t , Y t ) | W t = -1 is P XY . The goal is to design a test for the following pair of hypotheses:

̸

Oracle Proxy Sequential Regression-based IT. To construct an oracle test, we assume having access to the oracle predictor g ⋆ : X → Y , e.g., the minimizer of the squared risk is g ⋆ ( x ) = E [ Y | X = x ] . Formalizing the above intuition, we use E [ Wℓ ( g ⋆ ( X ) , Y )] as a natural way for measuring dependence between X and Y . To enforce boundedness of the payoff functions, we use ideas of the tests for symmetry from [Ramdas et al., 2020, Shekhar and Ramdas, 2023, Podkopaev et al., 2023, Shaer et al., 2023], namely we use a composition with an odd function:

where s ⋆ > 0 is an appropriately selected scaling factor 4 . Since under H 0 in (26a), s ⋆ · W t · ℓ ( g ⋆ ( X t ) , Y t ) is a random variable that is symmetric around zero, it follows that E [ f r ⋆ ( X t , Y t , W t )] =

0 , and, using the argument analogous to the proof of Theorem 1, we can easily deduce that a sequential IT based on f r ⋆ controls the type I error control. The scaling factor s ⋆ is selected in a way that guarantees that, if H 1 in (26b) is true and if E [ Wℓ ( g ⋆ ( X ) , Y )] > 0 , then E [ f r ⋆ ( X,Y,W )] > 0 , which is a sufficient condition for consistency of the oracle test. In particular, we show that it suffices to consider:

Without loss of generality, we assume that ν ⋆ is bounded away from zero (which is a very mild assumption since ν ⋆ essentially corresponds to a cubic risk of g ⋆ on data drawn from the product of the marginal distributions P X × P Y ). Let the oracle regression-based wealth process ( K r ,⋆ t ) t ≥ 0 be defined by using the payoff function (27) with a scaling factor defined in (28a), along with a predictable sequence of betting fractions ( λ t ) t ≥ 1 selected via the ONS strategy (Algorithm 1). We have the following result about the oracle regression-based IT, whose proof is deferred to Appendix D.4.

Theorem 3. The following claims hold for the oracle sequential regression-based IT based on ( K r ,⋆ t ) t ≥ 0 :

Suppose that H 0 in (26a) is true. Then the test ever stops with probability at most α : P H 1 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (26b) is true. Further, suppose that: E [ Wℓ ( g ⋆ ( X ) , Y )] > 0 . Then the test is consistent: P H 1 ( τ < ∞ ) = 1 .

Practical Proxy Sequential Regression-based IT. To construct a practical test, we use a sequence of predictors ( g t ) t ≥ 1 that are updated sequentially as more data are observed. We write A r : ( ∪ t ≥ 1 ( X × Y ) t ) ×G → G to denote a chosen regressor learning algorithm which maps a training dataset of any size and previously used predictor, to an updated predictor. We start with D 0 = ∅ and some initial guess g 1 ∈ G . At round t , we use the payoff function:

where a sequence of predictable scaling factors ( s t ) t ≥ 1 is defined as follows: we set s 0 = 0 and define:

After ( X t , Y t , W t ) has been used for betting, we update a training dataset: D t = D t -1 ∪ { ( X t , Y t , W t ) } , and an existing predictor: g t +1 = A r ( D t , g t ) . We summarize this practical sequential 2ST in Algorithm 3.

For simplicity, we consider a class of functions G := { g θ : X → Y , θ ∈ Θ } for some parameter set Θ which we assume to be a subset of a metric space. In this case, a sequence of predictors ( g t ) t ≥ 1 is associated with the corresponding sequence of parameters ( θ t ) t ≥ 1 : for t ≥ 1 , g t ( · ) = g ( · ; θ t ) for some θ t ∈ Θ . To argue about the consistency of the resulting test, we make two assumptions.

Assumption 3 (Smoothness) . We assume that:

Predictors in G are L 1 -Lipschitz smooth:

Algorithm 3 Proxy Sequential Regression-based IT

The loss function ℓ is L 2 -Lipschitz smooth:

In words, Assumption (31) states that the outputs of predictors, whose parameters are close, will also be close. Assumption (32) states that that the losses of two predictors, whose outputs are close, will also be close. For example, if G is a class of linear predictors: g θ ( x ) = θ ⊤ x , x ∈ X , then Assumption 3 will be trivially satisfied for the squared and the absolute losses if X and Y are bounded. Note that we do not need an explicit knowledge of L 1 or L 2 for designing a test. Second, we make a learnability assumption about algorithm A r .

Assumption 4 (Learnability) . Suppose that H 1 in (26b) is true. We assume that the regressor learning algorithm A r is such that for the resulting sequence of parameters ( θ t ) t ≥ 1 , it holds that θ t a . s . → θ ⋆ , where θ ⋆ is a random variable taking values in Θ and E [ Wℓ ( g ( X ; θ ⋆ ) , Y ) | θ ⋆ ] a . s . > 0 , where ( X,Y,W ) ⊥ ⊥ θ ⋆ .

We conclude with the following result for the practical proxy sequential regression-based IT, whose proof is deferred to Appendix D.4.

Theorem 4. The following claims hold for the proxy sequential regression-based IT (Algorithm 3):

Suppose that H 0 in (26a) is true. Then the test ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (26b) is true. Further, suppose that Assumptions 3 and 4 are satisfied. Then the test is consistent: P H 1 ( τ < ∞ ) = 1 .

Sequential Regression-based Independence Test (Seq-R-IT). Next, we instantiate this test for the sequential independence testing setting (as per Definition 2) where we observe sequence (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) iid ∼ P XY , t ≥ 1 . Analogous to Section 2, we bet on the outcome of two observations drawn from the joint distribution P XY . To proceed, we derandomize the payoff function (29) and consider

After betting on the outcome of the t -th pair of observations from P XY , we update a training dataset:

and a predictive model: ˆ g t +1 = A r ( D t , ˆ g t ) .

A.2 Synthetic Experiments

To evaluate the performance of Seq-R-IT, we consider the Gaussian linear model . Let ( X t ) t ≥ 1 and ( ε t ) t ≥ 1 denote two independent sequences of i.i.d. standard Gaussian random variables. For t ≥ 1 , we take

where β = 0 implies nonzero linear correlation (hence dependence). We consider 20 values of β equally spaced in [0 , 1 / 2] . For the comparison, we use:

̸

Seq-R-IT with ridge regression. We use ridge regression as an underlying model: ˆ g t ( x ) = β ( t ) 0 + xβ ( t ) 1 , where

Seq-C-IT with QDA. Note that P XY = N ( µ, Σ + ) and P X × P Y = N ( µ, Σ -) , where

For this problem, an oracle predictor which minimizes the misclassification risk is

where φ (( x, y ); µ, Σ) denotes the density of the Gaussian distribution N ( µ, Σ) evaluated at ( x, y ) . Recall that D t -1 = { ( Z i , +1) } i ≤ 2( t -1) ∪ { ( Z ′ i , -1) } i ≤ 2( t -1) denotes the training dataset that is available at round t for training a predictor ˆ g t : X × Y → [ -1 , 1] . We deploy Seq-C-IT with an estimator ˆ g t of (34), obtained by using plug-in estimates of µ + , Σ + , µ -, Σ -, computed from D t -1 :

and ˆ µ -t , ˆ Σ -t are computed similarly from D -t .

In addition, we also include HSIC-based SKIT to the comparison and defer the details regarding kernel hyperparameters to Appendix E.1. We set the monitoring horizon to T = 5000 points from P XY and aggregate the results over 200 sequences of observations for each value of β . We illustrate the result in Figure 5: while Seq-R-IT has high power for large values of β , we observe its inferior performance against Seq-C-IT (and SKIT) under the harder settings. Improving regression-based betting strategies, e.g., designing better scaling factors that still yield a provably consistent test, is an open question for future research.

B Two-sample Testing with Unbalanced Classes

In Section 2, we developed a sequential 2ST under the assumption at each round, an instance from either P or Q is revealed with equal probability. Such assumption was reasonable for designing Seq-C-IT, where external randomization produced two instances from P XY and P X × P Y at each round. Next, we generalize our sequential 2ST to a more general setting of unbalanced classes.

Definition 4 (Sequential two-sample testing with unbalanced classes) . Let π ∈ (0 , 1) . Suppose that we observe a stream of i.i.d. observations (( Z t , W t )) t ≥ 1 , where W t ∼ Rademacher( π ) , the distribution of Z t | W t = +1 is denoted P , and that of Z t | W t = -1 is denoted Q . We set the goal of designing a sequential test for the following pair of hypotheses:

̸

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

β

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

T

= 5000

Seq-C-IT

Seq-R-IT

SKIT

α

= 0

.

05

(a)

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

β

0

1000

2000

3000

4000

5000

Average Stopping Time

Seq-C-IT

Seq-R-IT

SKIT

(b)

For what follows, we will focus on the payoff based on the squared risk due to its relationship to the likelihood-ratio-based test (Remark 3). In particular, after correcting the likelihood under the null in (20) to account for a general positive class proportion π , we can deduce that (see Appendix D.5):

✶ ✶ where η t ( z ) = ( g t ( z ) + 1) / 2 , and hence, a natural payoff function for the case with unbalanced classes is

Note that the payoff for the balanced case (22b) is recovered by setting π = 1 / 2 . It is easy to check that (see Appendix D.5): (a) f u t ( z, w ) ≥ -1 for any ( z, w ) ∈ Z × {1 , 1 } , and (b) if H 0 in (35a) is true, then E H 0 [ f u t ( Z t , W t ) | F t -1 ] = 0 , where F t -1 = σ ( { ( Z i , W i ) } i ≤ t -1 ) . This in turn implies that a wealth process that relies on the payoff function f u t in (37) is a nonnegative martingale, and hence, the corresponding sequential 2ST is valid. However, the positive class proportion π , needed to use the payoff function (37), is generally unknown beforehand. First, let us consider the case when λ t = 1 , t ≥ 1 . In this case, the wealth of a gambler that uses the payoff function (37) after round t is

Note that:

is the MLE for π computed from { W i } i ≤ t . In particular, if we consider a process ( ˜ K t ) t ≥ 0 , where

✶ ✶ it follows that ˜ K t ≤ K t , ∀ t ≥ 1 , meaning that ( ˜ K t ) t ≥ 0 is a process that is upper bounded by a nonnegative martingale with initial value one. This in turn implies that a test based on ( ˜ K t ) t ≥ 0 is a valid levelα sequential 2ST for the case of unknown class proportions. This idea underlies the running MLE sequential likelihood ratio test of Wasserman et al. [2020] and has been recently considered in the context of two-sample testing by Pandeva et al. [2022]. In case of nontrivial betting fractions: ( λ t ) t ≥ 1 , representation of the wealth process (38) no longer holds, and to proceed, we modify the rules of the game and use minibatching. A bet is placed on every b (say, 5 or 10) observations, meaning

that for a given minibatch size b ≥ 1 , at round t we bet on { ( Z b ( t -1)+ i , W b ( t -1)+ i ) } i ∈{ 1 ,...,b } . The MLE of π computed from the t -th minibatch is

We consider a payoff function of the following form:

In words, the above payoff essentially compares the performance of a predictor g t , trained on { ( Z i , W i ) } i ≤ b ( t -1) and evaluated on the t -th minibatch, to that of a trivial baseline predictor to form a bet. In particular, setting b = 1 yields a valid, yet a powerless test. Indeed, we have ˆ π t = ✶ { W t = 1 } = ( W t +1) / 2 . In this case, the payoff (39) reduces to implying that the wealth can not grow even if the null is false. Define a wealth processes ( K u t ) t ≥ 0 based on the payoff functions (39) along with a predictable sequence of betting fractions ( λ t ) t ≥ 1 selected via ONS strategy (Algorithm 1). Let F t = σ ( { ( Z i , W i ) } i ≤ bt ) for t ≥ 1 , with F 0 denoting a trivial sigma-algebra. We conclude with the following result, whose proof is deferred to Appendix D.5.

Theorem 5. Suppose that H 0 in (35a) is true. Then ( K u t ) t ≥ 0 is a nonnegative supermartingale adapted to ( F t ) t ≥ 0 . Hence, the sequential 2ST based on ( K u t ) t ≥ 0 satisfies: P H 0 ( τ < ∞ ) ≤ α .

C Testing under Distribution Drift

First, we define the problem of two-sample testing when at each round instances from both distributions are observed.

Definition 5 (Sequential two-sample testing) . Suppose that we observe that a stream of observations: (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) iid ∼ P X × P Y for t ≥ 1 . The goal is to design a sequential test for

̸

Under the two-sample testing setting (Definition 5), we label observations from P Y as positive ( +1 ) and observations from P X as negative ( -1 ). We write A 2ST c : ( ∪ t ≥ 1 ( X × {1 , +1 } ) t ) ×G → G to denote a chosen learning algorithm which maps a training dataset of any size and previously used predictor, to an updated predictor. We start with D 0 = ∅ and g 1 : g 1 ( x ) = 0 , ∀ x ∈ X . At round t , we bet using derandomized versions of the payoffs (22), namely

After ( X t , Y t ) has been used for betting, we update a training dataset and an existing predictor:

Testing under Distribution Drift. Batch two-sample and independence tests generally rely on either a cutoff computed using the asymptotic null distribution of a chosen test statistic (if tractable) or a permutation p-value. Both approaches require imposing i.i.d. (or exchangeability, for the latter option) assumption about the data distribution, and if the distribution drifts, both approaches fail to guarantee the type I error control. In contrast, Seq-C-2ST and Seq-C-IT remain valid beyond the i.i.d. setting by construction (analogous to tests developed in [Shekhar and Ramdas, 2023, Podkopaev et al., 2023]). First, we define the problems of sequential two-sample and independence testing under distribution drift.

Definition 6 (Sequential two-sample testing under distribution drift) . Suppose that we observe that a stream of independent observations: (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) ∼ P ( t ) X × P ( t ) Y , t ≥ 1 . The goal is to design a sequential test for the following pair of hypotheses:

̸

Definition 7 (Sequential independence testing under distribution drift) . Suppose that we observe that a stream of independent observations from the joint distribution which drifts over time: (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) ∼ P ( t ) XY . The goal is to design a sequential test for the following pair of hypotheses:

̸

The superscripts highlight that, in contrast to the standard i.i.d. setting (Definitions 5 and 2), the underlying distributions may drift over time. For independence testing, we need to impose an additional assumption that enables reasoning about the type I error control of Seq-C-IT.

Assumption 5. Consider the setting of independence testing under distribution drift (Definition 7). We assume that for each t ≥ 1 , it holds that either P ( t -1) X = P ( t ) X or P ( t -1) Y = P ( t ) Y , meaning that at each step either the distribution of X changes or that of Y changes, but not both simultaneously 5 .

We have the following result about the type I error control of our tests under distribution drift. Corollary 2. The following claims hold:

Suppose that H 0 in (42a) is true. Then Seq-C-2ST satisfies: P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 0 in (43a) is true. Further, suppose that Assumption 5 is satisfied. Then Seq-C-IT satisfies: P H 0 ( τ < ∞ ) ≤ α .

The above result follows from the fact the payoff functions underlying Seq-C-2ST (41) and Seq-CIT (23) are valid under the more general null hypotheses (42a) and (43a) respectively. The rest of the proof of Corollary 2 follows the same steps as that of Theorem 2, and we omit the details. We conclude with an example which shows that Assumption 5 is necessary for the type I error control.

Example 2. Consider the following case when the null H 0 in (43a) is true, but Assumption 5 is not satisfied. We show that Seq-C-IT fails to control type I error (at any prespecified level α ∈ (0 , 1) ), and for simplicity, focus on the payoff function based on the squared risk (23). Suppose that we observe a sequence of observations: (( X t , Y t )) t ≥ 1 , where ( X t , Y t ) = ( t + W t , t + V t ) and W t , V t iid ∼ Bern(1 / 2) . It suffices to show that there exists a sequence of predictors ( g t ) t ≥ 1 , for which

If (44) holds, then using the same argument as in the proof of Theorem 2, one can then deduce that P ( τ < ∞ ) = 1 . Consider the following sequence of predictors ( g t ) t ≥ 1 :

We have:

Simple calculation shows that:

E [ g t ( X 2 t , Y 2 t )] = 11 / 16 , E [ g t ( X 2 t -1 , Y 2 t -1 )] = E [ g t ( X 2 t , Y 2 t -1 )] = E [ g t ( X 2 t -1 , Y 2 t )] = 0 and hence, for all t ≥ 1 , it holds that E [ f s t (( X 2 t -1 , Y 2 t -1 ) , ( X 2 t , Y 2 t ))] = 11 / 64 > 0 . This in turn implies (44), and hence, we conclude that Seq-C-IT fails to control the type I error.

C.1 Synthetic Experiments

Under the alternative, the power of our test depends on the performance of a classifier according to misclassification/squared risk. If the distribution drifts over time, then models updated via online gradient descent may retain predictive power. In such cases, our test will still have high power despite the present distribution drift. Of course, if the distribution shifts adversarially, the method will not have power, but neither will any other method. So the implicit goal is to maintain type-1 error control under the null despite shifting distributions (beyond the i.i.d. assumption typically made in the literature) while retaining power against reasonable (but not all) distribution drifts. To put this into context, we consider four settings:

P = Q = N (0 , 1) , i.e., the 2ST null is true;

Up to time t = 250 , P ( t ) = Q ( t ) = N (0 , 1) , but for t > 251 , we have P ( t ) = N (0 , 1) and Q ( t ) = N (0 . 5 , 1) , i.e., there is a distribution shift and the 2ST null is false;

P = N (0 , 1) and Q = N (0 . 5 , 1) , i.e., the 2ST null is false;

Up to time t = 250 , P ( t ) = Q ( t ) = N (0 , 1) , but for t > 251 , P ( t ) = Q ( t ) = N (0 . 5 , 1) , i.e., there is a change in distribution yet the 2ST null is still true.

Weuse a standard logistic regression model as an underlying predictor with weights updated via online gradient descent and monitor the tests until T = 2000 observations. In Figure 6, we observe that our test controls the type-1 error whenever the 2ST null is true even if there is a shift in distribution, and retains high power if the alternative is true.

0

250

500

750

1000

1250

1500

1750

2000

Sample size

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Rejection Rate

H

0

is true, no shift

H

0

is false, no shift

H

0

is true, changepoint

H

0

is false, changepoint

α

= 0

.

05

D Proofs

D.1 Auxiliary Results

Proposition 2 (Ville's inequality [Ville, 1939]) . Suppose that ( M t ) t ≥ 0 is a nonnegative supermartingale process adapted to a filtration ( F t ) t ≥ 0 . Then, for any a > 0 it holds that:

D.2 Supporting Lemmas

Lemma 6. Consider sequential two-sample testing setting (Definition 1). Suppose that a predictor g ∈ G satisfies E [ f ( Z, W )] > 0 , where f ( z, w ) := wg ( z ) .

(a) Consider the wealth process ( K t ) t ≥ 0 based on f along with the ONS strategy for selecting betting fractions (Algorithm 1). Then we have the following lower bound on the growth rate of the wealth process:

(b) For λ ⋆ = arg max λ ∈ [ -0 . 5 , 0 . 5] E [log(1 + λf ( Z, W ))] , it holds that:

Analogous result holds when the payoff function f ( z, w ) := w · sign [ g ( z )] is used instead.

Proof. (a) Under the ONS betting strategy, for any sequence of outcomes ( f t ) t ≥ 1 , f t ∈ [ -1 , 1] , it holds that (see the proof of Theorem 1 in [Cutkosky and Orabona, 2018]):

where K t ( λ 0 ) is the wealth of any constant betting strategy λ 0 ∈ [ -1 / 2 , 1 / 2] and K t is the wealth corresponding to the ONS betting strategy. Hence, it follows that

for some absolute constant C > 0 . Next, consider

We obtain:

where in (a) we used that log(1 + x ) ≥ x -x 2 for x ∈ [ -1 / 2 , 1 / 2] . From (48), it then follows that:

which completes the proof of the first assertion of the lemma.

(b) Since log(1 + x ) ≤ x -3 x 2 / 8 for any x ∈ [ -0 . 5 , 0 . 5] , we know that:

The optimizer of the above is

Hence, as long as E [ f ( Z, W )] ≤ (3 / 8) · E [ ( f ( Z, W )) 2 ] , we have:

If however, E [ f ( Z, W )] > (3 / 8) · E [ ( f ( Z, W )) 2 ] , then we know that:

To bring it to a convenient form, we multiply the upper bound in (50) by two and get the bound (46), which completes the proof of the second assertion of the lemma.

D.3 Proofs for Section 2

Proposition 1. Fix an arbitrary predictor g ∈ G . The following claims hold 6 :

For the misclassification risk, we have that:

For the squared risk, we have that:

Further, d s ( P, Q ) > 0 if and only if there exists g ∈ G such that E [ W · g ( Z )] > 0 .

Proof. 1. The first equality in (9) follows from two facts: (a) for any g ∈ G and any s ∈ (0 , 1] , it holds that R m ( sg ) = R m ( g ) , (b) R m (0) = 1 / 2 . The second equality easily follows from the following fact: sign [ x ] / 2 = 1 / 2 -{ x < 0 } .

(a) If E [ W · g ( Z )] ≤ 0 , then the RHS of (10) is zero. For the LHS of (10), we have that:

✶ 2. Consider an arbitrary predictor g ∈ G . Let us consider all possible scenarios:

so the bound (10) holds.

(b) Next, assume that E [ W · g ( Z )] > 0 , then it is easy to derive that:

A simple calculation shows that:

and hence, we conclude that the bound (10) holds.

To establish the second part of the statement, note that d s ( P, Q ) > 0 iff there is a predictor g ∈ G such that R s ( g ) < 1 . For the squared risk, we have:

and hence, R s ( g ) < 1 trivially implies that E [ W · g ( Z )] > 0 . The converse implication trivially follows from (10). Hence, the result follows.

Theorem 1. The following claims hold:

Suppose that H 0 in (1a) is true. Then the oracle sequential test based on either ( K m ,⋆ t ) t ≥ 0 or ( K s ,⋆ t ) t ≥ 0 ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (1b) is true. Then:

(a) The growth rate of the oracle wealth process ( K m ,⋆ t ) t ≥ 0 satisfies:

If R m ( g ⋆ ) < 1 / 2 , then the test based on ( K m ,⋆ t ) t ≥ 0 is consistent: P H 1 ( τ < ∞ ) = 1 . Further, the optimal growth rate achieved by λ m ⋆ in (13) satisfies:

(b) The growth rate of the oracle wealth process ( K s ,⋆ t ) t ≥ 0 satisfies:

If E [ W · g ⋆ ( Z )] > 0 , then the test based on ( K s ,⋆ t ) t ≥ 0 is consistent: P H 1 ( τ < ∞ ) = 1 . Further, the optimal growth rate achieved by λ s ⋆ in (13) satisfies:

Proof. 1. We trivially have that the payoff functions (11a) and (11b) are bounded: ∀ ( z, w ) ∈ Z×{1 , 1 } , it holds that f m ⋆ ( z, w ) ∈ [ -1 , 1] and f s ⋆ ( z, w ) ∈ [ -1 , 1] . Further, under the null H 0 in (1a), it trivially holds that E H 0 [ f m ⋆ ( Z t , W t ) | F t -1 ] = E H 0 [ f s ⋆ ( Z t , W t ) | F t -1 ] = 0 , where F t -1 = σ ( { ( Z i , W i ) } i ≤ t -1 ) . Since ONS betting fractions ( λ ONS t ) t ≥ 1 are predictable, we conclude that the resulting wealth process is a nonnegative martingale. The assertion of the Theorem then follows directly from Ville's inequality (Proposition 2) when a = 1 /α .

Suppose that H 1 in (1b) is true. First, we prove the results for the lower bounds:

(a) Consider the wealth process based on the misclassification risk ( K m ,⋆ t ) t ≥ 0 . Note that for all t ≥ 1 :

Since E [ f m ⋆ ( Z t , W t )] ∈ [0 , 1] , we also have ( E [ f m ⋆ ( Z t , W t )]) 2 ≤ E [ f m ⋆ ( Z t , W t )] . From the first part of Lemma 6, it follows that:

From the second part of Lemma 6, and (46) in particular, it follows that:

The first term in the above is smaller or equal than the second one whenever R m ( g ⋆ ) ≥ 5 / 16 . We conclude that the assertion of the theorem is true.

(b) Next, we consider the wealth process based on the squared error: ( K s ,⋆ t ) t ≥ 0 . Note that:

and hence from Lemma 6, it follows that:

In the above, we assume that the following case is not possible: g ⋆ ( Z ) a . s . = 0 (for such g ⋆ , the corresponding expected margin and the growth rate of the resulting wealth process are clearly zero, and will still be highlighted in our resulting bound). Next, note that since g ⋆ ∈ arg min g ∈G R s ( g ) , we have that:

meaning that g ⋆ can not be improved by scaling with s < 1 . From Proposition 1, and (51) in particular, it follows that:

and hence, the bound (53) reduces to

From the second part of Lemma 6, it follows that:

Next, we use that g ⋆ satisfies (54), which implies that the second term in (55) is smaller, and hence,

which concludes the proof of the second part of the theorem.

Corollary 1. Consider an arbitrary g ∈ G with nonnegative expected margin: E [ W · g ( Z )] ≥ 0 . Then the growth rate of the corresponding wealth process ( K s t ) t ≥ 0 satisfies:

and the optimal growth rate achieved by λ s ⋆ in (13) satisfies:

Proof. Following the same argument as that of the proof of Theorem 1, we can deduce that:

Hence, it suffices to argue that the lower bound (56) is equivalent to (18a). Without loss of generality, we can assume that E [ W · g ( Z )] ≥ 0 , and further, the two lower bounds are equal if E [ W · g ( Z )] = 0 . Hence, we consider the case when E [ W · g ( Z )] > 0 . First, let us consider the case when

Using (51), we get that:

and hence, two bounds coincide. For the upper bound (19), we use Lemma 6, and the upper bound (46) in particular. Note that the first term in (46) is less than the second term whenever

However, in this regime we also know that (58) holds, and hence the two bounds coincide. This completes the proof.

Theorem 2. The following claims hold for Seq-C-2ST (Algorithm 2):

If H 0 in (1a) is true, the test ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (1b) is true. Then:

(a) Under Assumption 1, the test with the payoff (22a) is consistent: P H 1 ( τ < ∞ ) = 1 .

(b) Under Assumption 2, the test with the payoff (22b) is consistent: P H 1 ( τ < ∞ ) = 1 .

Proof.

We trivially have that the payoff functions (22a) and (22b) are bounded: ∀ t ≥ 1 and ∀ ( z, w ) ∈ Z × {1 , 1 } , it holds that f m t ( z, w ) ∈ [ -1 , 1] and f s t ( z, w ) ∈ [ -1 , 1] . Further, under the null H 0 in (1a), it trivially holds that E H 0 [ f m t ( Z t , W t ) | F t -1 ] = E H 0 [ f s t ( Z t , W t ) | F t -1 ] = 0 , where F t -1 = σ ( { ( Z i , W i ) } i ≤ t -1 ) . Since ONS betting fractions ( λ ONS t ) t ≥ 1 are predictable, we conclude that the resulting wealth process is a nonnegative martingale. The assertion of the Theorem then follows directly from Ville's inequality (Proposition 2) when a = 1 /α .

Note that if ONS strategy for selecting betting fractions is deployed, then (49) implies that the tests will be consistent as long as

where for i ≥ 1 , f i = f m i ( Z i , W i ) and f i = f s i ( Z i , W i ) for the payoffs based on the misclassification and the squared risks respectively.

(a) Recall that

and Assumption 1 states that:

Since ✶ { x < 0 } = (1 -sign [ x ]) / 2 , we get that:

which, after rearranging and multiplying by two, implies that:

Hence, a sufficient condition for consistency (59) holds, and we conclude that the result is true.

(b) Recall that

and Assumption 2 states that:

which is equivalent to

It is easy to see that the above, in turn, implies that:

Hence, a sufficient condition for consistency (59) holds, and we conclude that the result is true.

D.4 Proofs for Appendix A

Theorem 3. The following claims hold for the oracle sequential regression-based IT based on ( K r ,⋆ t ) t ≥ 0 :

Suppose that H 0 in (26a) is true. Then the test ever stops with probability at most α : P H 1 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (26b) is true. Further, suppose that: E [ Wℓ ( g ⋆ ( X ) , Y )] > 0 . Then the test is consistent: P H 1 ( τ < ∞ ) = 1 .

Proof. 1. We trivially have that the payoff function (27) is bounded: ∀ ( x, y, w ) ∈ X × Y × {-1 , 1 } , it holds that f r ⋆ ( x, y, w ) ∈ [ -1 , 1] . Further, under the null H 0 in (26a), it trivially holds that E H 0 [ f r ⋆ ( X t , Y t , W t ) | F t -1 ] = 0 , where F t -1 = σ ( { ( X i , Y i , W i ) } i ≤ t -1 ) . Since ONS betting fractions ( λ ONS t ) t ≥ 1 are predictable, we conclude that the resulting wealth process is a nonnegative martingale. The assertion of the Theorem then follows directly from Ville's inequality (Proposition 2) when a = 1 /α .

Note that if ONS strategy for selecting betting fractions is deployed, then (49) implies that the tests will be consistent as long as

Note that:

Note that for any x ∈ R : tanh( x ) ≥ x -1 3 · max { x 3 , 0 } . Hence, for any s > 0 , it holds that:

where we used that max { W, 0 } = ( W +1) / 2 since W ∈ {-1 , 1 } . Maximizing the RHS of (61) over s > 0 yields s ⋆ defined in (28a). Hence,

Hence, we conclude that the oracle regression-based IT is consistent since the sufficient condition (62) holds.

Theorem 4. The following claims hold for the proxy sequential regression-based IT (Algorithm 3):

Suppose that H 0 in (26a) is true. Then the test ever stops with probability at most α : P H 0 ( τ < ∞ ) ≤ α .

Suppose that H 1 in (26b) is true. Further, suppose that Assumptions 3 and 4 are satisfied. Then the test is consistent: P H 1 ( τ < ∞ ) = 1 .

Proof. 1. We trivially have that the payoff function (29) is bounded: ∀ ( x, y, w ) ∈ X × Y × {-1 , 1 } , it holds that f r t ( x, y, w ) ∈ [ -1 , 1] . Further, under the null H 0 in (26a), it trivially holds that E H 0 [ f r t ( X t , Y t , W t ) | F t -1 ] = 0 , where F t -1 = σ ( { ( X i , Y i , W i ) } i ≤ t -1 ) . Since ONS betting fractions ( λ ONS t ) t ≥ 1 are predictable, we conclude that the resulting wealth process is a nonnegative martingale. The assertion of the Theorem then follows directly from Ville's inequality (Proposition 2) with a = 1 /α .

Note that if ONS strategy for selecting betting fractions is deployed, then (49) implies that the tests will be consistent as long as

(a) Step 1. Consider a predictable sequence of scaling factors ( s t ) t ≥ 1 , defined in (30a), and the corresponding sequences ( µ t ) t ≥ 1 and ( ν t ) t ≥ 1 , defined in (30b) and (30c) respectively. For t ≥ 1 , let F t := σ ( { ( X i , Y i , W i ) } i ≤ t ) . Since the losses are bounded, we have that:

is a bounded martingale difference sequence (BMDS). By the Strong Law of Large Numbers for BMDS, it follows that:

Since (( X t , Y t , W t )) t ≥ 1 is a sequence of i.i.d. observations, we can write

where ( X,Y,W ) ⊥ ⊥ ( θ t ) t ≥ 1 , θ ⋆ . Using Assumption 3, we get that:

since ∥ θ i -θ ⋆ ∥ a . s . → 0 by Assumption 4. In particular, this implies that µ t a . s . → E [ Wℓ ( g ( X ; θ ⋆ ) , Y ) | θ ⋆ ] . Similar argument can be used to show that ν t a . s . → E [ (1 + W ) · ( ℓ ( g ( X ; θ ⋆ ) , Y )) 3 | θ ⋆ ] , and hence,

Note that s ⋆ is a random variable which is positive (almost surely) by Assumption 4. (b) Step 2. Recall that for any x ∈ R : tanh( x ) ≥ x -1 3 · max { x 3 , 0 } and that max { W, 0 } = ( W +1) / 2 since W ∈ {-1 , 1 } . We have:

Note that θ i and s i are F i -1 -measurable (see Step 1 for the definition of F i -1 ). Under a minor technical assumption that ( s t ) t ≥ 1 is a sequence of bounded scaling factors (the lower bound is trivially zero and the upper bound also holds if ν t are bounded away from zero almost surely which is reasonable given the definition of ν t ), we can use analogous argument regarding a BMDS in Step 1 to deduce that:

Using argument analogous to (63), we can show that:

Combining (63), (64) and (66), we deduce that

Hence, from (65) it follows that:

where the RHS is a random variable which is positive almost surely. Hence, a sufficient condition for consistency (62) holds which concludes the proof.

D.5 Proofs for Appendix B

Payoff for the Case of Unbalanced Classes (known π ). To see that the payoff function (37) is lower bounded by negative one, note that:

To see that such payoff is fair, note that:

Theorem 5. Suppose that H 0 in (35a) is true. Then ( K u t ) t ≥ 0 is a nonnegative supermartingale adapted to ( F t ) t ≥ 0 . Hence, the sequential 2ST based on ( K u t ) t ≥ 0 satisfies: P H 0 ( τ < ∞ ) ≤ α .

Proof. First, we show that ( K u t ) t ≥ 0 is a nonnegative supermartingale. For any t ≥ 1 , the wealth K t -1 is multiplied at round t by

Since λ t ∈ [0 , 0 . 5] , we conclude that the process ( K u t ) t ≥ 0 is nonnegative. Next, note that since ˆ π t is the MLE of π computed from a t -th minibatch, it follows that:

Recall that F t -1 = σ ( { Z i , W i } i ≤ b ( t -1) ) . It suffices to show that if H 0 is true, then

Note that the individual terms in the above product are independent conditional on F t -1 . Hence,

For any i ∈ { b ( t -1) + 1 , . . . , bt } , it holds that:

Hence, we conclude that ( K u t ) t ≥ 0 is a nonnegative supermartingale adapted to ( F t ) t ≥ 0 . The timeuniform type I error control of the resulting test then follows from Ville's inequality (Proposition 2).

E Additional Experiments and Details

E.1 Modeling Details

CNN Architecture and Training. We use CNN with 4 convolutional layers (kernel size is taken to be 3 × 3 ) and 16, 32, 32, 64 filters respectively. Further, each convolutional layer is followed by max-pooling layer ( 2 × 2 ). After flattening, those layers are followed by 1 fully connected layer with 128 neurons. Dropout ( p = 0 . 5 ) and early stopping (with patience equal to ten epochs and 20% of data used in the validation set) is used for regularization. ReLU activation functions are used in each layer. Adam optimizer is used for training the network. We start training after processing twenty observations, and update the model parameters after processing every next ten observations. Maximum number of epochs is set to 25 for each training iteration. The batch size is set to 32.

Single-stream Sequential Kernelized 2ST. The construction of this test is the extension of 2ST of Shekhar and Ramdas [2023] to the case when at each round an observation only from a single distribution ( P or Q ) is revealed. Let G denote an RKHS with positive-definite kernel k and canonical feature map φ ( · ) defined on Z . Recall that instances from P as labeled as +1 and instances from Q are labeled as -1 (characterized by W ). The mean embeddings of P and Q are then defined as

where N + ( t ) = | i ≤ t : W i = +1 | and N -( t ) = | i ≤ t : W i = -1 | . The corresponding payoff function is

To make the test computationally efficient, it is critical to update the normalization constant efficiently. Suppose that at round t +1 , an instance from P is observed. In this case, ˆ µ ( t +1) Q = ˆ µ ( t ) Q . Note that:

Hence, we have:

In particular,

Note that:

Next, we assume for simplicity that k ( x, x ) = 1 , ∀ x which holds for RBF kernel. Observe that:

By caching intermediate results, we can compute the normalization constant using linear in t number of kernel evaluations. We start betting once at least one instance is observed from both P and Q . For simulations, we use RBF kernel and the median heuristic with first 20 instances to compute the kernel hyperparameter.

MLP Training Scheme We begin training after processing twenty datapoints from P XY which gives a training dataset with 40 datapoints (due to randomization). When updating a model, we use previous parameters as initialization. We use the following update scheme: we start after next n 0 = 10 datapoints from P XY are observed. Once n 0 becomes less than 1% of the size of the existing training dataset, we increase it by ten, that is, n t = n t -1 +10 . When we fit the model, we set the maximum number of epochs to be 25 and use early stopping with patience of 3 epochs.

Kernel Hyperparameters for Synthetic Experiments. For SKIT, we use RBF kernels:

For simulations on synthetic data, we take kernel hyperparameters to be inversely proportional to the second moment of the underlying variables (the median heuristic yields similar results):

Spherical model. By symmetry, we have: P X = P Y , and hence we take λ X = λ Y . We have

HTDD model. By symmetry, we have: P X = P Y , and hence we take λ X = λ Y . We have

Sparse signal model. We have

Gaussian model. We have

Ridge Regression. Weuse ridge regression as an underlying predictive model: ˆ g t ( x ) = β ( t ) 0 + xβ ( t ) 1 , where the coefficients are obtained by solving:

Let Γ = diag(0 , 1) . Let X t -1 ∈ R 2( t -1) × 2 be such that ( X t -1 ) i = (1 , X i ) , i ∈ [1 , 2( t -1)] . Finally, let Y t -1 be a vector of responses: ( Y t -1 ) i = Y i , i ∈ [1 , 2( t -1)] . Then:

E.2 Additional Experiments for Seq-C-IT

In Figure 7, we present average stopping times for ITs under the synthetic settings from Section 3. We confirm that all tests adapt to the complexity of a problem at hand, stopping earlier on easy tasks and later on harder ones. We also consider two additional synthetic examples where Seq-C-IT outperforms a kernelized approach:

Sparse signal model. Let ( X t ) t ≥ 1 and ( ε t ) t ≥ 1 be two independent sequences of standard Gaussian random vectors in R d : X t , ε t iid ∼ N (0 , I d ) , t ≥ 1 . We take

where B s = diag( β 1 , . . . , β d ) and only s = 5 of { β i } d i =1 are nonzero being sampled from Unif([ -0 . 5 , 0 . 5]) . We consider d ∈ { 5 , . . . , 50 } .

3

4

5

6

7

8

9

10

d

0

1000

2000

3000

4000

5000

Average Stopping Time

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

(a) Spherical model.

1

2

3

4

5

6

w

0

1000

2000

3000

4000

5000

Average Stopping Time

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

(b) HTDD model.

Nested circles model. Let ( L t ) t ≥ 1 , (Θ t ) t ≥ 1 , ( ε (1) t ) t ≥ 1 , ( ε (2) t ) t ≥ 1 denote sequences of random variables where L iid ∼ Unif(1 , . . . , l ) for some prespecified l ∈ N , Θ t iid ∼ Unif([0 , 2 π ]) , and ε (1) t , ε (2) t iid ∼ N (0 , (1 / 4) 2 ) . For t ≥ 1 , we take

We consider l ∈ { 1 , . . . , 10 } .

In Figure 8, we show that Seq-C-ITs significantly outperform SKIT under these models. We note that the degrading performance of kernel-based tests under the nested circles model (67) has been also observed in earlier works [Berrett and Samworth, 2019, Podkopaev et al., 2023].

E.3 Sequential vs Batch 2STs

The role of sequential tests is to complement existing batch tests and not replace those. Generally, if one is ready to commit to a particular sample size, it is better to refer to batch tests. To understand the loss of power, we conducted an experiment where we take P = N (0 , 1) , Q = N ( δ, 1) and use k -NN predictor as an underlying predictor. As we point out in Remark 1, if an analyst stops the experiment early, i.e., before the wealth exceeds 1 /α , there is one modification that allows using non-exhausted type-I error budget to strictly improve power [Ramdas and Manole, 2023]: one may use a different threshold for rejecting the null, namely U/α , where U is an independently drawn (stochastically larger than) uniform random variable on [0 , 1] .

We set the sample size to 1500 and compared three tests: batch (with half of the data used for training and half for inference), Seq-C-2ST (our sequential test), and Seq-C-2ST + Randomization (as per Remark 1). In Figure 9, we observe that the batch test has only slightly higher power than our sequential one when the sample size is specified beforehand. Yet in cases where the power is less than one using a sequential test allows collecting more data to improve it, but with the batch test, nothing can be done since the type-1 error budget is fully exhausted.

10

20

30

40

50

d

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Power at Time

T

= 5000

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

(a) Sparse signal model.

10

20

30

40

50

d

0

1000

2000

3000

4000

5000

Average Stopping Time

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

(b) Sparse signal model.

2

4

6

8

10

l

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Power at Time

T

= 5000

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

(c) Nested circles model.

2

4

6

8

10

l

0

1000

2000

3000

4000

5000

Average Stopping Time

SKIT

Seq-C-IT (MLP)

:

f

s

t

f

m

t

Seq-C-IT (

k

-NN)

:

f

s

t

f

m

t

(d) Nested circles model.

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

0

.

5

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

Rejection Rate

Seq-C-2ST + Randomization

Seq-C-2ST

Batch

α

= 0

.

05

δ

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\n[Update: After reading the other reviews and the rebuttal, I have increased my score from 7 to 8]\n\nThe paper provides an algorithm for sequential testing of the homogeneity (two-sample) and independence hypothesis. \nThe core idea is that one learns a classifier to distinguish $P$ from $Q$ (in case of two-sample testing) or $P_{XY}$ from $P_X \\times P_Y$ (in case of Independence testing) from the previously seen data. This learned algorithm is used to place a bet on a new observation (how likely it is that it is from one class or the other). If it matches one updates the betting budget (starting from 1). There is also a parameter that trades-off how much of the existing budget is used at a step. If the betting budget ever exceeds $1/\\alpha$, with $\\alpha$ the significance level, then one rejects the test.\n\nThe authors show that the test controls type-I error and is consistent under the alternative if a non-trivial classifier can be learned. Furthermore they provide a strategy to select betting fractions which improves performance, especially if the classifier during the first iterations has poor quality.\n\nSTRENGTHS:\n- The paper nicely combines recent work on sequential testing with the power of general classification algorithms. It does not require any specific learning algorithms but can use any existing learning framework. \n- The theory is presented at an adequate level. \n- The experiments nicely underline the theoretical statements of the work and show good empirical performance.\n\nWEAKNESSES:\n- If I understand correctly, the main innovation over Pandeva et al [2022] is that the present work uses adaptive betting fractions? Is this a correct interpretation? I struggle a bit to clearly see the parallels and differences to Pandeva et al. They formulate their test in terms of E-values. Can your tests also be rephrased in terms of E-values to make a comparison simpler?\n\n\n\n**Minor**: \n- It took me a bit to understand the notation in some places:\n  - l. 75. What does $\\sigma$ stand for?\n  - After l. 104. Say that probability and expectations are wrt 1/2 (P+Q) (at least that's what I figured)\n  - Introduce $\\vee \\wedge$ notation for min/max\n  - l. 127 what is the expectation taken over?\n\nQUESTIONS:\n- Please see my remark above regarding Pandeva et al.\n\nBelow are curiosities that could further improve the paper, but are not absolutely critical.\n- In Batch C2ST, where the sample size is fixed upfront, usually half the data is used for learning/testing. However, I am not aware that this is a principled choice. What if you used your approach even in batch testing to circumvent the need of selecting the splitting ratio. Can we get better power? I think it would be nice to have an experiment with exactly the same algorithm  / architecture and compare the two approaches.\n- You prove that the type-I error is controlled even if the test is run indefinitely. I am curious: Does the test eventually exhaust the significance level? Do you have any theoretical or empirical insights?\n- you apply your test to every new datapoint. Panderas et al only update after a small batch of new data is presented. Have you considered this? It might be computationally a bit more convenient, and I am curious how it would affect test power.\n\nLIMITATIONS:\nI do not see much limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-16"}

comments: SUMMARY:
The authors propose methods of two-sample and independence tests in the setting of sequentially released data. Theoretical and empirical evaluations of the proposed methods are included.

STRENGTHS:
The authors propose algorithms for sequential two-sample and independence tests. The setting is interesting and important for practical applications. They provide theoretical results regarding the stop of the algorithm and the growth rate of the wealth. They also study the proposed methods numerically with synthetic and real-world data.

WEAKNESSES:
The readability can be improved. The introduction part is so long that I understand the contributions of the paper. I think the authors should split Section 1 into 2 sections: one is for explaining the motivation and the contribution, the other is for the problem setting.

QUESTIONS:
- In the paper, the initial values of $\lambda_t^{ONS}$ and $g_t$ are set as 0. Are there any strategies to set better initial values, or are there any empirical observations about that?

- What is $\sigma(Z_1,\ldots,Z_{t-1})$ in line 75? Does it mean the $\sigma$-algebra?

- In line 121, can we guarantee the existence of the minimizer $g_{\star}$? or is it just an assumption?

LIMITATIONS:
In the proposed algorithm, the computation is the same for all $t$. Thus, the estimated $\lambda_t^{ONS}$ and $g_t$ for small $t$ seem to perform worse than those for large $t$. It cannot be helped to a certain extent, but in the sequential setting, I think investigating some strategies to improve the performance for small $t$ would be important.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two sequential predictive two-sample tests based on betting, one is constructed by the payoff function $W\\cdot \\mathrm{sign}[g(Z)]$ for the misclassification risk, and the other is by the payoff function  $W\\cdot g(Z)$ for the squared risk.  The limiting growth rate and the expected growth rate of both tests for an optimal $g^*\\in G$ are given, and additionally, the same quantities of the squared-risk based test for an arbitrary $g\\in G$ are given.\n\nSTRENGTHS:\nOriginality\\\nThe paper proposes and thoroughly analyzes the sequential predictive two-sample tests; to the best of my knowledge, the proposed tests and the theoretical results are new.   \n\nQuality\\\nThe theoretical result is decent, which gives a growth rate comparison for the test based on the misclassification and the test based on the squared risk. \n\nClarity\\\nThe paper overall presents well.\n\nSignificance\\\nThe paper fills the blank that there is no formal construction of a classifier two-sample test using the betting framework.\n\nWEAKNESSES:\n1. The variance of the growth rate is not given and it is actually also important in the derivation of the testing power. \n\n2. The expected growth rate is a result of optimal $\\lambda$ instead of an arbitrary one. \n\n3. The comparison with the likelihood ratio test (Alix Lhéritier, 2018) is missing in the real-data experiment.\n\nQUESTIONS:\n1. Why $\\lambda$ is restricted to $[0, 1]$ instead of $[-1, 1]$. I can imagine a case where the payoff is negative, then using negative $\\lambda$ would help increase the wealth in betting. \n\n2. Is fitting the betting framework into the coin flip case the reason why the payoff functions are restricted to $[-1,1]$?\n\n3. Why the ONS in algorithm 1 is different from the one in  (Cutkosky and Orabona, 2018) work and  (Shubhanshu Shekhar and Aaditya Ramdas, 2021), e.g., the update of $z_t$ and $\\lambda_{t+1}^{ONS}$.\n\nLIMITATIONS:\nPlease see the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work considered the problems of sequential nonparametric two-sample and independence\ntesting. The researchers propose a novel approach, which overcomes the issues of kernel-based testing, such as finding an appropriate kernel for high-dimensional or structured data like text and images. The authors empirically demonstrate the superiority of their tests over\nkernel-based methods in structured settings. Furthermore, the tests remain valid and powerful even when the data distribution drifts over time.\n\nSTRENGTHS:\n1. The authors propose a prediction-based betting strategies to help alleviate problems associated with kernel selection and adaptability with high-dimensional or structured data.\n\n2. The authors provide compelling empirical evidence that their method outperforms existing kernel-based tests, especially in the context of structured settings.\n\n3. The author provides the theoretical analysis about the properties of the proposed method.\n\nWEAKNESSES:\n1. The authors should provide more technical details about the principle of testing by betting strategies. This is the key concepts in the paper but the author does not elaborate it too much in the paper.\n\n2. The paper provides a high-level overview of the research making it challenging for general readers to comprehend.\n\n3. The presentation is poor is a way mixed with methodology, theory, and numerical study in a single section (e.g. Section 2).\n\n4. It is uncommon that one paper focused on two testings: the two-sample test and the independence test.\n\nQUESTIONS:\n1. Does the proposed method have any potential limitations on data-dependency due to the use of nonparametric techniques?\n\n2. Could the authors clarify how your method can retain its power when data distribution drifts over time?\n\n3. It is not clear how the proposed techniques can be scalable to larger datasets?\n\nLIMITATIONS:\nThe authors do not mention the limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Kernel-based nonparametric two sample and independence tests performance can break down in the cases of complex data like images. The paper proposes sequential two sample and independence tests based on the misclassification rate. 

Compared to kernel-based tests, the proposed tests, which use CNNs, reject the null hypothesis faster than sequential kernel-based two sample and independence tests on image data while still controlling the type I error rate. The power comparison between the two approaches is somewhat inconclusive.

STRENGTHS:
The proposed approach is quite general and well suited to complex data like images that can be modeled with DNNs. 

The work is well supported theoretically.

The empirical results argue favorably for the approach. 

The paper is well written.

WEAKNESSES:
Some discussion of the practical computational complexity of the approach compared to kernel based tests would improve the paper.

An empirical comparison to kernel-based tests on simpler data would highlight the advantages/disadvantages of each approach.

QUESTIONS:
Optimizing the bandwidths is one drawback of kernel tests - how much hyperparameter tuning was required to achieve the performance reported?

Have the authors compared the approach to kernel-based tests on simpler data?

How do the runtimes compare to kernel-based tests?

LIMITATIONS:
Could be expanded
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-08