E-Valuating Classifier Two-Sample Tests
Authors: Teodora Pandeva
Tim Bakker
Christian A. Naesseth
Patrick Forré
Venue: Accepted by TMLR ()

Abstract: We introduce a powerful deep classifier two-sample test for high-dimensional data based on E-values, called E-C2ST.  Our test combines ideas from existing work on split likelihood ratio tests and predictive independence tests. The resulting E-values are suitable for anytime-valid sequential two-sample tests. This feature allows for more effective use of data in constructing test statistics. Through simulations and real data applications, we empirically demonstrate that E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches, beyond the conventional two-split (training and testing) approach of standard two-sample classifier tests. This strategy increases the power of the test, while keeping the type I error well below the desired significance level.

Published in Transactions on Machine Learning Research (04/2024)
E- E-Valuating Classifier Two-Sample Tests
Teodora Pandeva
Ep.pandeev@@gmmiil.com com
University of Amsterdam
Tim Bakker University of Amsterdam
J.b.bkker@uvv.nl
Christian A. Nacsscth
w.o.nassschhuww...
University of Amsterdam
Patrick Forré University of Amsterdam
p. p.d.forre@e d.Joore@uvo..t
Reviewed on OpenReview: https: // http:///pprreveeeee.mm net/ forum? id= duFRovEahr
Abstract
We introduce a powerful deep classifier two-sample test for high-dimensional data based on R-values, called E-value Classifier Two-Sample Test (F-C2ST). Our test combines ideas from existing work on split likelihood ratio tests and predictive independence tests. The resulting E-values are suitable for anytime-valid sequential two-sample tests. This feature allows for more effective use of data in constructing test statistics. Through simulations and real data applications, empirically demonstrate that E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches, beyond the convent ional two-split (training and testing) approach of standard two-sample classifier tests. This strategy increases the power of the test, while keeping the type error well below the desired significance level.
1 Introduction
We consider two-sample tests, which aim to answer the statistical question of whether two independently obtained populations are statistically significantly different. Often these tests help to distinguish real from generated data (Lopez-Pat and Oquab] (2017), noise from data (Hastie al. 2001 Gutanann and Hyvärinen, 2012 Mikolov et al.) 2013 Goodfellow et al. 2014), and widely used in simulation-based inference
Lucckmann et al. 2021 Miller et al. (022). In the general setting, consider the scenario where we are given two independent samples from two possibly different distributions:
x
(0) Lid. No Po.
(1) i.d. N₁
Pt.
Based on these samples, we want to test if the distributions are equal or not. Thus, we can define a corresponding statistical test with null and alternative hypotheses follows:
Ho Po Pi
HA Pr.
In this paper, consider the two-sample testing problem in the context of sequential testing, where the user accumulates data from Po and P1 in time-dependent manner. The primary goal is to evaluate, at each observed time step, whether the null hypothesis defined above remains valid. Thus, if enough evidence against the null is acquired, we can reject the null and stop collecting data.
Related work. Two-sample testing has a long history in statistics, giving rise to classical techniques such as Student's and Welch's t-tests (Student) 1908 Welch 1947), which compare the means of two normally distributed samples. Tn addition, nonparametric tests such the Sitokoon-Mnn-WWittee test. (Mann)

---

Published in Transactions on Machine Learning Research (04/2024)
and Whitney 1947| the Kolmogorov-Sminng Smiinov tests (Kolmogorov 1933 Smirnov 1939 and the Kuiper
test. (Kuiper 1960) have been established. In the area of high-dimensional data, kernel methods have been introduced, which focus on comparing the kernel embeddings of two populations (Gretton et al. 2012 Chwialkowski et al. 2015 Jitkrittum et al.) 2016). However, these traditional two-sample statistical become less powerful when dealing with more complicated data types such as images and text.
Recent advances have led to the development of classifier- two-sample tests (Kim et al.) 2021 and their deep learning extensions, seen in (Lopez-Paz and Oquab] 2017 Kirchler et al., 2020 Liu al. 2020 Cheng and Cloninger 2022). In these methods, a model is trained to discriminate between the two populations using training data, and then statistical test is performed on separate test set. However, all listed methods share a common limitation: when applied sequentially, they can lead to inflated type I error. In simpler terms, these methods assume that the sample size is known in advance, which can be a drawback in practice.
To address this limitation, sequential testing procodures offer a solution by allowing practitioners to dynamically reject the null hepothesis as new batches of data arrive. Within this context, B-value-based sequential tests have revived in the work of Shafer (2019); Ramdas et al. (2023 Grünwald et al. (2024), where they are interpreted as bets against the null hypothesis. More formally, E- Evaaiaalles are simply non non-negative variables E that satisfy
for all P Ho: Er[E] 1
i.e. the expectation of E with respect to any distribution from the null hypothesis distribution class Ho is less than one. An example of E-variables for singleton hypothesis classes are Bayes factors, i.e. we test if the unknown probability density p equals Po or PA:
Ho p Po
VS.
HA p P PA-
The Bayes factor given by E(x) = (x(2) is an E-variable w.r.t. Ho since Epm [=]= = pu(a)
po(a) Po (z( dr 1 S Note that observing a very large value of E, which we call an E-calue, provides evidence against the null hypothesis.
The appealing theoretical properties of vavariables (details in Section 2.2 have led to the development of a growing body of work on B-value- based (conditional) independence tests spanning several domains, including two-sample tests for contingency tables (Turner and Grünwald 2023 sequential data (Balsubramaní and
Ramdas 2016 kernel-based approaches (Podkopaev et al. 2023 Shekhar and Ramdas 2024), rank-based conditional independence tests (Duan et al. 2002). and conditional independence tests under 'model-X' assumptions (Grünwald et al.] 2023 Shaer et al.] 2023), split-likelihood two-sample tests (Lhéritier and
Cazals 2018), the concurrent work by Podkopacy and Ramdas 2024).
Contributions. We extend the work of [héritier and Cazals (2018) by introducing R-values for conditional independence testing which is larger class of testing problems (see Section 4 The resulting E-values
combine the ideas of the split-likelihodd testing procedure of Wasserman et al. (2020) (sec Section 3 and the
existing work on predictive conditional independence testing frameworks of Burkart and Király (2017) (see Section In contrast to (Lhéritier and Cazals 2018). our framework, when applied to two-sample testing, referred to as E-C2ST, assumes a composite null hypothesis. Furthermore, it leverages the representational capabilities of machine learning models, allowing us to design tests for complex data structures.
We show that the described tests (including E-C2ST) provide non-asymptotic type error control under the null, and are consistent (i.e., reject the null almost surely) in both a sequential and a non-sequential setting (details in Section 4p. Moreover, when restricted to the two-sample test setting, we establish milder conditions required for the machine learning model than those in (Lhéritier and Cazals 2018) for the consistency guarantees to hold.
In our empirical analysis in Section f we use the theoretical properties of E-C2ST to design sequential tests that optimize data usage by segmenting it into multiple batches. Each batch contributes to the cumulative test statistic. This method contrasts with traditional two-sample classifier tests, which derive a test statistic solely from the test set conditioned on the training data. Our approach not only achieves maximum power faster than standard methods, but also consistently keeps type I errors well below the significance level.
2

---

Published in Transactions on Machine Learning Research (04/2024)
2
Hypothesis Testing with E-Variables
Building on the recent work of Ramdas et al. 2023 Grünwald et al. 2024), we give detailed introduction to E- variables and their properties in Section 2.2 and establish their connections to hypothesis lesting in Sections 2.3
and 2.4
We deferred all proofs to Appendix First, we introduce the notation used throughout this paper.
2.1
Notation
Consider a sample of data points 22,....... reflecting realizations of random variables X1, X2..., drawn from an unknown probability distribution P E P(12) coming from some unknown sample space f2, where P(2)) is the of all probability measures on Q. In hypothosis testing, we usually consider two model classes:
Ho = (Po E P(2) E Oo} (null hypothesis),
HA = {Po E P(Q) E @A} (alternative),
where Oo and OA represent the parameter sets of the distributions that are valid under the null and alternative, respectively. We want to decide if P comes from Ho from HA:
Ho: PE Ho
vs. HA: PE HA.
In most cases, the data points come from the same space X' and we would at most observe countably many of such data points Xn. In this setting, we can w.loog. assume that Ω XN, If we, furthermore, assume that the Xn, n E IN, are drawn i.i.d. from P then P((Xn)nEN/ & P(Xn), we can directly incorporate the product structure into Ho and HA and restrict ourselves to one of those factors to state H4. By slight abuse of notations we re-write for i = A:
H. (Po E ( O.),
and implicitly assume that Po((Xn)neN) = 8 N (x())) Moreover, assume that our probability measures, Po Hi, are given via a density with respect to product reference measure p. We denote the density by po px))) or p(x(0) interchangeably in this work.
2.2 Conditional E-Variables
Now consider the more general relative framework where we allow hypothesis classes to come from set of Markov kernels, which can be used to model condilional probability distributions for i 0, A:
H. = (Po - P P(X)3
(1)
where 2 denotes the space of all Markov kernels from to X, i.e for each Po E H, for fixed e Z (-12)) is a valid probability measure on X. An example of conditional hypothesis classes is given in Section 4
where the null hypothesis class represent the set of distributions that reflect the conditional independence
of two variables after observing third one. With respect to Ho defined in Equation ( we can define corresponding E-variables which call conditional E-voriable,
Definition 2.1 (Conditional E-variable). A conditional E-variable w.r.t. Ho S 2(X)² is non-negative measurable map:
E X Rzo.
(52))E((1))))
such that for all Pg E "o and a E Z we have:
[=[E2]] ((de))) <
(ddd)))
In the following will write small if we either mean the realization of raudom variable X the argument of function living the same space. We capital X for data point if we want to stress its role a random variable. 2A formal definition of the "unconditional" E-variables introduced Section [ can be casily derived from Definition 2.1 by dropping z. Moreover, if
vvrriable and x X fixed point then we call F(x:) the R-walue of w.r.t. E. A similar definition can be found in (Güünald 2024
3

---

Published in Transactions on Machine Learning Research (04/2024)
One of the notable features of E-variables is their preservation under multiplication. We can easily combine (conditionally) independent F-variables by simply multiplying them which results in a proper E-variable. This property makes E-variables appealing for meta- anellsss studies (Vovk and Wang 2021 Grünwald et al. 2024 A more general result is that backward-dependeet conditional E-variables can be combined by multiplication. This property of E-values is analogous to the chain rule observed in probability densities. Such a property becomes key in the development of the E-C2ST in this framework, which is formally stated as:
Lemma 2.2 (Products of conditional E-variables (based on Grünwald al. (2024))). Ef) is a a conditional E-voriable w.r.t. H6" _ P()) and E(2) conditional E- E-voriable w.r.t. H S P(X)>x2 then E defined via their product:
E (yl)))
is a conditional E-variable w.r.t.
Ho = (2)) @ 3H6 c x12,
where we define the product hypothesis as:
H
Ho == { Pa 8 Pv Pv
Po E Hb? Ps E H1' H
with the product Markov kernels given by:
(Po 6 Pa) (dr, dylz) = Poldxly, 2) (dylz).
2.3 Hypothesis Testing with Conditional E- E-Variables
In the context of statistical testing, we evaluate an E-variable based on the given data points, which are realizations of the random variables X1,.... XN. Subscquently, the decision criterion for rejecting the mull hypothesis at significance level E [0 is as follows
Reject Ho in favor of HA E(N)....NNN) 2 aa
Lemma tells tells us that with this rule the type I error, the error rate of falsely rejecting the Ho, is bounded by a. Lemma 2.3 (Type error control). Let E be a conditional R-variable w.r.t. Ho S P(x)2 Then for every a E [0,1], Po E Ho and & E Z we have:
Po 1 a 11) Sa.
Thus, the E-values can be transformed into more conservative p-values via the relation = min(1,1/E) such that for Pg E Ho it holds Po(p & a/2) S a. Note that a valid way of constructing an E-variable from the independent random variables XI...., XN w.r.t, the observed sample points according to Lemma 2.2 is E(XI,..., X II E(X,).
2.4 Sequential Hypothesis Testing with Conditional E-Variables
Up to this point, we have focused primarily on E-value-based tests for scenarios where the sample size N is predetermined. Now suppose that the data does not arrive all once, but instead we observe an infinite stream of data. In this context, new sample X, becomes available at each time t. Consequently, we are interested in developing statistical tests that allow us to reject the mill hypothesis at any given time t. These tests are called sequential lests and can be constructed using E-variables.
Building on the concepts introduced in the previous section, we can define a conditional variable g(+) = E(X,|X,,.... X conditioned on past obsorvations X,,...,X, X₂-2 with respect to the null hypothosis H(") C P(x)x*) Importantly, Lemma 2.2 suggests combining all the evidence available up to time t to construct a backward-dependers E-variable. In other words, the running product E(St) I EO0, where F(1) = F((X)), proves to be valid E-variable with respect to Ho.
4

---

Published in Transactions on Machine Learning Research (04/2024)
This sequence of E-variables, also known as E-process [Ramdas al 2023 offers theoretical advantage over non-sequential p-value-based tests by allowing what it is called "optional continuation" (Grünwald et al. 2024). In simple terms, the user can make an informed decision at any given time t: whether to accumulate more data from additional experiments or to stop the process. This decision can be driven by, for example, the decision to reject the null hypothesis. The optional continuation properly is facilitated by the following result, which ensures an anytime type-I-error bound for the process (F())))))1.
Proposition 2.1 ( RRamdas al. 2023 Grünwald al. 20244). Let F)SS)) be the running E-variable described above. Then for all Pe E Ho and all (0,
Po ( 2 0 such that E(50) 2 a Sa.
This result implies that maintain lype-I-error control not just at individual time points but consistently throughout the entire data collection period. More precisely, the decision rule for rejecting the null hypothesis:
Reject Ho in favor of HA if E 2 a- for any t 2 11
has type error bounded by a. Additionally, we consider this sequential test to be consistent if correctly rejects the null with a finite number of steps: Po 2 such that E(St) 2 a- for all Po E HA.
3 M-Split Likelihood Ratio Test
In general, constructing an E-variable with respect to any Ho is not straightforward task. There exist two main approaches. The first approach, see (Grünwald et al. 2024), is based on the reverse information projection of the hypothesis space HA onto Ho. It is not data-dependent and can be shown to be growth- optimal in the worst case. However, the reverse information projection is not very explicit in general settings, especially when working with non-convex hypotheses, HA and Ho. The second approach is based on constructing a data-driven E-variable. By utilizing the M-split likelihood ratio test by Wasserman et al. (2020) introduced in this section, we establish an E-variable for a fixed sample size. Subsequently, we will demonstrate that the same E-variable can be adapted for sequential testing for infinite data stream.
Assume that our data set D = (X ...... XN) is of size N. We now split the index set [N] (1....,N) N) into M 2 2 disjoint batches: [N] I(1 U ... (Z(A), For m = I....,M M we also abbreviate:
[(<m) = ( ... Úz(m-1),
(m) = (In)neZim) E II
x (m))
NEI(m)
and yf (<5m) g(Sm), I(Sm), analogously. Then for m = .,..., M we follow these steps:
Train a model on O^ on all previous points (<mm) in an arbitrary way (MLE, MAP, full Bayesian, etc.) and get pa(a)at (=m)) To achieve high power of the test, the density pa (x/aa( (<])))<m) should reflect the true distribution in the best possible way to generalize well to unseen data.
2. Train model Oo the data points of the current batch (m)) (conditioned on the previous ones xl x(<m) via maximum-likelihood fitting (MLE):
0g5m) = 0(m)) (a(Sm)
argmax po (m (S())) PEa,
and get: (o(2) p( (6m)) (a(Sm))) Note that under i.idd. assumptions there is no dependence on xl p(<m)
3. Evaluate both models on the current points (m) and define E(m) via their ratio:
(m)) (x(mm)) Po (o( (m) (=(m)))
(x( ( (x(=))
(x(
(2)
maxace, Po (m((m))aammm))
Then ((m) constitutes conditional E-variable, conditioned on the space {(sm), w.r.t. H6 (m))(m)) (see Appendix A for the proof).
5

---

Published in Transactions on Machine Learning Research (04/2024)
From the previous section and Lemma 2.2 we know that the running product
M = II F(m)
E = F(EM)
33)
m=1
defines an E-variable w.r.t. Ho H(SM) (see Appendix A for the proof). For fixed M, Lemma 2.3 gives type guarantees of the derived test. In other words, the M-split likelihood ratio test, for significance level a E [0, 1], rejects the null hypothesis Ho if E(X XN) 2 a- with type I error bounded by a.
Remark 3.1 (Intuition for the m-th conditional E-variable.). For a fixed m, the m-th conditional E-voriable F(m) intrinsically compares the HA-model's test performance: lo (x(m)) (x (x(m))))))) which is trained
z(<m), tested on æ(1²), with the Ho-model's train performance: - log po p0(2 (z(1m) (x(1)) both trained and tested on the same x(m) in the i..dd case. This means that if the olternative is true, then the Ha-mmdel PA has to perform better on mm)) than the Ho-model Po- while the latter was allowed to divectly be (over)filted on (m)).
Now consider scenario where the number of batches M is not fixed, and we are dealing with continuous stream of incoming batches of data. Using the insights from Proposition 2.1 the resulting E-value from (3) can be used for sequential testing. In this way, we can achieve an even more robust form of batch-wise anytime type I error control (details in Appendix A for the proof), as follows:
Corollary 3.2 (Batch-wisc anytime type error control). Consider the sequence of E-variables (E(SM) MEN from equation Equation (3) for an infinite stream of finite batches of data points. It follows that for every Po Ho and every (0, we have the anytime type I error bound:
Po (EM EN E(SM)
E(SM) 2 2a-1) S a.
3.1 Test Consistency
In this section, we explore the consistency of the test introduced earlier, both for the specific case of M 2 and for that of M oo, addressing standard predictive testing and sequential testing scenarios. Consistency is key concept in statistical lesting, it guarantees that as more data are accumulated under the alternative, the test becomes more reliable in accurately detecting true hypotheses. This property allows us to examine how the test behaves as the sample size increases.
Now consider the split-likelhood case for M = 2 and the singleton set Ho = (Po} with density po for which the E-variable is given by
PA(x (x()) Po(xn)
E (2))) E (2())))))))
II
NEZR)
where with (N())) want make explicit the dependence of the R-variable the train (N()) and test (2)) data sizes. By making mild assumptions about the learner and ensuring the boundedness of the (conditional) E-variable, we prove (details in Appendix B.1 the consistency of the test with respect to the R-variable defined above:
Theorem 3.3. Let Ho (Po} be singleton set. Consider model class HA and learning algorithm that for every realization (Nn)NEw E AN and every number (()) E IN fits a model p E P(()) to the first NO) entries x(1)_ (In)neza) of x. Assume that for every Po E HA and Po-almost every i.i.d. realization x = (In)EN of Po there erists a number (2) E N such that for all N() 2 (x)
KL(Po|IPHm))) (o)
sup log E(x, (x()] < co. z, Ex
(4)
Then, Po (EIN <a=1) - 0 for N(1),N()) oo.
N2

---

Published in Transactions on Machine Learning Research (04/2024)
To ensure the consistency of the test, certain conditions required. The first key assumption is that, given sufficiently large data set, the output model PA p(r(") has greater fit to the class of alternative hypotheses than the null distribution has to the same class. The second assumption implies that both PA
(u(a) and Po must
(ax (z) 0 and inf Po(x) > 0. For example, this condition can be easily satisfied for certain discrete ristribu,,, such as the categorical distribution,
have lower bounds. In other words, it requires that infzex PA
as shown in Section 5 A more precise formulation of this theorem can be found in Theorem B.6 Lhéritier
and Cazals (2018) discuss similar but stronger assumptions. In their work, plz is required to be strongly
pointwise consistent, i.e. pla(1) (x) N= (x( almost surely, for which they can provide X-consistency results (a weaker notion of consistency). They also assume that the mull hypothesis known. Next, we will show that in the case M = 00; we can remove this condition and only assume that the learner is a better approximation of the true distribution than the estimated mull one. Under similar conditions as Theorem 3.3 we can prove that the sequential test defined in Equation (3) is consistent. The proof is deferred to Appendix B.2
Theorem 3.4. Consider the sequence of E-variables (E(SM)) MEN from Equation (3) for an infinite stream
of finâte batches of data points. Let the learning algorithm fit a model plu(sM) batches r(<M)- (In)neziean). Assume that for every Po HA and for all M and every instantiation of
E P(Z) to the first M
x(SM) the learner satisfies
Eo log po(x! po(o(Mya(oo)) KL (1x))1100))))))) TM O, (
log E(x)o!<M)) 8M.
sup =ExiZ(a0)
where for the positive sequences (raN)MeN and (SM) MEN hold
lim sup M-000 M
1
Σ I'm > 0,
lim +000
pm
< 00
m
Then, Po(EM E IN such that E(SM) 2 1)-1)=11 1
The requirement regarding the first sequence can be understood as guaranteeing that the learner consistently provides a better approximation of the true distribution compared to the estimated mull distribution, averaged over sequence of M consecutive steps. In this context, T' could even be a decreasing null sequence, such as log(M)/M. The second condition is milder assumption than uniform boundedness. Similar to the previous result, this condition is relatively easier to satisfy when dealing with categorical random variables or random variables with a compact support.
4 Predictive Conditional Independence Testing
In this section, we combine the ideas of predictive conditional independence testing by Burkart and Király (2017) with E-variables from the M-split likelihood ratio test from Section 3 based Wasserman et al. [2020] to derive a proper E-variable for conditional independence testing. The desired two sample test will later be reformulated as an independence test by utilizing the theoretical results discussed in this section.
As a reminder, in conditional independence testing, we want to test a random variable X is independent of Y, or not, conditioned on Z:
Ho: : X 1 Vi, Z) ..... (XN, YN, Zn)) The corresponding (full) hypothosis spaces, in the i.i.d.
vs. HA: X z,
based on data D= ((,,, setting, are:
N/ (Po(X ZZ @ (Z) 0 E 0o)
H (Po θa) \ Ho.
If we assume that P(X, Z) is fired for Ho and HA then this simplifies to the following product hypothesis classes, i A: Hf* : H.dd & (P(X, Z)}, where the conditional hypothesis classes H.d C X(y) of predictive distributions are given by:
Hgd (X)(Y Oo),
Hpd
OA) Ho.
(5)
7

---

Published in Transactions on Machine Learning Research (04/2024)
Equation ( applied to H..h under ii.d. assumptions leads the following m-th conditional E-variable, using the abbreviation = (r, 2):
Eim)(u (wlm)) (u( = PA (A (mo(m), 2(m), wv(m))
(6)
p(y(m) (z(m), (o(y(m), (
The reason that we applied Equation (2) Hix instead of Hi that F( will automatically be a valid conditional E-variable for Hg and even Ho as well.
Remark 4.1. According to Shah and Peters (2020), in the general cuse, valid test for conditional independence lacks power against all alternatives unless certain additional assumptions are introduced. One such assumption is the model X assumption, where the user can access the conditional distribution P(X In the construction outlined above, this assumption is implicit, since assume that the joint distribution
P(X,2) remains ftxed under both hypotkeses.
5 Classifier Two- Two-Sample Tests with E-Variables
Algorithm Algorithmic description of F-C2ST.
1. Input:
Data stream (x(m), (m(m) (En, (nnne Significance level a, Initial A1, Training epochs T
Initialize:
Dtrain, Doat f- Split((En, (ilneinnn))))))))) E() =1
for m = 2..... M do
for t= = I....I T do
9g f- Train Dtrain)
if FaarlyStopping(j: Dunt) then
61
7:
break
Compute ((m) (mns Un (n) as in Equation (9)
Compute E: II g(m)
9:
if E> a1 then
10:
reject and break
11:
Obtain Am+1 that solves
10
13c Drrain + Drrain U (In, Vn 1), Dent +- (Ins Un mi
In this section, we will formalize the classifier two-sample test using E-variables. The following test can be casily derived from the conditional independence test introduced in Scction 4 by introducing a binary variable denoted Y, along with the following abbreviation:
P(X 0) Po(X),
P(X = 1))==(()x())
If we pool the data points (()) via augmenting them with Y-component: (x((1) y-(1) with yr()) == then the pooled data set can be seen as one i.i.d. sample from P(X, Y) of size N = No + N₁ for some unknown marginal P(Y). We can then reformulate the two-sample test as an independence test:
Ho Y
vs.
HA XLY.
This allows us to use the E-variables from Section (without any conditioning variable Z) for (conditional) independence testing. Furthermore, since Y is a binary variable we can write any Markov kernel P(Y a Bernoulli distribution Po x) = (er(a(gg(x)) for some parameterized measurable function go and where a()) = I+e is the logistic sigmoid--unction. So our hypothesis spaces look like:
Ho (Ber(go) 10 0o).
40 [0, 1),
(7)
HA (Ber(((gx)) θA) \ Ho.
8

---

Published in Transactions on Machine Learning Research (04/2024)
and the m-th conditional E-variable is given by:
E(m) (y)m)) (m)) (((m), y(sm)) pa (a! sx((m)) ( ((sm))
(8)
a (gatem) (Fn)) Vn
(- (1- o (fg (Ir))
1-8o
II REZ
N(m) /N(m)
Ny(mm) /N(m)
The maximum likelihood estimator for go with respect to fm)) is represented by g(m) = N/(m) /Nr(), This estimate corresponds to the frequency of data points in the m-th batch that are classified as belonging to class y = Additionally, the function go is trained on the data set (I(<m), yf (())) using binary classification.
Therefore, we combinc all conditional E-variables to create anytime valid E-variable, explained in Section 3 In addition, we can use Theorem 3.4 to prove the consistency of the classifier's two-sample test. Consequently, we introduce minor adjustment to the conditional R-variable in g resuuting in a bounded E-variable that yields the following consistency result:
plz((m sy(tm)
Lemma 5.1. Let the learning algorithm fit model PA (x(<m)) (((m)) = (Inn yn neziim)) Farthermore, let PA
P(2) to the first m -1 batches platsm) y/sm) = Pm polmm)
+ (1 Am P pl
ssmm o sm with
corresponding density p)x
((((m)) and Am E (0,1). Then
(y ((mm) y ((x)) Pa(y(m)) (o((m) x(<m), ( (((m))
(9)
ply(m) Bo (o(y(m)))
7 i.e. for every instantiation (x(sm), (((sm),((mm) y((m)) it holds sup(x,))xxx)) log E(m) (y|z, x ((m)) yy {(<m)) < cc. Additionally, if the batch sample size at most N together with rest of the conditions in Theorem 3.4 it follows that the E-variable (E(SM)) with increments defined in yields a consistent test.
constitutes a bounded conditional R-variable w.r.t
In the above lemma, we also make the assumption that the batch size cannot exceed a fixed number N. This assumption is not particularly restrictive, since we have the flexibility to construct the conditional E variable using maximum of B samples from each incoming batch. Furthermore, one can think of the sequence Am sequence of hyperparameters. We propose a hyperparamcter selection procedure that derives Am from the previous step, leading to the constrained optimization problem:
A p(yn (o(2))) + (1 A) PA (e, (((m) s((mm)) (0
Am+1 E argmax (m)) argmax XE(0,1)
Σ log
(10)
XE(0,1) REZ(m)
In step m - 1, we determine the mixing parameter Am+1 to optimize the conditional E value calculated in the previons step m, as given in (9. When considering an alternative hypothesis, we expect a general decrease in Am with each successive step due the model's enhancement with the accumulation of more data. Under the null hypothesis, this parameter is expected to maintain higher value. Thus, this mixing parameter becomes important in balancing our assumptions about the most plausible hypothesis class. The resulting test we call E-C2ST whose steps are summarized in Algorithm 1
6 Experiments
We compare our method to other classifier two-sample tests on the Blob, MNIST, and KDEF data. We will empirically show that we can exploit the ability of E-C2ST to construct test statistics by using the entire dataset to gain statistical power over the other tests that compute a p-value based only on the train-test data split. Mcanwhile, E-C2ST keeps the type error strictly below the alpha significance level.
6.1 Baselines Implementation and Training
We compare E-C2ST the following basclines: S-C2ST (standard C2ST), the C2ST proposed by Lopez Paz and Oquab (2017); Kim et. al. (2021) b; L-C2ST (logits C2ST) by Cheng and Cloninger (2022 and

---

Published in Transactions on Machine Learning Research (04/2024)
Type Error
Power
0.30
1.0
C25T
L-C2ST M C25T
0.20
5-C25T
0.6
0.15
0.4
0.10
0.05
0.2
0.0
400
630
800
1000
500
1000
1500
Samples
Figure 1: Type error and Power experiments for the Blob dataset. Compared to the baselines, E-C2ST reaches maximum power faster than the baselines while maintaining the type error strictly below the significance level.
M-C2ST by Kirchler et al. (020). Each method involves training classifier to differentiate the two classes and then using the trained model for computing the test statistics. The training procedure is with early stopping, and we use the same fixed network architecture across all tests for all experiments. The resulting p-values are reported from 500 permutations for all baseline tests. The main paper or the appendix details all experiments and tests' implementations (see Appendix D
6.2 Evaluation
For each sample size we sample a dataset from fixed distribution or as subsample from a given data set. In the bascline case, we split the dataset into train, validation and test sets with ratio 5:1:1 and we fit a classifier. Using a significance level of a = 0.05, we decide whether to reject the null hypothesis on the test set. In comparison, E-C2ST evaluated sequentially, More precisely, for each fixed sample size, we sample a data set that is divided into batches of equal size. We apply the procedure described in Section 5 by constructing the running E variable in (3) with initial \ 0.5 in all experiments unless specified htheewise. We decide whether to reject the mull
continue testing each time new batch is observed. We run 100 independent. experiments and report the rejection rates for all methods, corresponding to type I error power, depending
on whether the two classes are from the same distribution or not.
6.3 Empirical Results
Blob Datasct. The blob data set is a twvo-dimensional Gaussian mixture model with nine modes arranged on x grid, used by Gretton et al. (2012); Chwialkowski et al. (2015) in their analysis. The two distributions differ their variance, as visualized Figure 0 in Appendix 1..2 Tn the case of F-C2ST, we split the data into mini-batches of size 90 and compute the test statistics as explained in the previous section. The results are plotted in Figure I where the x-axis refers to the sample sizes of the total data used and the y-axis is the rejection rate. E-C2ST reaches maximum power faster than the baseline methods while achieving type 1 error strictly below the a significance level.
KDEF Data. The Karolinska Directed Emotional Faces (KDEF) dataset [Lundqvist et al., 1998 is used by Jitkrittum al. (2016); Lopez-Paz and Oquab (2017); Kirchler et al. (2020) to distinguish between positive (happy, neutral, surprised) and nogative (afraid, angry, disgusted) cmotions from faces. We draw datasots of sizes 3- 64, 4 64, from the two classes on which perform the statistical tests. We set the E-C2ST batch size to 64 and run 100 independent experiments per sample size. The results are shown in Figure 2 where we compute the rejection rate (type error and power) per sample size. Although our method has lower power for smaller sample sizes than the baselines, it reaches maximum power faster while having type error consistently below 0.05.
Corrupted MNIST Data. The MNIST datasct [LeCun al., 1998 consists of 70 000 handwritten digits. As in (Luu al. 2020), we make use of generated MNIST images (LeCnn et al. 1998) from a pre-trained DCGAN model Radford et al. 2015) for this benchmark experiment. More precisely, assume that under
10

---

Published in Transactions on Machine Learning Research (04/2024)
Type I Error
Power
0.5
1.0
E C2ST L-C2ST
M C25T
5-C25T
0.3
0.6
0.2
0.1
0.2
0.0 450
0.0
2001
600
200
400
000
Samples
Figure 2: Power analysis and type I error for the KDEF data. All methods show very good power performance. The baselines start off with higher power. However, F-C2ST reaches power the fastest while keeping the type error lower than the baselines.
p-
p
0.8
1.0
1.0
1.0
E-C2TT
0.55
LC2ST
M-C25T
S:C25T
0.5
0.0
0.8
0.2
0.1
0.2
0.2
0.
0.0
Samples
Figure 3: Power analysis for the Corrupted MNIST Data for different proportion of corruption p= 0.0.5,0.7, Compared to the baselines, E-C2ST shows the highest power.
the alternative, we are given two sets of MNIST digits where one of them contains "corrupted" images, i.e. images generated from the trained DCGAN. We vary the portion these images in the mini-batches to be p 0, 0.5, 0.7, and we resample the two datasets of size 64, 4 64,.... We fix the mini-batch size to be for training E-C2ST and we conduct 100 independent runs. The rejection rates per sample size displayed in Figure 3 for the four different cases. Note that the rejection rate in the case p= 0 refers to the estimated type I error. We can see that E-C2ST has superior power across the three levels of corruption compared to the baseline methods while keeping the type I error strictly below the significance level.
The mini-batch size. Here, we aim to determine the average number of samples required elfectively reject the null hypothesis. We perform power experiments on two datasets: MNIST (p [p=1] 1) and KDEF, using different batch sizes of 8, 16, 32, 64, and 128. Our methodology differs from previous experiments as we use a sequential testing approach. We contimuuusly sample new batches and stop only when the null hypothesis is rejected. This procedure allows for dynamic adjust.mntts of the sample size needed for maximum power.
We illustrate our results in Figure 4 using 100 independent experiments per scenario. The lines in Figure represent the estimated power. Our results show that smaller batch sizes (c.g., 16 or 32) lead to faster rejection of the null hypothesis terms of the number of samples, but not terms of the number batches required. Conversely, larger sample sizes require more samples reject the null hypothesis, but reduce the number of steps involved, implying less computational power. For example, in the KDEF scenario, maximum power is achieved in only three steps when the batch size is 128. However, reducing batch size too strongly (e.g. batch size 8) can lcad to reduced power (sce MNIST case). We believe this is result of training instabilities for very small batch sizes, leading to suboptimal networks at each step and thus small E-values.
The initial value of We conducted power experiments on two datasets: MNIST (with p = 1) and KDEF, keeping the batch size constant 32 samples. We varied over 0.1, 0.3, 0.5, 0.7 and 0.9. We aim determine the average number of samples required to effectively reject the mull hypothesis. As in the previous experiment, sequential testing strategy.
11

---

Published in Transactions on Machine Learning Research (04/2024)
MNIST
KDEF
1.0
batch size-120
batch size-66
0.8
batch size
Preportion
0.6
0.6
batch size=
23.0 4000
600
800
1000
1200
2000 400 G00
Samples
Figure 4: Power experiments performed on the MNIST and KDEF datasets using different batch sizes (8, 16, 32, 64, 128). The lines indicate the estimated power. Tn general, we can conclude that smaller batch sizes (with the exception of very small batches) allow faster rejection of the null hypothesis in terms of number of samples, and larger batch sizes require fewer steps but more samples.
MNIST
KDEF
0.8
0.8
0.5
0.7
0.4
0.2
0.2
200
200
400
200
1800
700
100
200
300
400
500
Samples
Figure 5: Power experiments performed on the MNIST and KDEF datasets for varying À = 0.1, 0.3, 0.5, 0.7, 0.9 and fixed batch size of 32 samples. The lines indicate the estimated power. The initial value of X had no significant impact on the test performance the KDEF scenario, while the MNIST case, higher values increased the test performance. This effect is due to the early stages of testing, where lower initial values and the suboptimal neural network performance lead to lower batch E-values.
We visualized the results using 100 independent experiments for each scenario, shown in Figure The lines represent the power of the tests. From these results, observe that the initial value of À does not significantly affect the power of the test in the KDEF scenario. However, in the MNIST case, we observe a more visible effect: higher values of X seem to increase the power the test. This occurs because in the early stages of testing when sample sizes are smaller, the neural network tends to show suboptimal test performance. This is particularly evident when the initial E-value, potentially resulting in reduced power performance. However, important to note that the initial À value docs not drastically affect the number of samples required to achieve maximum power in tests.
low, resulting lower conditional E-values. They aflect the overall
To summarize the two ablation studies, compared the best E-C2ST performer based on the last two experiments with the baseline methods in Figure 7 in the Appendix, where we observe a significant gain in terms of power compared to the initial E-C2ST.
7 Discussion
We present E-CS2T, deep E-value-based classifier two sample lest lailored for both fixed and streaming data testing scenarios. This method combines predictive conditional independence tests with M-split likelihood ratio tests, resulting in consistent test basod on a E-variable that guarantces finite sample batch- wicc anytime type error control. Through empirical evaluations, we demonstrate that E-CS2T outperforms traditional svalue-based methods in terms of power, effectively utilizing the data and maintaining type
12

---

Published in Transactions on Machine Learning Research (04/2024)
error below the specified significance level. In addition, our observations highlight trade-off between batch size and computational efficiency, and suggest an interesting future direction for exploring this trade-off from a theoretical and practical perspective.
Online learning. Even though our test computationally efficient for large data sets, for small data regimes it more expensive than the considered baseline methods (see Table A promising direction for future work to integrate an online training procedure into our method. By doing so, we could significantly reduce the computational time and make it linearly proportional to the number of batches processed.
Active Learning. Another interesting direction for future work is to use active learning techniques to improve the performance of our method. For example, we could prune each batch before including it in the training set by actively selecting the most informative samples. With this stralegy, we could potentially improve the learning efficiency and effectiveness of the test.
Acknowledgments
We would like to thank Peter Grünwald for his exciting talk at the AI4Science Colloquium, which introducod us to the theory and background of E-variables and safe lesting. He inspired us to learn more about these topics and to pursue this research project.
Tim Bakker partially supported by the Efficient Dcep Learning rescarch program, which financed by the Dutch Research Council (NWO) in the domain "Applied and Engineering Sciences" (TTW).
References
Balsubramani, A. (2020). Sharp finite-sample concentration of independent variables. arXiv preprint arXiv:2008.1329.
Balsubramani, A. and Ramdas, A. (2016). Sequential Nonparametric Testing with the Law of the Iterated Logarithm. In Proceedings the Thirty-Sccond Conference on Uncertainty in Artificial Intelligence, UAI166, page 42-51, Arlington, Virginia, USA. AUAT Press.
Burkart, S. and Király, F. J. (2017). Predictive Independence Testing, Predictive Conditional Independence Testing, and Predictive Graphical Modelling. ArXie, abs/1711.05869.
Byrd, R. H., Lu, P., Nocedal, J., and Zhu, C. (1995). A Limited Memory Algorithm for Bound Constrained Optimization. SIAM Journal on Scientific Computing, 1665):1190-
Cheng, X. and Cloninger, A. (2022). Classification Logit Two-Sample Testing by Neural Networks for Differentiating Near Manifold Densities. IEFE Transactions on Information. Theory.
Chwialkowski, K. P., Ramdas, A., Sejdinovic, D., and Gretton, A. (2015). Fast Two-Sample Testing with Analytic Representations of Probability Measures. Advances in Neural Information Processing Systems.
Csiszár, 1. (1984). Sanov Property, Generalized I-Projection and Conditional Limit Theorem. The Annals of Probability.
Duan, B., Ramdas, A., and Wasserman, L. (2022). Interactive Rank Testing by Betting. In Conference on Causal Learning and Reasoning, pages 201-233 PMT.R.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. In Advances in Newral Information Processing Systems.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., and Smola, A. (2012). A Kernel Two-Sample Test. The Journal of Machine Learning Research.
Grünwald, P., de Heide, R., and Koolen, W. M. (2024). Safe Testing, Journal of the Royal Statistical Society Series B: Stalistical Methodology-
13

---

Lundqvist, D., Flykt, A., and Öhman, A. (1998). Karolinska Directed Emotional Faces. Cognilion and Published in Transactions on Machine Learning Research (04/2024)
Grünwald, P., Henzi, A., and Lardy, T. (2023). Anytime Valid Tests of Conditional Independence Under Model-X. Journal of the American Stalistical Associalion, 0(0):1-12.
Gulmann, M. U. and Hyvärinen, A. (2012). rontrasvee Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics. Journal of Machine Learning Research.
Hastie, T., Tibshirani, R., and Friedman, J. (2001). The Elements of Statistionl Learning. Springer Series in Statistics. Springer New York Tnc., New York, NY, USA.
Jitkrittum, W., Szabó, Z., Chwialkowski, K. P., and Gretton, A. (2016). Interpretable Distribution Features with Maximum Testing Power. Advances in Neural Information Processing Systems.
Kim, L., Ramdas, A., Singh, A., and Wasserman, L.. (2021). Classification accuracy as a proxy for two-sample testing. The Annals of Statisties, 49(1):411-44 4334.
Kingma, D. P. and Ba, J. (2015). Adam: A Method for Stochastic Optimization. International Conference on Learning Representations.
Kirchler, M., Khorasani, S., Kloft, M., and Lippert, C. (2020). Two-Sample Testing Using Deep Learning. International Conference on Artificial Intelligence and Statistics. PMLR.
Kolmogorov, A. N. (1933). Sulla determinazione empirica di una legge di distribuzione. Inst. Ital. Attuari. Kuiper, N. H. (1960). Tests concerning random points circle. In Nederl, Akad. Wetensch. Proc. Ser. A.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient- Learning Applied to Document. Recognition. Proceedings of the IEEE.
Lhéritier, A. and Cazals, (2018). A sequential non-parametric multivariate two-sample test. IEEE Transactions on Information Theory, (3355333613337))
Liu, F., Xu, W., Lu, J., Zhang, G., Gretton, A., and Sutherland, D. J. (2020). Learning Deep Kernels for -on-Parametric Two-Sample Tests. In International Conference on Machine Learning. PMLR.
Lopez-Paz, D. and Oquab, M. (2017). Revisiting Classifier Two-Sample Tests. In International Conference on Learning Representations.
Lueekmann, J.-M., Boelts, J., Greenberg, D., Goncalves, P., and Macke, J. (2021). Benchmarking Simulation- Based Inference. In Proceedings of The 24th International Conference on Artificial Intellggnce and Statistics.
Emotion.
Mann, H. B. and Whilney, D. R. (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems. Curran Associates, Inc.
Miller, B. K., Weniger, C.,, and Forré, P. (2022). Contrastive Neural Ratio Estimation. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A., editors, Advances in Newral Information Processing Systems, volume 35, pages
Curran Associates, Inc.
Podkopaev, A., Blöbaum, P., Kasiviswanathan, S., and Ramdas, A. (2023). Sequential Kernelized Indepen- dence Testing. In International Conference on Machine Learning, pages 27957 27993. PMLR.
Podkopaev, A. and Ramdas, A. (2024). Sequential Predictive Two-Sample and Independence Testing, Advances in Neural Information Processing Systems, 36.
Radford, A., Metz, L., and Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. arXiv preprini.
14

---

Published in Transactions on Machine Learning Research (04/2024)
Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. (2023). Game-Theoretic Statistics and Safe Anytime- Valid Inference. Stalistical Science, 38(4):576 601.
Shaer, S., Maman, G., and Romano, Y. (2023). Model-X Sequential Testing for Conditional Independence via Testing by Betting. In International Conference Artificial Intelligence and Statistics, pages 2054-2086. PMLR.
Shafer, G. (2019). The language of betting as a strategy for statistical and scientific communication. arXiv preprint.
Shah, R. D. and Peters, J. (2020). The hardness of conditional independence testing and the generalised covariance measure. The Annals of Slatistics, 48(3):1514 1538.
Shekhar, S. and Ramdas, A. (2024). Nonparametric Two-Sample Testing by Betting. IEEE Transactions Information Theory.
Smirnov, N. V. (1939). On the Estimation of Discrepancy between Empirical Curves of Distribution for Two Independent Samples. Bulletin Mathématique de L L'Université de Moscow.
Student (1908). The Probable Error of Mean. Biomelrika.
Turner, R. and Grünwald, P. (2023). Exact anytime-valid confidence intervals for contingency tables and beyond. Statistics & Probubility Letters, 198:109835.
Ville, (1939). Étude Critique de la Notion de Collectif. Bull. Amer. Math. Soc, 45(11):824.
Virtanen, P., Gommers, R., Oliphant, T. E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Millman, K. J., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carcy, C. J., Polat, I. Feng, Y., Moore, E. W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., and SciPy 1.0 Contributors (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272.
Vovk, V. and Wang, R. (2021). E-values: Calibration, Combination and Applications. The Annals of Statistics.
Wasserman, L., Ramdas, A., and Balakrishnan, S. (2020). Universal Inference. Proceedings of the National Academy of Sciences.
Welch, L. (1947). The Generalization of 'Student's' Problem when Several Different Population Variances Involved. Biometrika.
15

---

Published in Transactions on Machine Learning Research (04/2024)
A Proofs
Proof of Lemma 2.2
Proof. By Fubini's theorem we get:
E
= / Fi (x, (Po @ Pv) (dx, dylz)
/ ( (vlz)
Po(dxl 2) (dyl))
/ ( E (xly 2) Poldx w,
F (v/=) Po (dyl=)
/ FO (dyl=) 1
Lemma A.1 (Convex combinations of conditional E-variables). If E and E are two conditional E-variables w.r.t. Ho C ? and g Z + [0, 1] a measurable map, then:
E(al=) -0()) E(xlz) + (1 9(=)) E(x|2),
also defines a conditional E-variable w.r.t. Ho.
Proof of Lemma
A.1
Eo E =]
/ P(
(o(>) E(xlz) + ((1-g(()) E(alz)) (d]])))
= g(x)| / ((ubt)
(1 ((=)) / E(al=)
S g(2) 1+ (1 (1-g(=)) 1=1
Proof of 2 being an e-variable
Proof. For 0g we have:
Eg
E F(m)
2. (<m)
f (2(m)) (<m)) 1 ( ((m))) (mlz(m))
/ pa (m)) (m)))) z(<m))
maxagen Pa(y((m) (y((m)) po (g((()))) (m(m))) pa (md))in))
< (mm))))) 2((m)
pe (m( (m)) (ld
(o(y(mn) ly(سm))
pa (m)) , (((m)) mm) = 1.
16

---

Published in Transactions on Machine Learning Research (04/2024)
Lemma A.1. Let E1...., ED D E-voriables with respect to the some null hypothesis class Ho. Then, the average over all D E-variables is R-variable F. : D Ei= Fs is E-voriable.
Proof. Let po Ho. Then, for E we get
Fo F.
/ ( Fa(x) (o (o())(a)()))
Σ
((()))(())m(dd)) S
1 D
1=
1=1
Proof of Corollary 3.2 First, here shortly review Ville's inequality:
Theorem A.2 (Ville's Inequality, see Ville (1339)) )- Let (S(M) be a non-negative supermartingale,
S(M) (St, Bn, P) [0, oo], M N, w.r.t. filtration J (FM)MEN Far Bn. Then for every s > we
have the inequality:
P IM E N. S(M) 25) E[S(U]
Proof. The sequence (E(SM) MEN constitutes non-negative mapernmartingale of E-variables w.r.t. the filtration = (o(X(()))) due to the following computation for Po E Ho:
M+1
Ea [E(M++) {(كا)
II m==1
((x(m))/f (((m)))))))))))))))) ((dz(M+1)
M
f (M)
II ((X(m) ( ((())) uldx (M++1)
= II ((x(m))a(em) / E E(X(M+1) (2( o ((dx(())1) m=1
M
M
< II ((x(m)))2(<m) (<<))) 1 m=1
(MSM) (x(MM)) 1.
By Ville's inequality, see Thm. A.2 we get:
Po (3M E N. E(SM)
2 2a1)) sa.
17

---

Published in Transactions on Machine Learning Research (04/2024)
B Type II Error Control
A general finite sample bound for the type II error of testing based the product of (conditional) i.i.d. E-variables can be achieved by Sanov's theorem, see Csiszár 19884] Balsubramani (2020)
Theorem B.1 (Conditional type II error control for conditional i..d. E-variables). Let E - be conditional variablle w.r.t. Ho given Z, Let X X, Ωx 2 + X' be conditional random variables that are i.i.d. conditioned on Z. Let E(M) II== E(X, Let a (0, 11, YN Ñ log a 2 0 and for r Rgo put:
A = (QQEP(2) (EX)))EXxxlQ Ex= [(x
Then for every Po E Ha and E Z we have the following type II error bound:
Po (E(M) S a a 6 == =2) exp (-N- C- (L(A)
which converges to ( > 0 for some 7 0. Note that for a subsel A S P(X) we abbreviale:
KT((AIP) = inf (II(Q))).
QEA
Proof. If PN = 1 +EEn 6x,12 the empirical distribution then we get the following equivalence, when N
conditioned on Z =
E(N)& aa E(X,I2) Sa a
log E ((xmlz) S loga =: YN
P- [log E [([[=)] SN
p N
E Ali
The bound then follows by simple application of Sanov's thcorem, see Csiszár 1984 for each z Z individually:
Balsubramani 2020
Po (a = (Px Ale 2 =2) s exp (-N (NN- KL IIP)))
<a-1|z
which requires the i.dd. assumption (conditioned on Z) and that Ali,, is completely convex, which it is.
Lemma B.2. Consider the siluation in Theorem B.1 and fir 2 E 2. Then the first statement implies the
second:
1. KL(A 7(2) (Pp)) for some 7(2) 0.
E Exwp, [log E(X|=)] 0. 0.
If. furthermore, sup=ex log F((x|=)] < oc then the set Ali is TV-closed in P(X) for every 7(x) 2 0. In this case, the second statement also implies the first one, where then have the implication:
0 72)) < E x [log E(X|2)
KL(A 0.
Proof. = ": E xx [log E(X[=)] < then p| E Ao. Since Ao S A, for every 7 > we get:
[ 0 for every 7 0.
18

---

Published in Transactions on Machine Learning Research (04/2024)
":"; Assume C == sep llog E(x|2)] < oo and 7 0. Let Q E P(X) be TV-limit point of sequence P. E Ay, n E N. Then we have the inequality:
Ex~oloog [(X[2)]
= Exoqllog [[ - Ex~P, flog [(X + Ex~P, flog [
v Ex~o (log [[ - Ex-P, [log [ b)] + ExwP, [log E [)]
2 C- Pw) +7
7%
This shows that Q E A as well, and, thus, A TV-closed. By way of contradiction now assume that E Ex (log [x > 0, but (L = 0 for all y > O. Since A is TV-closed and (completely) convex we have that P E A, for all > 0. So we get:
E Ex log E(X S,y
for all y > O, and thus: Ex_ri_ flog E(X < 0, which is a contradiction to our assumption.
The unconditional version follows from the above by using the one-point space Z {*} and reads like:
Corollary B.3 (Type error control for i..d. E-variables). Let XI...., Xx be an i.i.d. sample, E X - Rz0 be an E-variable w.r.t. Ho and E(N) := II n=1 E(X,). Let a E (0, 1], YN == - log a 2 0 and for E Rzo put:
A, == (Q E P(X) E] S 7)
Then for every Po E HA we have the following type II error bound:
Po (F S sa S exp Oxxp--NKL(AA,
which converges to 0 KL(A > 0 for some O.
Relating to the simpler unconditional case of the Corollary we can make the following clarifying remarks. Remark B.4. 1. The condition: KL(A,||P) > 0 for some 0, is slightly stronger than the condition: show that both those conditions equivalent.
Ep, [log E) 0. If supzex log B(x) < then
If there exist 8, y > such that for all Pa E HA we have KI((A, 2 then we easily deduce the uniform type II error boand for N 2 loga
sup Pe (E(N) san s sexp(-N--
PoEHA
we can also get a type II error control for conditional i..d. E-variables we assume that the distribution for the conditioning variable is a marginal part. of the hypothesis.
From Theorem B.1
Corollary B.5 (Unconditional type error control for conditional i.i.d. R-variables). Let E: X' x 2 -+ Rz0 + Z be a faxed random variable with volues in and let X1..... X,= Ω X be random variables that are i. conditioned on Z. Let E(N) II E(X,lZ). Let (0, 1), 7N = - log 0. Then for every Po HA we have the following type II ervor bound:
be a conditional E-variable w.r.t. Ho given Z. Let
Po (E(N) sa-t) s Eo exp (-N KL(AALL [1P
S exp (-N inf (A IP!) ZEZ
where the middle term converges to for N + 00 if for aalmooll 2 E Z there erists some ? > O such that (T](AA]:||/) 0. The latter is e.g. the case if infzez (TA p/²) > 0 for some 7 >
19

---

Published in Transactions on Machine Learning Research (04/2024)
Proof. The inequalities directly follow from Theorem B.1 by plugging the random variable Z into and
laking expectation values:
Po [a = Eg Po E(N) S aa sa-12))
<a
(-N- KIL |P(
S En
exp
-N KL KL 11Pb)))
s sup
exp
Polo
exp
-N
inf z~/6(2)
KL(A! (P))))
S exp
-N inf ZE2
KL 7. [1p/5)
Here (u) and Po(2= denote the essential supremum, essential infimum, resp., w.r.t. Pz(
The statement of the convergence follows from the dominated convergence theorem and the observation that. for every z E Z we have the trivial bounds:
S exp (-N KLN (1P)))) S 1.
This shows the claim.
B.1 Type II Error M=2
Theorem B.6 (Type-II error control for conditional E-variable for singleton Ha). Let Ho = {Po} be a singleton sel. Consider a model class HA and a learning algorithm that for every realization x (In)nEn E XN and every wamber N(1) E IN fits model plz(!) E P(X) to the first N(1) entries x11) (In)neru of Assume that for every Po HA and Po- Pe-almost every i.i.d. realization (In)NEN of Pa there erists number (x) N and e(x) such that for all ((1) (z) the model p(z(() P(X) has density pa (x))))((1)) and satisfies:
KT.(PolPPA < (T((Po|PP)) - <(x))
sup log F(( (1)0 < oo. #.Ex'
Then for every NO)NeO N we have the bound:
Po ( well <a-t) < Exxuip eexp (-NO)
(-NO)KL(41X
KL(AXx0) Txto (PP))
which converges to zero for min(N() N(2)) 00.
Proof. The bound directly follows from Corollary B.5 Note that by the independence assumptions, we have
pla")) Po. Then note that for Po-almost-all E N and for N(1) 2 (Z)
Ex,~Po [og E(x,) E(X,
KL KL(Po|[Ph (1) e(x) 0.
By assumption and Lemma B.2 we have that for Po-almost all x E N and for N(O > N() and for (xx1))
with:
0 < <((11) < (xx) < Ex.mp, [Lo E(X,,
we have: (L(Alz0()
PP) > So for (2) big enough we get:
Yxt) == N22
log < c(r).
20

---

Published in Transactions on Machine Learning Research (04/2024)
This shows that for:
N(2) > -loga (x),
c(z)
we have: (LA (1 xrta) PP) > O. This shows that for Po-almost-all x E X we have that:
exp
(-N12) KT wray [IPo))
0,
for min(N(), ,N(2)) ((2)) _
Since we always have the trivial bounds:
0 S exp (-N(2) (L (y()
(Po)) S
the theorem of dominated convergence tells that we also have the convergence:
Ex exp (-N(9) (T.((A)))))
(N(z) [o))] - O, for
min(N(") - 0o.
This shows the claim.
Lemma B.7. Let PA x x (1) ,0)) = (1 2) y0)) + + Po(ylv for X (0,1) and y E (0,1).
Then the conditional E-voriable defined by
PA ar, (1)), 12 Po(yly(2)
B(x, (1) ()
X + A)E gl1
(11)
is bounded, i.e. || log Bl < cc.
Proof. For every I(2) X' y and every (z,y) I(2)
log E(z, (1))) = log (x+ (c1=x)((z, w1 y,))) .01)))
log A
and
log E(x, () (1) = log
() + (1 A)E(I, 21) (1
1
< log
x
minz Py y(2))
1 loogol + (11-)) 2)
< log
x
1/N(2)
Corollary B.8. Let the assumptions about the learner in Theorem x = (In)nEN E NN and every number N(1 E IN the learner fits a model P E P(X) to the first N entries x(1) (En)neze) of x. Assume that for every Po E Hr and Po-almost every i.í.d. realization
B. hold, t.e. for every realization
(Ia)NEN of Pa there exists a rumber (x) E and e(x) > such that for all ((1) (2) the model pla E P(X) has density pa(zz)z(!) and satisfies:
KL((P|IPAA)) KL(Po]|Po) - c(x),
Then the statististical test w.r.t. E(z, M)) ())) from Equation 11 is consistent.
Proof. The claim follows directly from Theorem B.6 since É(x, () D) is bounded according to Lemma B.7
21

---

Published in Transactions on Machine Learning Research (04/2024)
B.2 Type II Error Control M o0
Theorem B.9 (Strong Law of Large Nunbers for Martingale Difference Sequences). Consider the probabilily space (R,F,P) Let (Xn)neN be sequence random variables that satisfies for some r 2 1
Ep | 2r] nt <0
Consider the natural filtration Fn = o(X1, X,) C F. Additionally, let En = for all n E N.
Then, it holds Ek= Xx + O.
Proof of Theorem 3.4
Proof. Consider the stopping time adapted to the natural filtration FM = ((X(MM))) given by
T [ 2 0: E(SM) 2 1/aj.
For this stopping time we have the equivalent relation
T << 3M 2 0 such that E(SM) 2 a-
The probability that the null distribution will not be rejected in favor of the alternative is given by
PT oo) P&(VM 2 20: E(SA) <a- Po VM 0: 1
Σ
log F
log
M
M
Next, define the random variables W
II E(cz/okemm))
W.. = Ea [log E(m] = Ep
log
Fm-
(EI(m)
= Eo log Po (o(
KL x(cm)
po (m((mm)) 100 (2((m)))
rm
Then the above probability equals
Pa
VM 2 0:
M
log (mm)
<
log a -1
= Pa
VM 2 0:
M
log E(m)
Wm +
M
Wm
log a
M
M
M
M
M
m2
m=l
m=l
M
M
1
log a
< Pe
VM 2 0:
log k(m)
-mm +
Σ
Tm
< 0
M
M
M
m=l
m=l
Next, we will prove that 1 EM log E(m) - Wm - 0. Note that M log ((m) Wm is martingale w.r.t. the filtration FM with bounded martingale differences log E(m) W. in L2. This results from the boundedness of log F(m);
M
(log ((m) Wm W
(log E(m) Wm) < 4s2
sup (mn) (mn) (ex)ne)((m)
It follows that
El (log (m))
W. 21
4s2 m2
m
m
m=
22

---

Published in Transactions on Machine Learning Research (04/2024)
With Theorem B.9
get that Σm= log E(m) Wm " 0. This implies
1 M
log -1
M Σ
lim sup -00
log E(m) Wm
+
Tm =
=0-0+0 0,
M
M
M
m=1
m=1
where lim M Em=1
M
I' M
>
Note that the sequence can even diverge, i.c. == +00. Thus,
1 M
M
loga
log
Po
PM 2 0:
log ((m)
Wm +
Σ
mm
<
0
M
M
M
mml
log
Po
lim
Σ
log ((m)
W.
Σ
I'm
0
0.
M
M
M
m=1
It follows that PQ 00) = 0.
Proof of Lemma 5.1
Proof. By Lemma B.7 it follows that the defined E-variable is bounded with bound depending on the batch size. With [Z(m] < for all m the statement follows directly from Theorem 3.4
23

---

Published in Transactions on Machine Learning Research (04/2024)
C Expected Log Growth Rate
Theorem C.1. Consider the sequence of E-C2ST E-variables (E(SM)) with increments E(m) for
m m=l,... M defined as in Equation 99 Let P(X,Y) denole the true joint distribution of X and Y with probability density function p(a,y) y) = p)(x)
Then it holds for all M 21
y(gan [log E(SM)] < | ( ((
Proof. For any M 2 we get duc to the independence of the observations:
E yx(sm),y(MM) log E(SMD]
E Ex(sm),y(sm) [log E{m)
Σ E Extsm), v(sm) log E ((m)
Ex(cm) y(cm)
[xtm)
[log (rm)
| (x(m), y(em)]
Let P be the estimated joint distribution of X and Y under the alternative with ppzx, where p(x) is the unknown marginal distribution of X and pa(ulo, (al,,,,,(<MM) x(<M), , ((M)) is the learner trained on x(<M) (orsM,yy<<M, y(<M), We get for the increments
probability density function PA (x, v) = DA (yla, (A x(<M), y y(<M)))(z))))
Da (a(y(mm)a (e(m), x(<m), y (mm))
Ex(m), (())
log (mm) ( (x(m), ((mm) = Exim),
log
p(y(m) (o(()))
log PA Pa (x(m) xllm), (((m) yp (y("m)) (m) (x (p(((())) plymm) 100 (m(()) }) (m)) (c(( p(y)(m)) (m)) x (m)) y (m)))
= -[
PA (y( (m)
log Hylmo)//l
log p(y(m))
= =xx y(m)
log
+ E X y(m)
na plymmyy
+ E Ex(m) y(m)
(y(m))|((m)
((y 100
log
p(ylmm))
(L(p(m)) p + (l(m) I(X;Y) + Eyom)
plyims (8 (y(m)))
where
(y (m)) x xm(m) yf y(mm) )p (m(m)))) (m))
(L( p(mil(<m)) [ = -Ex(m) y(m)
log
p(y(mn) ( )
Plugging this into the above expression for Extsan, yisso
[log E(SM)) we gel
Ex(sm)yy(sM) (ogglss))
Extem)
( + |I(SMO) I(X:Y)
log (()
Σ Ey
p(y(m)) ()(()))))) p(y(())0((((mm)))
First, note that (00 2 (m)) for any m and thus Ey
log
(my) p(y(m) 18 (x)()))
< 0. Together with
the conneeiativtty of the KL we get that
E Extsan, y(sa) [log BiSM] < <-0+ - + - - - Mb I(X;Y) for constant batch size)
24

---

Published in Transactions on Machine Learning Research (04/2024)
Figure 6: The two classes the Blob dataset.
Experiments
In this section, we explain the implementation and training our models in detail. In Section D.2 we discuss the architecture choice and training of ET-2STT and the other baseline methods for the synthetic and image data experiments. Code will be provided upon acceptance.
D.1 Baselines
We compare E-C2ST to the following baselines.
S-C2ST (standard C2ST), is the C2ST proposed by Lopez-Paz and Oquab 2017 We train binary classifier on the augmented data. The null hypothesis is that accuracy is 0.5 and the alternative is Ihat is larger 0.5. The p-values is computed via a permutation test.
L-C2ST (logits C2ST) proposed by Cheng and Cloninger (2022) is a kernel based test, which again trains a binary classifier to distinguish the two classes. The mull hypothesis is rejected if the difference between the classes logits average is not significant. The p-values is computed via a permutation test.
M-C2ST We conduct the tests by means of the proposed test statistics based on maximum mean discrepancy (Kirchler et al. 2020). The p-values computed via permutation test.
D.2 Training
We used Adam optimizer (Kingma and Ba. 2015 with learning rate 1- 1e (and 5 1c - 4 for the Blob data). For fitting the parameter À from (10p used L-BFGS-B (Byrd et al. 1995) implemented in (Virtanen ct al. 2020) and we set the initial value to 0.5 unless specified otherwisc. Note that in all experiments we consider a paired two-sample lest for simplicity, iee. each observation consists pair X and Y that possibly come from different observations.
Blob data. The two Blob distributions used in the corresponding type 2 error experiment are visualized in Figure 0 The means the same for both classes and are arranged in a. 3 3 grid. The two populations differ in their variance. The used network architectures are described in Table We trained the models with carly stopping with patience 20 for all methods in all cases.
MNIST. The dataset is obtained from https://github.
Table 3
outlines the neural network architectures. We trained the models with early stopping with patience of 15 epochs for the baseline methods and 10 for E-C2ST.
Face Expression Data. For all methods we used the network architecture provided in Table ! We set. the patience parameter to 20 epochs.
D.3 Additional Experiments
Best models according to the ablation study. In the main paper, we performed two experiments investigate the effect of batch size and the effect of initial lambda value. Here summarize our results by
25

---

Published in Transactions on Machine Learning Research (04/2024)
MNIST
KDEF
1.0
1.0
C.CT b2-32 E C2ST, b4 L-C2ST
0.6
0.6
0.4
0.2
0.2
0.0
0.0
230
500
730
1000
1250
1500
200
3001
400
500
600
700
Samples
Figure 7: Comparison between the E-C2ST trained with batch size=32 and the initial E-C2ST and baseline C-C2 We observe a significant increase power when using batch size of 32.
MNIST
KDEF
1.01
1.1
0.8
Power
0.6
0.2
0.2
200
400
600
800
200
300
800
Figure 8: Robustness to batch order. Here we visualize the mean and 95% confidence interval based on 10 batch permutations of the power curves computed from 100 independent runs.
comparing the best F-C2ST performer according to the ablation studies with the best baseline T-C2ST Figure 7 For both MNIST and KDEF, we can conclude that there is a significant gain performance by using the enhanced E-C2ST.
Robustness the batch order. We aim empirically assess the robustness of our statistical test changes in the order of data batches. To achieve this, we design a sequential experiment in which iteratively collect new data batches, stopping the process when the null hypothesis is rejected. This experiment is repeated 100 times, with each iteration involving random shuffling of the predetermined data batches, repeated 10 times to generate 10 different sequence orders. For each sequence order, we calculate the power of the test based on the results of the 100 independent runs. We then aggregate the results from all 10 sequences by calculating and visualizing the mean power curve and its corresponding 95% confidence interval Figure S for the MNIST and KDEF datasets, where the batch size is 64 and the initial 0.5. Figure S shows that. the observed deviations from the mean are small in both scenarios. This indicates that our method is robust with respect to permutation in the order of the data batches.
Wall clock time. We performed comparative analysis the exccution time, measured in soconds, for the E-C2ST and L-C2ST algorithms. This analysis based 100 independent trials with different sample sizes within the KDEF and MNIST scenarios, shown in Table [ In this context, data were generated under the alternative hypothesis. Both E-C2ST and L-C2ST were trained with identical training parameters (learning rate, patience, network architectures). Batches of 64 samples were used for E-C2ST. For L-C2ST, the total sample size shown in the second row of the Table I was divided according to a ratio we used in our experiments, Le. 5:1:1 for training, validation and test data, respectively.
We do not inchude the other basclines, since their execution times are comparable those of L-C2ST. This similarity in execution time is due to the fact that all baselines, including T-C2ST, the same trained
26

---

Published in Transactions on Machine Learning Research (04/2024)
model and the same number of permutations to compute the corresponding p-values. All experimental runs were performed on NVIDIA GP102 [GeForce GTX 1080 Tij GPUs.
KDEF
MNIST
64-3
64.6
64.9
64.
64
64 10
64-14 14
64 : 17
E-C2ST
1.5 0.6
1.4
5.6
2.5 0.4
4.9 + 0.8
10.5 1.8
11.5 1 2.8
11.6 1 2.9
L-C2ST
2.7
2.1
3.4
3.5 0.7
4.4 + 0.98
7
11.6 3.99
15.96 + 5.1
Table 1: Wall Clock Times in Seconds
Our discussion of the results is organized around three different scenarios: 1) when the number of data batches equals three; 2) when the total sample size falls below the threshold required to achieve maximum statistical power; and 3) when the total number of samples exceeds the minimum data set size required to achieve maximum power.
Looking at the first columns of the Table ! for both the KDEF and MNIST scenarios, we see that E-C2ST requires less computation time compared to I-C2ST when the dataset is segmented into three batches. In this case E-C2ST performs a single training iteration (as L-C2ST) utilizing the three batches for training, validation, and testing, respectively. Then, the reason for the increased computational cost for L-C2ST can be attributed to permutation test used for calculating the p-value that uses 500 permutations.
In our work, as demonstrated by our experimental results, including those shown in Figure empirically determined the optimal sample sizes for maximizing the statistical power of F-C2ST when using batch size of 64. In particular, we found that approximately 384 samples (equivalent to 64 x 6) are required for KDEF and 896 samples (oquivalent to 64 x 14) are required for MNIST. This indicates that to effectively reject the null hypothesis, E-C2ST requires a maximum of and 14 batches the KDEF and MNIST scenarios, respectively.
S have
When the available data set is smaller than these thresholds, E-C2ST becomes more computationally expensive than L-C2ST (see the second and third columns for the MNIST scenario). This increased computational cost occurs because E-C2ST updates the model with cach additional batch of data, using the full dataset available at that time, whereas L-C2ST trains the model only once using the train, validation, test data split.
However, when the dataset size exceeds the identified thresholds for achieving maximum power with E-C2ST, E-C2ST becomes less computationally expensive than L-C2ST (sce the second and third columns for KDEF and the fourth and fifth columns for MNIST). This efficiency gain is due to E-C2ST's ability to terminate further training upon early rejection of the null hypothesis in large datasets. This means that for datasets larger than the specified thresholds, the computational time for E-C2ST becomes constant and does not increase with additional data, providing a significant computational advantage over L-C2ST.
Why can't we use p-values for sequential testing? Traditional statistical tests, such the t-test chi-squared test, assume that the number of observations or experiments is determined before data collection begins. This setup means that researchers decide in advance how much data to collect, and that predetermined sample size does not change no matter what the outcome of the test is. Sequential testing, on the other hand, allows data to be analyzed as it comes in. This approach allows researchers to make decisions about whether to continue collecting data at any point during the study. Therefore, soquential testing focuses on designing tests that can keep the lype I error rate below a certain significance level throughout the data collection period. This is much more strict requirement than in classical statistical testing, where controlling for type error is only necessary for a single, fixed sample size.
Standard lesting methods are not directly applicable to sequential testing scenarios. In situations where data arrives in batches over time, recalculating the p-value for the entire dataset each time new data arrives and making testing decisions based on that recalculated value likely to push the type error rate above the alpha threshold. This increased risk due to the fact that performing multiple tests on the same data increases the likelihood of incorrectly rejecting the null hypothesis. We illustrate this in the following example.
27

---

Published in Transactions on Machine Learning Research (04/2024)
I-CIST appled sequentially
1.0
0.9
Type Emor
9.0
200
400
1000
1200
1800
Samples
Figure 9: Type error of T-C2ST, applied on a sequential testing task. We fix the number of batches at 20 and compute the type error over time, as the number of batches revealed to the model increases.
Estimated
from independent cumulative
085
0.6
Figure 10: Estimated type I error from 100 independent sequential experiments for batch sizes randomly selected in the interval [32, 64]. The estimated cumulative lype I error is way above the significance level a = 0.05 and naturally increases over time.
We ran a sequential testing experiment, where in each round of this experiment, two samples were drawn, with their sizes randomly chosen from the range [32, 64). Both were sampled from standard normal distribution.
As new batches of data arrived, t-test was performed to compare the means of the samples collected up that point (this includes the current. batch and the past data). The test was stopped if the null hypothesis was rejected at significance level of 0.05; if not, new samples were drawn until the null hypothesis rejected at that level.
This procedure was repeated 100 times to estimate the lype I error rate over time. Our experiment shows that the type I error increases over time and reaches a level of 60%, as shown in Figure I0 Let's look at the implications for the tests we are considering this work. If we use the same batch splitting technique for any standard method
do for E-C2ST, we compromise the type error guarantees that would be in place if caleulated the p-value only once. Tn particular, if we have M batches and thus recalculate a p-value M times, the standard methods face M times more opportunities to falsely reject the null hypothesis. This significantly increases the likelihood of inflating the type I error rate when evaluated over the entire data set. For example, Figure g shows the increasing type error of L-C2ST as the number of batches increases, where the maximum batch number 20 and the batch size is 64. Thus, for M=20, in this sequential setting, the type error of L-C2ST is above 0.6 >> 0.05.
On the other hand, the way F-C2ST is designed addresses this problem, i.e., it ensures that the type error in this remains below the prodetermined significance level. See Corollary 3.2 for these thcoretical guarantecs.
28

---

Published in Transactions on Machine Learning Research (04/2024)
Layer (type)
Output Shape
Linear-1
[batch size, 30]
LayerNorm-2
[batch size, 30
ReLU-3
[batch size, 30
Linear-4
[batch size, 30)
LayerNorm-5
[batch size, 30)
ReLU-6
[batch size, 30)
Linear-7
[batch size,
Table 2: The network architecture employed in the Blob experiment for all methods
Layer (type)
Parameters
Conv2d-1
16, kernel (3, 3), (2r 2), padding-(1,, 1)
LeakyReLU-2
negative slope=0.2
GroupNorm-3
eps=1e-05
Conv2d-4
32, kernel size=(3, 3), stride= 2), padding=(L4 (1, 1)
LcakyReLU-5
negative slope=0.2
GroupNorm-66
eps=1e-05
Conv2d-7
64, kernel size=(3, 3), stride=(2, 2), padding=(1,) 1)
LeakyReLU-8
negative slope=0.2
GroupNormm-9
eps=1c-05
Conv2d-10
kernel s3ze=(3, 3), stride= 2), padding=(1,
Table 3: The network architecture employed in the MNIST experiment for E- C2ST,T, L-C2ST, S-C2ST, M-C2ST,
Layer (type)
Parameters
Linear-1
size=32
LayerNorm-2
ReLU-3
Dropout-4
0.5
Linear-5
size=32
LayerNorm-6
ReLU-7
Dropout-8
0.5
Linear-9
size=1
Table The network architecture employed in the KDEF experiments for all baselines.
29


Reviews:
[{"review_id": "Kk6P80DNZF", "reviewer": "Reviewer_8rXQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper tackles an interesting problem which is to perform two-sample tests for sequential data and to provide the test result at anytime\n+ The writing is generally clear (although I still have some comments regarding the writing, please find in the Weaknesses section)\n+ The proposed methods are developed from various solid works in the literature. The proposed method seems to be reasonable.\n+ Theoretical analysis for the proposed method are included (although I couldn’t check in detail)\n+ The proposed method seems to be perform well in the conducted experiments\n\nWeaknesses:\n+ I find it sometimes hard to distinguish between the existing works and the new method presented in this paper, especially for researchers that do not work directly in this topic. For example, when introducing the conditional e-variables in Section 2.2, it’s unclear whether this concept was already proposed in (Grünwald et al., 2020; Vovk and Wang, 2021) or was developed in this work. Similarly, the concept of M-Split Likelihood Ratio Test, the predictive conditional independence testing, and other theoretical analysis are not clearly described if they’re new in this work or from existing works. The writing should be modified much clearer to distinguish between existing techniques and the new proposed method.\n\n+ In Section 3, bullets 1 and 2, why in bullet 1, an arbitrary method can be used but then in bullet 2, MLE needs to be used? And normally, how the probability p_\\theta is chosen (e.g. Normal distribution or any other distributions)? How can we ensure p_A reflect the true distribution on unseen data? \n\n+ In Section 5, the methodology to perform the hyperparameter is not clear to me. Why do we have the formula in Eq. (11)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a deep classifier two-sample test for high-dimensional data based on e-values. The main idea is developed from the split likelihood ratio tests and predictive independence tests. The proposed method can provide anytime two-sample tests for sequential data at any time point. The paper provides some theoretical analysis to understand the behaviour of the proposed method. The paper also conducts various experiments through some simulation and real data applications to demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ The writing needs to be modified in order to distinguish the existing works and the new methods and contributions proposed in this work.\n+ Clarifications regarding my questions in the Weaknesses section (please refer to the section above).", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BcuH5CRJe5", "reviewer": "Reviewer_RA3k", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. The paper is exceptionally well-written and flows by itself. (It is one of the best-written papers I have reviewed recently.)\n2. The theory is well-explained, and sections §1, §2, §3 provide a good introduction for a reviewer who does not know the area well (that is me).\n3. The experiments make sense and are convincing. The ablation addresses precisely what I wanted to ask for after reading the earlier sections.\n\n### Weaknesses\n\nWhile writing this up, I finally understood that the conditional E-variables are only used via the chain rule for the split likelihood piece. While reading it initially, this did not become clear from the main text.\n\nSee below for more fine-grained remarks.\n\n### In Summary\n\nThe claims are evidenced, and it will be of great interest to readers of TMLR, given the increased data efficiency. This will be of interest in online settings.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a classifier-based two-sample test that is more data efficient than prior tests by making use of sequential testing. It explains E-variable-based hypothesis testing, conditional E-variables and hypothesis testing, split-likelihood ratio tests, before expanding on these with conditional predictive independence tests to introduce the E-Valuating Classifier Two-Sample Tests (E-C2ST). \n\nThe paper provides convincing experimental validation, comparing convincingly to other literature.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Abstract: mention \"E-Valuating Classifier Two-Sample Tests (E-C2ST)\" explicitly instead of \"E-C2ST\" + mention \"sequential testing\" explicitly in the abstract as well to increase searchability.\n2. Mention the chain rule of probability around Lemma 2.2 in §2.2 on Conditional E-Variables. It seems to provide good intuitions for what is going on?\n3. Below (5), there might be a typo. There is a $P(...) \\to \\infty$ --- should it $\\to 0$?\n4. Please refer to the proofs in the appendix from the main text, so it is easy to jump there in the PDF. In particular, \"Proof of (3) being an e-variable\" was vital for my understanding.\n4. For the ablation (Fig 4.), why not run with batch size $1$? I would very much like to see what the trade-off is. (In Active Learning, the lesson is that one has to train for many more epochs than one thinks with very small datasets.) Could you add a figure to the appendix that includes the best baseline as well to make E-C2ST's additional advantage easier to compare?\n\n5. For future work in the conclusion, could you consider the question of data order? There might be some exciting connections to active learning and active testing, which might further boost the data efficiency of this principled approach.\n\n6. For my understanding: How does this two-sample test compare to other methods in ML/deep learning, which measure the mutual information between samples and their distribution: X ~ P_Z, Z ~ Bernoulli, I[X ; Z] ought to measure something similar and might provide another interesting connection for further research?", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sKzy92Dcrj", "reviewer": "Reviewer_zK4n", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ The thorough theoretical analyses seem to be good for archival purposes for future works.\n\n+ The experimental results demonstrate the effectiveness of the proposed method (unless sample size is small; less than ~600).\n\nWeakness:\n\n- Regarding the statement in page 2 \"when applied sequentially, they can lead to inflated type I error. In simpler terms, these methods assume that the sample size is known in advance, which can be a drawback in practice.\":\nwhy not just use random sample size per sequence to avoid the known sample size during testing? I am not sure this is a good reason to motivate the following claims.\n\n- According to the paper, Ramdas et al. (2022); Grünwald et al. (2020); Shafer (2019) already discussed e-value-based sequential tests, and it seems like the proposed e-value-based two-sample tests fall into a specific case of them. Also, authors argue that they \"extend the work of Lhéritier and Cazals (2018) by introducing e-values for conditional independence testing.\" So the title looks somewhat too general and/or overclaimed.\n\n- Regarding the statement in page 2 \"we assume that both hypotheses (H_0 and H_A) occur with equal probability.\":\nI think this is a strong assumption. Is there any problem on generalizing this, i.e., making the odds to be arbitrary?\n\n- There are some hyperparameters that the values are not specified, such that the experimental results are not reproducible. For example, what are the initial value of \\lambda_1 and the significance level \\alpha in the experiments? Also, ablation studies on hyperparameters are mostly missing, except for the batch size.\n\n- The test is sequential, so I wonder if the test result can be different if the order of batches becomes different?\n\n- What is the time/memory complexity of the proposed method and how it can be compared with other works, in terms of big-O notations and/or wall-clock time? Two-sample tests are computationally heavy, so the development at the expense of more costs might not be desirable.\n\n\nOther issues:\n\n- \\theta and \\Theta first appeared in section 2.1 are never defined.\n\n- Is there a specific reason to define the alternative hypothesis as H_A, such that you abuse notations? Why not just use H_1, like the conventional literature?\n\n- In Algorithm 1 line 9, E^(1) is never computed; note that line 8 starts from E^(2).\n\n- (I don't expect an answer from the authors regarding this) I am not 100% sure if this work is interesting to TMLR's audiences. I see that there are several works on two-sample tests in ML venues, but it seems TMLR had a difficulty to find reviewers interested in this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a variation of two-sample tests with deep classifiers based on E-values. To motivate the necessity of the proposed method, authors argue that previous deep classifier-based two-sampled tests suffer from inflated type I error when applied sequentially. To this end, recent works dynamically reject the null hypothesis for different batches, and this is where e-value-based sequential test comes from. Specifically, this work extends Lhéritier and Cazals (2018) by introducing e-values for conditional independence testing. This paper provides some theoretical analyses and empirical observations on small synthetic and real-world datasets to support the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address concerns above.", "broader_impact_concerns": "Nothing special.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission appears to be of good quality and reviewers only had minor comments.

Reviewer RA3k requests: For the camera-ready, I'd only request one more proof-read to e.g. fix: "from (Burkart and Király, 2017) with e-variables from the M-split likelihood ratio test from Section 3 based on (Wasserman et al., 2020)" to use \citet instead of \citep.
Decision: Accept (poster)