AutoML Two-Sample Test
Authors: Jonas M. Kübler
Vincent Stimper
Simon Buchholz
Krikamol Muandet
Bernhard Schölkopf
Venue: NeurIPS 2022 Accept (2022)
Keywords: Two-Sample Test
AutoML
MMD

Abstract: Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts.
This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems. 

AutoML Two-Sample Test
Jonas M. Kübler!, Vincent Stimper', Simon Buchholz', Krikamol Muandet', and Bernhard Schölkopf'
Maa Planck Institute for Intelligent Systems, Tübingen, Germany CCISPA Helmholtz Center for Information Security, Saarbrücken, Germany {jmkuebler, vstimper, sbuchholz, ba) @tue. mpg. de, muadettccispa.de
Abstract
Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts. This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We of a witness function as the test statistic and prove that minimizing squared loss leads to a witness with optimal testing power. This allows us leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on diverse distribution shift benchmark as well as on challenging two-sample testing problems.
simple test that takes the mean discrepancy
We provide an implementation of the AutoML two-sample test in the Python package autotst.
Introduction
Testing whether two distributions are the same based on data is fundamental problem in data science. classical application is to test whether two differently treated groups have the same characteristics not [Student, 1908, Welch, 1947, Golland and Fischl, 2003]. Testing independence of two random variables can also be phrased as a two-sample problem by testing whether the joint distribution equals the product of the marginal distributions [Gretton et al., 2005]. A more recent application in machine learning is to detect distribution shifts, i.e., whether the distribution model was trained on equals the distribution the model is deployed on [Lipton et al., 2018, Rabanser al., 2019, Koch et al., 2022].
Classical methods have a fixed test statistic that makes strong parametric assumptions. For example, Student's 's two-sample t-test only tests whether the distributions have equal mean, assuming both distributions follow a normal distribution with the same (but unknown) variance. With modern datasets, which are often high high-diimnsional,, such test cannot be applied because the strong assumptions are often not justified. Nonparametric kernel-based test such as the Maximum Mean Discrepancy (MMD) [Gretton et al., 2012a] are very flexible and, theoretically, given enough data. However, this generality often harms test power at finite data size. This can simply be understood in terms of a classical bias- variance tradeoff. To mitigate this, it has become common to optimize the test statistic first on a held-out dataset and then apply the test only on the other part of the data [Sutherland et al., 2017, Liu et al., 2020]. However, the derived objective as well as optimizing kernel function are no standard tasks in machine learning and no automated packages exist, making it hard for practitioners to apply them.
detect differences of any kind
Tests that well into the standard machine learning pipeline are based on the classification accuracy. First, classifier is trained to detect the difference between the two samples, and then its accuracy on
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

AutoML: h (Sp, 1); (S 0); MSE, bmax)
Sp: Q
T = hsp his
p Prob/T' r|
Input:5p,, Sq
hsp = h(s 1(($))
Out: p S a
hA hB hA,hB = Permute&Split(h heg
Figure 1: AutoML two-sample test: Sp, Sq denotes the available data from P and Q. which is first split into two parts of equal size. A witness h X' + R is trained using (weighted) squared loss Eq. (6), denoted by MSE, and using AutoMI. to maximize predictive performance. Users can easily control important properties, for example the maximal runtime fmax- The test statistic is the difference in means on the test sets. Permuting the data and recomputing r allows the estimation of the p-values. The null hypothesis p = Q is rejected if p S a.
a held-out set is taken as a test statistic [Golland and Fischl, 2003, Lopez-Paz and Oquab, 2017, Kim al., 2021, Cai et al., 2020, Hediger al., 2022]. Liu et al. [2020] argued, however, that optimizing classification accuracy does not directly optimize test power and considered this one reason why kernel-based test outperform classifier tests. Kübler et al. [2022] challenged this and considered the mean of an optimized witness function as test statistic finding that kernels are not necessary for good performance. Generally, such two-stage procedures are very intuitive and arguably also how human would approach the two-sample problem on complicated data. One could look at some part of the data, try to come up with a simple hypothesis, and then try to test its significance on held-out data.
Despite the recent progress in the theoretical understanding of machine learning-based two sample tests [Kim et al., 2021, Liu et al., 2020], there is still little guidance on how apply these tests in practice and a substantial amount of engineering and expertise is required to implement them. On the contrary, in supervised learning, namely regression and classification, the past years have shown tremendous advancements in making machine learning models applicable essentially without any expert knowledge leading to the field of Automated Machine Learning (AutoML) [Feurer et al., 2015, Hutter et al., 2019, He et al., 2021]. The goal of AutoMI. is to automate the full machine learning pipeline: Data cleaning. feature engineering and augmentation, model search, hyperparameter optimization, and model ensembling [Dietterich, 2000]. All of it with the goal of achieving the best possible predictive performance on unseen data.
The goal of our work is to bring the advancements of AutoML research to the field of two-sample testing. Our main contributions are:
1. We prove that minimizing squared loss is equivalent to maximizing the unwieldy signal- to- to-noise ratio, which determines the asymptotic test power of a witness two-sample test (Section 3.1).
2. Thanks to the former result we can use AutoMI. to learn the test statistic, thereby harnessing the power of many advancements in machine learning such as hyperparameter optimization, bagging, and ensemble learning in a user-friendly manner (Section 3.2).
3. Our test is usable without any specific knowledge and skills in two-sample testing. Users can easily specify how many resources they want to use when learning the test, for example the maximal training time (Section 3.2 and Section 5). Furthermore, one can easily interpret the results (Section 3.3).
4. We extensively study the empirical performance of our approach first by considering the two low-dimensions datasets Blob and Higgs followed by running large benchmark on a variety of distribution shifts on MNIST and CIFAR 10 data. We observe very competitive performance without any manual adjustment of hyperparameters. Our experiments also show that a continuous witness outperforms commonly used binary classifiers (Section 5).
5. We provide the Python Package autotst implementing our testing pipeline.

---

The proposed testing pipeline is described in Fig. 1: First, the two samples are split into training and test sets. Then a witness function h trained by first labeling samples in Sp with and samples from SQ with 0 and then minimizing (weighted) Mean Squared Error (MSE) to maximize test power, see Section 3 for further details. To maximize the predictive performance and to require as little user input as possible, we use AutoGluon [Erickson et al., 2020], an existing AutoML framework, when optimizing the witness. Our test statistic is then simply the difference in means of the test sets Sp, Sq. see Section p-values are computed via permutation of the samples [Golland and Fischl, 2003), which is a standard technique in two-sample testing.
2 Preliminaries
Notation. We consider the non-empty set X C Rd as the domain of our data. We are given samples Sp [II..... and Sq = {y1...... ym J. which are iid. realizations of the random variables X and Y distributed according to P and Q. Let us define = カ+ The proposed approach splits the data into disjoint training and test sets of size Thr Rie, Mar, mhe that we denote Sp, S and $Q. Sq- Unless otherwise stated, we assume that the data is split in equal halves, which is the default approach [Lopez-Paz and Oquab, 2017, Liu et al., 2020].
Our goal is to test the null hypothesis Ho : P = Q against the alternative hypothesis P # Q. Our hypothesis test rejects when the observed value of the test statistic is significantly larger than what we would expect if the null hypothesis were true. Naturally, such tests can make two types of errors:
Type-I: The test rejects the null hypothesis, although it is true.
II: The test fails to reject, although the null hypothesis is false.
Our goal is to design a test that controls the rate of Type-I errors at a given significance level (0, 1), and maximizes the test power defined as I minus the probability of a Type-ll error.
Witness two-sample test. We consider witness- based hypothesis test [Cheng and Cloninger, 2019, Kübler et al., 2022]. Given a function h X - I, called wimess, the mean discrepancy is
Q h) Ex~P [x)] Æy~ [h(Y),,
and we use its empirical estimate on the test set as test statistic
1
1
h(x)
h(y).
The #EST
Me
yE
As show in Section 4, this test statistic can be sample tests [Lopez-Paz and Oquab, 2017]. We assume that c tet
continuous extension of classifier two- Th converges to a constant
as n, m + xo. With (h) (1-c)Varx [kXx)]+taaaygghh(h) we have that the test statistic is e(1-c)
asymptotically normally distributed [Kübler et al., 2022, Theorem 1]
VTu + mi [T(S,, SS h) - (PP.Q h)] $ SN (1))
(3)
Let us for now assume that we know Te For any level a E (0,1) we can set the analytic test (k( $ -1 a), where & denotes the CDF of standard normal and o its threshold to ta The mae inverse. We can then compute the asymptotic probability of rejecting as:
Pr [reject] = Pr [T(S), 1b)n) > ta] - + (VTLe m
mhe T Q h) - -- -0-(1-))) (h)
(4)
Under the null hypothesis P Q we have T(P.Q h) = O. Therefore, Eq. (4) reduces to (-0-1(1 a)) = + ( a)) a. Hence, the asymptotic test correctly controls Type-I eтror. On the other hand, for given P Q, Eq. (4) corresponds to the test power. Since & is monotonically increasing function, the test power is maximized by the witness h that maximizes
SNR(h) T(PQQ1A)
(5)
(h)
3

---

where SNR is the Signal-to-Ne Noise Ratio. From Eq. (4) it follows that the overall testing pipeline is consistent (approaches power if we find witness function with SNR E O with probability going to I as the training data set size grows (see Appendix B.2 for more discussion). Kübler et al. [2022] showed the optimal witness can learned when using kernel methods (using kernel Fisher Discriminant Analysis), but it was left open how this can be done efficiently with other machine learning frameworks. Such an SNR is not commonly implemented and also common approaches like mini-batching are not easily adapted, as plug-in estimate of the SNR based on mini batch would be a biased estimate, In the next section, we show how to circumvent this and optimize squared loss instead.
3
The AutoML two-sample test
3.1 Equivalence of squared loss and signal-to-noise ratio
Since it is known for linear models that minimizing squared loss over two labelled samples is equivalent to Fisher Diseriminant Analysis [Duda al., 2001, Mika, 2003], we attempt to find more general relation between the squared loss and the SNR. Our goal is to use the squared loss as nee the optimization objective when learning the witness. Let +y us mark all data from P with label '1' and all data from Q with label '0'. We define the following (weighted) squared loss
analogously to the above. Let
(6) Note that the weights (1 1-d c) and c are swapped as it will be more important to fit the set with fewer samples. Given a function h, notice that shifting and scaling it leaves the SNR (5) invariant. We can then show the following relationship of its squared loss and its SNR.
(h) = (1-c) (1-c) [(L -1-10) h(x))² + e [(0 )(Y))²)
Lemma 1. Let the function be fixed. We apply the linear transformation h - - + u with y E TR and v E TR. Let (n", u²) be the minimum of the quadratic function (7, (p + v). Then, the following holds true (Proof in Appendix A);
c(1 (11c))
L ++))+=++++++))))) + SNR(h)2
Let assume that the supports of the two distributions P. Q.Q overlap. Hence, for any function h the loss Lp.Q.e is strictly positive. Assume that h" is the function that minimizes the loss over all possible functions. This implies that  1 and n² 0, otherwise could still improve the loss by scaling or shifting. Thus, by Lemma we have:
Proposition 1. Assume that ht minimizes the squared loss (6). Then ht maximizes the signal-to-noise ratio, i.e.,
L(h") min L(h) + SNR(H") = max SNR(h). h
Proof. A solution that minimizes the loss has Ex~P [x)( 2 Ey~q [h (Y)) and hence a non- negative SNR. Assume there exists h such that SNR(h) > SNR(h"). Then Lemma 1 implies the existence of 7, i such that L(5h + 8) < I.(h",, which is contradiction.
We can further derive closed-form solution for the population optimal witness: Proposition 2 (Optimal Witness). Assume P and Q have densities p(x) and q(x)) The function minimizing Eq. (6)
(1 (1- c)p(w:) h" (x)x) (1 (1- c)=))p(z) + cg(x)
Proof. We rewrite Eq. (6) L(h) = (x - (z) dr. Minimizing the integrand for each x yields the claimed result. A similar result was obtained by Mao et al. [2019]. O
Remark 1. Consider the balanced case c = 1/2, iee. equal prior probabilities of labels '1' and 'O'. Then he (x) is the posterior probability that the example came from P. or, using our defined labels.
h² (x)x) Pr [r]x). Thus, minimizing a log loss, i.e. the binary cross-entropy, and using output probability for class I as witness function also maximizes test power.

---

Notice that for * 1/2, we need to weight our samples with the inverse weights, i.e., it is more important to get the less frequent samples right.
Proposition 1 and Remark lead to our main conclusion: optimize the (weighted) squared error or a cross-entropy loss. This allows us seamlessly integrate existing AutoML frameworks, which are designed to solve this task in an automated fashion, to learn powerful witnesses. In the following we mainly focus on the squared error.
To find an optimal witness, we can simply
3.2 Practical implementation
Stage I- optimization. In the first stage, we optimize the witness function to minimize the MSE via the training data S and S. as motivated in the previous section. We simply label the data with 1 or 0 depending on whether they come from P or Q. We can then use any library that implements an optimization of squared loss. If * 1/2 we additionally need to specify weights according to Eq. (6). Note that, unsurprisingly, the relevant quantity for the test power is the loss on the test data and not on the training data. Thus, it is of crucial importance to find a witness with good generalization performance. To make this as simple as possible for practitioners, we propose to use an AutoML. framework. This also has the advantage that users can specify runtime and memory limits, and can explicitly trade computational resources for better statistical significance.
Although we strongly argue towards using AutoML for the test, this can of course not circumvent the no-free-lunch theorem. Thus, whenever users have good intuition about how their two samples might differ, we strongly encourage taking this into account when designing the test. To put it to the extreme: If one knows that their -in-dimesiional) data follows a normal distribution and only differs in mean (if at all), one should use a classic f-test rather than our approach.
Stage II - testing. Given a witness function learned as detailed in the previous section, we compute the test statistic in Eq. (2). To compute a p-value or decide whether to reject the null hypothesis P Q. we can either approximate the asymptotic distribution or use permutations [Kübler et al., 2022]. To estimate an asymptotically valid p-value" we first estimate (h) (see Eq. (3)) based on SV. S8
which we denote as (h). The v-vuuue is then given as 4 (V + The (Sp, S h) (h))
For two-sample tests, a cheap alternative that guarantees correct Type-I error control even at finite sample size is based on permutations [Golland and Fischl, 2003]. In case of witness functions, one can simply permute the values hz11)... h( h(y)).... Mym,,) ) and split them in two sets of size The and mc. respectively. One then recomputes the test statistic over B E N iterations. When the permuted test statistic was T times at least extreme as the original, one needs to use biased estimator p = T+1
B+T to control the Type-I error [Phipson and Smyth, 2010]. We reject whenever p S a. We emphasize that we do not need to retrain the model, and it even suffices to evaluate the witness once on all elements of the test sets. We can then directly permute the witness" values.
Runtime. The overall runtime of the AutoML based witness test is the sum of the runtimes of the training phase, the evaluation of the witness, and the evaluation of the test statistic. We denote the scaling of the former by Swain [Tur + ma). where the square brackets indicate a functional dependency, It will depend on the AutoML framework but can usually be controlled by setting a time limit. Even with a limit of one minute or less AutoGluon can already train powerful models on large datasets and even performs model-selection, hyperparameter optimization, and so on. In contrast, deep kernel-based methods typically train a neural network with a fixed architecture, which can be expensive. Although neural networks belong the suite of models AutoGluon trains, they are optimized for speed and if the runtime limit does not permit training them another faster model will be selected.
The scaling of evaluating h, denoted by Seval [ny + m,], is usually linear in the dataset size, but it can be sublinear if the evaluation is parallelized. It can also be controlled with AutoGluon by using different hyperparameter presets which might optimize the model selection towards fast inference times. Compared to that, kernel-based tests have a quadratic runtime. Furthermore, the test statistic has to be evaluated on the original partition of the data as well as B permutations requiring (nu + m + 1) steps. In practice, this is usually the cheapest step. but could also be further
Asymptotic p-values are strictly speaking only valid for fixed /s as the size of $$, S% goes infinity.
5

---

7
7
2
f
/
4
2 8
O O
O
O
O
O
Figure 2: Testing MNIST against shifted MNIST with 'O's knocked out. The optimized witness assigns the highest values to the images on the left, and lowest values to the images on the right, allowing us to interpret the difference.
reduced by parallelization. The overall runtime is given by
O (Strin My + me] + Seval (re + miel + (ne - m 1))
(8)
Generally, training the witness will be the most expensive step of test. A main advantage of our test over others is that practitioners can easily trade-off spending more time and resources on the training phase potentially get a better witness and thus more significant results. Thanks to AutoML, specifying the time and resources does not require any detailed knowledge of the underlying algorithm and is hence easily done.
3.3 Interpretability
Suppose our test finds significant difference between Sp and Sq. An additional task would be to interpret how the distributions differ. This particularly simple in our framework and shown in Fig. 2: We can check which examples attained the highest value of the witness to find which inputs are much more likely under P than under Q. On the other hand, inputs with small witness values are more likely under Q. Similar procedures were used in Jitkrittum et al. [2016], Lopez-Paz and Oquab [2017], Rabanser et al. [2019]. An additional advantage of using the AutoML framework AutoGluon is that it allows to compute feature importance values easily. Therefore, for datasets which are hard to visualize the important features data points with high or low witness values can identified.
Relation to prior work
Gretton et al. [2012a] introduced the maximum mean discrepancy test statistic for two-sample testing. For a given reproducing kernel Hilbert space H. the maximum mean discrepancy is
MMD = max Ex-P [h(x)] Ey-q [4(Y]] AEHS
Commonly an empirical estimate of the squared MMD taken as test statistic and thresholds estimated via permutations. The connection to our test is quite apparent, since both use the mean discrepancy as test statistic solely that the MMD optimizes the function over the RKHS unit ball, and the witness test tries to maximize the test power when learning the function set. The past years have shown that learning the kernel data-driven manner improves testing performance [Gretton et al., 2012b, Sutherland et al., 2017, Liu et al., 2020, Kübler et al., 2020, Liu et al., 2021].
held-out data
Schrab et al. [2021] test with finite collection of different kernels and reject if one of these MMD- based tests rejects. To ensure correct Type-I eитоr control, they need to aggregate the test and modify the test thresholds to account for the multiple testing (MMDAgg). This allows them to use the full dataset, without having to split train and test sets, but in turn this only enables using a countable candidate set. Other approaches rely on data splitting: Liu et al. [2020] proposed to learn deep kernel, by using an asymptotic test power criterion [Sutherland et al., 2017] and considering rather involved kernel function (MMD-D), see their Eq. (1). They concluded that this is often better than learning one-dimensional representation, like classifier does. In Appendix B. we show that our results in Section 3.1 similarly apply to learning kernels. Concretely, one can also use squared loss or cross-entrope loss when optimizing the kernel and the asymptotically optimal kernel is given as
1e (x, x²) = h" (x))
(10)
with h+ given in Eq. (7).
Kübler et al. [2022] questioned the insight of Liu et al. [2020] that one should learn kernel, by showing that learning simple witness function via test power criterion often suffices. They showed

---

how to use cross- validation and kernel Fisher Discriminant Analysis (kfda) to find powerful witness functions (kfda-witness), which serves as the blueprint for our more general approach. Chwialkowski et al. [2015] proposed another kernel-based fast two-sample test with smooth characteristic functions (SCF) and projected mean embeddings (ME), which was refined by Jitkrittum et al. [2016] who optimized this test statistic in the first stage. Kirchler et al. [2020] trained a deep multidimensional representation and used its mean distance as test statistic. Recently, Zhao et al. [2022] proposed a general framework that also includes MMD and Shekhar et al. [2022] proposed another variant of MMD two-sample tests.
Classifier two-sample tests (C2ST) also rely on a data splitinng approach and have extensively been studied in the literature [Friedman, 2003, Golland and Fischl, 2003, Lopez-Paz and Oquab, 2017, Kim et al., 2021, Cai et al., 2020, Hediger et al., 2022]. For simplicity, we focus on the balanced case. C2ST trains classifier with Sp, labelled with '1' and S labelled with "0' and then estimates its classification accuracy on Sp, SQ If the estimated accuracy is significantly above chance (that's what would be under the null hypothesis), the test rejects. Let f X - {0, 1] denote the binary classifier, then
write the accuracy as + E and estimate it
1 + f
1
1
f(x.))
1 The
Σ (1
f(wi))
1
1
1
The f(u))
2
2
The
The
2
The
The
$
2
Thus using the classification accuracy as test statistic is equivalent to using the mean discrepancy as test statistic with the binary classifier f as witness function in Eq. (2). However, using binary classifiers is quite limiting and results in quite high variance. Using continuous witness functions allows for higher power. An alternative to using the mean discrepancy is to rank the test data under the witness function and apply a Mann- Whitney test [Vayatis et al., 2009]. Some might also speak of "classifirr' test when referring to witness test, but using the term 'witness" emphasizes that it is continuous.
Lipton et al. [2018] proposed to use pretrained classifier to detect label shift. Rabanser et al. [2019] extended this to detect covariate shift. They investigate different ways of reducing the dimensionality and then applying different (classical) hypothesis test on them. While they also consider basic C2ST, their best performing method uses the softmax outputs of pretrained image classifier. They then run a univariate Kolmogorov-Smirnov test on each of the output 'probabilities' separately and correcting via Bonferroni correction. We refer to this as (univariate) BBSDs (black box shift detection soft). For more details on their other methods, we refer the reader to their work directly.
5 Experiments
To show the power of utilizing AutoML the same setup for all datasets consider. The data is split into two equally sized parts since this is the standard approach [Lopez-Paz and Oquab, 2017, Liu et al., 2020, Rabanser al., 2019]. We label data from P with '1', data from Q with 'O' and fit a least square regression with AutoGluon's TabularPredictte use the configuration presets='best limit. For more details, we refer to the AutoGluon documentation. We run all experiments with significance level a = 5%. Results of correct Type-I error control are provided in Appendix C. The sample size we report is always the size of the datasets before splitting, i.e, = m, since we only consider balanced problems.
[Erickson et al., 2020].² We and by default optimize with five-minute time
All experiments were done on servers having only CPUs and we spend around 100k CPU hours on doing all the experiments reported in the paper, which is mainly because we did various configurations and many repetitions for all the test cases we consider. Further details are given in Appendix C.
Blob& & Higgs. We first compare the performance on two low-dimensional datasets used to examine the power of two-sample tests. Following Liu et al. [2020) we consider the Blob dataset, which is a
LLpez-Paz Pazz and Oquab [2017] also observe that using binary classifier might be too restrictive (see their Remark 2). but they did not investigate this in detail.
We also run one experiment with Auto-Sklearn, see Table 5. A recent benchmark of AutoML frameworks be found in Gijsbers al. [2022].
7

---

kith -wttesss
AutoML
MMD-D
ME
SCF
1.0
1.0
0.8
0.8
Rejection Rare
0.6
0.6
0.2
0.0
0.0
200
300
400
500
600
700
800
900
2000
3000
4000
5000
6000
7000
8000 9000
10000
Samplesize
Semplesize
Figure 3: Experiments on low dimensional problems. The simple approach of learning one- dimensional witness function with AutoML or optimizing a witness via kfda and a grid search can outperform more involved approaches. Left: Blob, Right: Higgs.
Table Shift detection on MNIST and CIFARIO based on Rabanser et al. [2019].
(a) Test power across all simulated shifts MNIST and CI We propose the AutoML methods, and additionally run new baselines (MMDAgg. MMD-D).
(b) Test power depending the shift for the AutoML test on the features (raw) the AutoML test on the output pretrained features (pre).
Number samples from
Number of samples from
Stift
100
200
10,000
500
10,000
NoRed
0,03
0.15
0.26
0.36
0.41
0.47
0.54
0.72
0.20
0.27
0.33
0.40
0.43
0.50
0.63
0.80
PLA
0.15
0.41
0.54
0.63
pre
SRP
0.15
0.15
0.23
0.27
0.34
0.42
0.55
0.68
UAB
0.12
0.16
0:25
0.27
0.33
0.41
0.43
0.49
0.56
0.69
0.77
m_ga
1pte
0.00
0.03
0.17
0.00
0.00
0.13
0.07
0.83
0.13
x²
BBSDb
BBSDs
0.03
0.07
0.12
0.22
0.22
0.40
0.46
0.57
Lga
raw
pre
0.23
0.17
0.33
0.27
0.50
0.67
0.57
0.70
0.60
0.77
0.73
0.80
1.00
0.90
1.00
Bia
NoRed
Classit
0.114
0,01
0.03
0.15
0.22
0.11
0.28
0.21
0.32
0.28
0.44
0.42
0.55
0.51
0.67
s_img
FBW
0.15
0.20
0.27
0.30
0.30
0.50
0.35
0.57
0.40
0.67
0.50
0.83
0.53
12.8
0.83
1.00
Malie
UAE
PCA
0:15
0.20
0.27
0.18
0.40
0.33
0.43
0.38
0.45
0.40
0.53
0.46
0.55
0.61
kmg
FBW
pre
0.03
0.00
0.00
0.00
0.10
0.20
0.30
0.60
0.57
BBSD:
TAE
0.16
0.18
0.26
0.20
0.37
0.35
0.38
0.45
0.35
0.47
0.52
0.59
0.50
Limg
pre
0.20
0.07
0.27
0.37
0.40
0.50
0.47
0.70
0.83
AssoML (row)
0.17
0.24
0.37
0.46
0.50
0.62
0.67
0.87
0.70
0.90
(pre)
pre
0.60
0.83
0.87
AmnoMI (class)
0.19
0.19
0.38
0.46
0.52
0.61
0.67
0,87
AutoML (bia)
0.03
0.14
0.31
0.43
0.49
0.51
0.59
0.86
ko
FBW
pre
0.27
0.47
0.57
0.77
0.67
0.83
0.87
0.87
0.97
0.97
MMDAgg
0.20
0.28
0.40
0.43
0.46
0.32
0.58
0.79
MMDDD
0.22
0.19
0.25
0.36
0.40
0.48
0.56
0.65
m_img +ko
FRW
0.00
0.03
0.23
0.53
0.53
0.57
0.67
1.00
0.17
0.43
0.50
0.73
0.80
1.00
1.00
1.00
96
FaW
0.37
0.77
0.97
1.00
1.00
1.00
1.00
1.00
0.60
1.00
1.00
1.00
1.00
mixture of nine Gaussian modes, with different covariance structures between P and Q [Liu et al., 2020, Fig. 1]. We also consider the Higgs dataset [Baldi et al., 2014], which was introduced by Chwialkowski et al. [2015] as a two-sample problem. As baselines, we use MMD-D, ME, SCF, and kfda-witness as reported by Kübler et al. [2022]. We report the results in Fig. 3, where +1 standard eтror are shown as shaded regions. Since we estimated the performance over 500 runs, we obtain a smaller error than the other methods. We observe that both approaches based on the mean difference of a witness function (kfda-witness, AutoML) perform competitively. AutoML performs best on Blob, and kfda-witness is best Higgs.
Detecting distribution shift. Rabanser et al. [2019] introduced a large benchmark for the de- tection of distribution shifts. We repeat their experiments by considering the datasets MNIST [LeCun et al., 2010] and CIFARIO [Krizhevsky, 2009]. We consider sample sizes n, m E {10, 20, 50, 100, 200, 500, 1000, 10000). Each shift is applied on fraction 8 E {0.1, 0.5, 1.0]
8

---

of the second sample in different runs. We consider the following shifts: Adversarial (adv): Turn some images into adversarial examples via FGSM [Goodfellow et al., 2015]; Knock-out (ko): Re- move samples from class O: Gaussian noise (gn): Add gaussian noise to images with standard deviation a E (1, 10, 100) (denoted s_ s_gn, m m_gn, and l_gn); Image (img): Natural shifts to images through combinations of random rotations, (It, aeii--xi-ttranllation
well as zoom-in with different strength (denoted s s_img, m_ m_ing, and L_im);; Image + knock-out (m__mgg mm_img+ko): Fixed medium image shift and variable knock-out shift; Only-zero + image (oz+m_ _ing): Only images from class 0 in combination with a variable medium image shift. More details are given in [Rabanser et al., 2019]. In total, we run 33 different shift experiments MNIST and CIFARIO each and for each sample size. Every setting is repeated for 5 times.
The methods of Rabanser et al. [2019] perform a dimensionality reduction by using the whole training set (50.000 images for MNIST, 40.000 images for CIFARTO). The actual tests compare examples from the validation set (10.000 images) to examples from the shifted test set (10.000 images). They also consider a C2ST trained on the raw features, i.e. without seeing the whole training set.
We add four univariate AutoML witness tests: a) AutoML (raw) trains regression model on the raw data with MSE, which is our default, b) AutoML (pre) uses the same setting, but trains on the softmax output of a pretrained classifier for MNIST/CIFAR 10 respectively, which is the same representation as BBSDs used. <) AutoML (class) trains a classifier and uses its predicted probabilities of class '1' as witness function, d) AutoML (bin) uses the same as c) but only considers binary outputs.
As additional baselines we also, for the first time, the shift detection pipeline with MMD-D [Liu et al., 2020] and gMDAgg [Schrab et al., 2021], where we use the settings recommended in their paper. For MMD-D we use the exact architectures and hyperparameters that Liu et al. [2020] used for their MNIST and CIFAR-10 Tasks. For MMDAgg we report results with Laplacian kernels with bandwidth in Amed
c C {10, 11, .1, 19, 203.
Our findings are reported in Table 1. From Table la we see that AutoML (raw) achieves overall very competitive performance in detecting the shifts, especially for large sample sizes. Moreover, see that AutoML (raw) and AutoML (class) achieve comparable performance which confirms our findings of Remark 1. Thresholding the classification probabilities to binary outputs always harms the performance, see AutoMI. (class) vs. AutoMI. (bin). We can also compare AutoMI. (bin) with "classif", as reported by Rahanser et al. [2019]. While both use binary classifiers for the testing, "classif" used a fixed architecture across all shifts. This illustrates the power of using AutoM as find significantly better performance across all sample sizes. If instead of training on the raw features we start from the ten dimensional pretrained features, i.e. AutoML (pre), the performance is improved when the sample size is small. For large sample sizes, instead working with the raw features gives higher power. We also see that the AutoML test outperforms MMDAgg and MMD-D except for very small sample size.
In Table lb we report the test power for comparing AutoML (raw) with AutoML (pre) for the different shifts. Using the pretrained probabilities of the softmax output, it is extremely hard to detect Gaussian noise, while AutoML (raw) does a fairly good job here. This is consistent with the findings of Rabanser al. [2019, Table 1(b)]. Apparently, the output probabilities of the pretrained models quite invariant under small and medium noise on the inputs. For the other shifts, such as knock-outs, using the pretrained features improves performance, particularly at small sample sizes. The Code to reproduce our experiments is provided at htt.///oooninuueeeeeuuueuM----T-
Discussion
Bias-variance tradeoff. Our results on the distribution shift benchmark indicate bias- variance tradeoff when optimizing the witness in Stage-I. Learning the witness function over a ten dimensional pretrained representation gives good test power for some shifts even for small sample sizes, however, at the cost of being almost unable to detect other shifts, such as local Gaussian noise. Thus, learning on pretrained features introduces a strong bias, On the other hand, learning directly on the raw features introduces little bias, even more so since we used AutoGluon's 's TabularPredictor ctor, which is not specifically designed for images. This has the effect that on small sample sizes the test power is
'The authors of Schrab et al. [2021] proposed different parameter settings and ran the benchmark with those.
For their additional results please visit

---

reduced, but when large data is available, we observe good test power across almost all shifts. For practical applications this implies that using models with the right bias when learning hypothesis tests is just as important as in any other supervised learning setting.
Stand on the shoulders of giants. As we see from the Blob and Higgs experiments the conceptually simple witness two-sample test can outperform more sophisticated test statistics like the deep MMD. This is possible through both the use of cross-validation the distribution shift benchmark, we saw much better performance even when comparing binary classifier (AutoML (bin)) with classifier having prespecified architecture (classif). Furthermore, using an AutoML framework allows practitioners to stand on the shoulders of giants and removes the need for specialized expertise. Instead, they can directly control how much time and resources to spend on optimizing the witness, which can lead to improved significance and/or inference time.
( or a full AutoML. pipeline. In
Which test to use? Obviously, there is general answer to this question, and we not claiming that our AutoML two-sample test should always be used. In special settings, a simple parametric test would perform much better than our AutoML witness test (see Appendix C.3). Similarly, using MMD with a kernel can be the right choice in some settings. Nevertheless, a few points should be considered. For example, we demonstrate that test using binary outputs of classifier underperforms a test using the predicted probabilities of the same classifier. Therefore, we do recommend choosing the latter instead of the former. Furthermore, when using data splitting should ensure that in the first stage we are actually optimizing the test power or directly related proxy loss. To this end, it is important to use techniques that ensure good predictive performance and prevent overfitting. This brings us the last point: We should also consider the resources available, both computational and human, that are relevant when implementing the test. That is, testing framework should be easy to apply by large group of users and should be adaptable to the computational resources the user is willed to spend on the test. The AutoML witness test can tick off all boxes. It learns a continuous witness function to optimize test power, leverages well well-engineered toolboxes to maximize predictive performance, and requires little engineering expertise to apply and gives easy control over the computational resources used to learn the test, by setting a time limit and providing the available hardware.
Limitations and future work. While we recommend using a 50/50 split for learning and testing, this is generally not optimal. The splitting ratio balances the need to train a powerful witness function, while retaining enough data to obtain significant results in the testing phase. A potential extension could be to adaptively choose how much data is used for training [Lhéritier and Cazals, 2018], by estimating whether the expected improvement of the witness function outweighs the reduction of the test set. Alternatively, we could follow the idea of k-fold cross-validation. Setting 2, we could estimate two witness functions and estimate the witness value on the respective held-out sample. One could then effectively use the whole dataset compute the test statistic. However, this approach creates again a dependency and requires a new method to obtain reliable p-values.
7 Conclusion
We showed that optimizing a squared loss or cross-entropy loss leads to a witness function that maximizes test power, when using the mean discrepancy of the witness as a test statistic. This allows us harness the advances in Automated Machine Learning, where regression and classification are the standard tasks, for two-sample testing. Although less studied, the use of a well-engineered toolbox to maximize the predictive performance of the learned function is just as important for hypothesis testing as it is for supervised learning tasks. The result is testing pipeline that is theoretically justified, leads to competitive performance, and is simple to apply in various settings. Our work thus constitutes a step towards fully automated statistical analysis of complex data [Steinruecken et al., 2019].
Acknowledgments and Funding Disclosure. We thank Lisa Koch and Wittawat Jitkrittum for helpful discussions and Vincent Berenz for contributing to autotst. Furthermore, Antonin Schrab and Arthur Gretton for helping resolve the nemory-efficiene, issue of the MMDAgg test used in the original experiments and for providing complementary results to those in our Table 1. This work was supported by the German Federal Ministry of Education and Research (BMBF) through the Tübingen Al Center (FK7: 01IS18039B) and the Machine Learning Cluster of Excellence number 2064/1 Project 390727645.
10

---

References
Baldi, P. Sadowski, and D. Whiteson. Searching for exotic particles high-energy physics with deep learning. Nature communications, 5(1):1-9, 2014.
H. Cai, B. Goggin, and Q. Jiang. Two-sample test based on classification probability. Statistical Analysis and Data Mining: The ASA Data Science Journal, 13(1):5-13, 2020.
X. Cheng and A. Cloninger. arXiv:1909, 1298, 2019.
Classification logit two-sample testing by neural networks.
K. Chwialkowski, A. Ramdas, D. Sejdinovic, and A. Gretton. Fast two-sample testing with analytic representations of probability measures. In NeurIPS, 2015.
Dietterich. Ensemble methods machine learning. In International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classification, 2nd Edition. Wiley, 2001.
N. Erickson, J. Mueller, A. Shirkov, H. Zhang. P. Larroy, M. Li, and A. Smola. Autogluon-tabular: Robust and accurate automl for structured data. UrXiv:2003.06505 2020.
v. Erven, P. Grünwald, and De Rooij. Catching up faster by switching sooner: A predictive approach adaptive estimation with an application to the aic-bic dilemma. Journal of the Royal Statistical Society: Series (Statistical Methodology). 74(3):361-417) 2012.
M. Feurer, A. Klein, J. Eggensperger, Katharina Springenberg, M. Blum, and Hutter. Efficient and robust automated machine learning. In NeurlPS, 2015.
M. Feurer, K. Eggensperger, S. Falkner, M. Lindauer, and F. Hutter. Auto-sklearn 2.0: Hands-free automl via meta-learning. arXiv:2007.04074 2020.
H. Friedman. On multivariate goodness of fit and two sample testing. Stanford Linear Accelerator Center-PUB-10325,, 2003.
P. Gijsbers, M. L. Bueno, S. Coors, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren. Amib: an automl benchmark. arXiv:2207. /2560, 2022.
Golland and B. Fischl, Permutation tests for classification: Towards statistical significance in image- based studies. In C. Taylor and J. A. Noble, editors, Information Processing in Medical Imaging, pages 330-341, Berlin, Heidelberg, 2003. Springer Berlin Heidelberg. ISBN 978-3-540-45087-0
Goodfellow, Shlens, and Szegedy. Explaining and harnessing adversarial examples. In ICL.R, 2015.
A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with hilbert-schmidr norms. International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. Journal of Machine Learning Research, 13:723-773, 2012a.
A. Gretton, D. Sejdinovic, H. Strathmann, S. Balakrishnan, M. Pontil, K. Fukumizu, and B. K. Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. In NeurIPS, 2012b.
X. He, K. Zhao, and X. Chu. Automl: A survey of the state-of-the-art. Knowledge- Systems, 212:106622 2021.
Hediger, L. Michel, and Nåf. On the use of random forest for two-sample testing. Computational Statistics and Data Analysis, 770:107435. 2022.
F. Hutter, L. Kotthoff, and J. Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
W. Jitkrittum, 7. Szabó, K. Chwialkowski, and A. Gretton. Interpretable distribution features with maximum testing power. In NeurIPS, 2016.

---

I. Kim, A. Ramdas, A. Singh, and L. Wasserman. Classification accuracy as proxy for two-sample testing. The Annals of Statistics, 49(1):411 -434, 2021.
M. Kirchler, S. Khorasani, M. Kloft, and C. Lippert. Two-sample testing using deep learning. In AISTATS, 2020.
L. Koch, C. Schübach, Gretton, and P. Berens, Hidden in plain sight: Subgroup shifts escape ood detection. In Medical Imaging in Deep Learning, 2022.
A. Krizhevsky. Learning multiple layers of features from tiny images, 2009.
M. Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. Learning kernel tests without data splitting. In NewrIPS, 2020.
M. Kübler, W. Jitkrittum, B. Schölkopf, and K. Muandet. witness two-sample test. AISTATS, 2022.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.
http://aan.elcu.ccoo.. lecun.ccmm/xxd//miiss 2, 2010.
A. Lhéritier and F. Cazals. A sequential non- pa-paaametric multivariate two-sample test. IEEE Transac- tions on Information Theory, 64(5), 2018.
Z. Lipton, Y.-X. Wang, and A. Smola. Detecting and correcting for label shift with black box predictors. In JCML, 2018.
Liu, W. Xu, Lu, G. Zhang, A. Gretton, and D. Sutherland. Learning deep kernels for conpparmetrric two-sample tests. In ICML, 2020.
Liu, W. Xu, J. Lu, and D. J. Sutherland. Meta two-sample testing: Learning kernels for testing with limited data. In NeurlPS, 2021.
D. Lopez-Paz and M. Oquab. Revisiting classifier two-sample tests. In ICLR, 2017.
X. Mao, Q. Li, H. Xie, R. K. Lau, Z. Wang, and Smolley. On effectiveness least squares generative adversarial networks. IEEE Transactions on Pallern Analysis and Machine Intelligence, 41(12):2947-2960, dee 2019.
Mika. Kernel Fisher Discriminants. Doctoral thesis, Technische Universität Berlin, Berlin, 2003.
B. Phipson and G. K. Smyth. Permutation p-values should never be zero: calculating exact p-values when permutations are randomly drawn. Statistical applications genetics and molecular biology, 9(1), 2010.
S. Rabanser, S. Günnemann, and Z. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. In NewrIPS, 2019.
A. Schrab. I. Kim, M. Albert, B. Laurent, B. Guedj, and A. Gretton. Mmd aggregated two-sample test. arXiv:2 10.01.50033. 2021.
S. Shekhar, I. Kim, and A. Ramdas. A permutation-free kernel two-sample test. In NewrIPS, 2022.
C. Steinruecken, E. Smith, D. Janz, J. Lloyd, and Z. Ghahramani. The automatic statistician. In Automated Machine Learning. pages 161-173. Springer, Cham, 2019.
Student. The probable error of mean. Biometrika, pages 1-25, 1908.
D. J. Sutherland, H.-Y. Tung, H. Strathmann, S. De, A. Ramdas, A. Smola, and A. Gretton. Generative models and model criticism via optimized maximum mean discrepancy. In ICLR, 2017.
N. Vayatis, M. Depecker, and S. Clémençon. Auc optimization and the two-sample problem. In NeurlPS, 2009.
B. L. Welch. The generalization of 'student's' problem when several different population varlances are involved. Biometrika, 34(1-2):28-35, 1947.
Zhao, A. Sinha, Y. He, Perreault, J. Song, and S. Ermon. Comparing distributions by measuring differences that affect decision making. In ICLR, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Section and Section 3.2
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Partially in the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes]] Code to reproduce experiments is provided at github. htt.///nnueueeerttoot--SS-S- package autotst is provided at gjihub.com/jmkubbbe/atttooo
The
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes] In the tabular results, we omitted the error estimates for improved readability.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Section
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "l9YLjs8TDsg", "reviewer": "Reviewer_R3Sj", "summary": "This paper uses AutoML to optimize a classification function for a training set, and then the optimized function is used to perform a two-sample test in the test set. The proposed approach can be used in the MMD test and classifier two-sample test which needs a proper kernel/classifier. ", "questions": "I didn't see the comparison with model selection based on cross-validation. I think this is a very important baseline. Did I miss something here?  ", "limitations": "Please see \"Strengths And Weaknesses\".", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: finding a good classification function to reveal the discrepancy for the two-sample test is important, and this work is inspired to solve this problem. The proposed approach to using AutoML is straightforward yet relatively effective as shown in related datasets. \n\nWeakness:  a developer designs a two-sample test with the following guidance: make sure the Type I error is controlled and decrease the Type II error as much as possible. As Type II error in the finite-sample case is usually intractable, one usually resorts to proving that a two-sample test is consistent or a statistic is optimal asymptotically. Although the test with AutoML can be proved to be consistent, there is no theoretical evidence to show that the AutoML asymptotically captures an optimal classification function (described in eq (7) and this is the so-called Bayes Optimal Classifier.), which makes the biggest weaknesses of this work. But this is fixable by using the model switch proposed by [Lhéritier, 2018] and [Erven, 2012]. For example, Automl can be used in the early stage of training, and then switch to some point-wise consistent classifier (e.g. knn, please see sec. 25.6 in [Györfi , 2002]) in the late stage of training. This is based on the fact that a non-parametric regression asymptotically approaches true posterior probabilities.  \n\nReference:\nLhéritier, Alix, and Frédéric Cazals. \"A sequential non-parametric multivariate two-sample test.\" IEEE Transactions on Information Theory 64.5 (2018): 3361-3370.\nErven, Tim van, Peter Grünwald, and Steven De Rooij. \"Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC–BIC dilemma.\" Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74.3 (2012): 361-417.\nL. Györfi and A. Krzyzak. A distribution-free theory of nonparametric regression. Springer, 2002.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "akYAxBpM7F3", "reviewer": "Reviewer_k88M", "summary": "This work proposes a framework in order to perform a multivariate two-sample test using classification accuracy as a proxy to two sample problem. They prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML.", "questions": "- Due to the fact that the witness two-sample test is a newly proposed test, and we do not fully understand (currently) its limitations, I would recommend the authors to carefully mention also in this work their intuition about the cases (datasets) where such a test might be appropriate or not appropriate. \n- I kindly refer the authors to the work of  Clemencon et al AUC optimization and the two-sample problem (2009). They offer exactly the same framework and additionally, they take advantage of the strong theory between AUC and U-statistic of Mann Whitney hypothesis test. Therefore in a way, their work is very close to theirs. Moreover, linked to the above publication's theory, I kindly recommend the authors to see and discuss also the recent publication of Bargiotas et al 'Revealing posturographic profile of patients with Parkinsonian syndromes through a novel hypothesis testing framework based on machine learning (2021)' where many of the weaknesses of the initial framework (and the framework of others that are already mentioned (Liu 2020, Lopez-Paz 2017)) are addressed and tested in a \"real-life\" situation.\n  ", "limitations": "The authors mentioned clearly some limitations. However, it is not clear to me (although slightly mentioned) in which cases the auto-ML should be preferred or not. I believe that there are some more limitations/discussions that should be addressed (see weaknesses), especially concerning sample sizes, non-parametric cases, unbalanced datasets, type of datasets and ways of using such a test. I do not obviously ask for extended experimental results. However, It is useful to be discussed to some extent.  ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Well written article\n- Interesting theory that maximizes the statistic of the newly proposed witness two sample test. However, the reader should read also the authors work (Kubler et al 2022) in order to fully understand. \n- The authors try to secure the direct link between the optimized criterion of 1st stage and statistic of 2nd stage.\n\nWeaknesses\n-  The framework  (split dataset, optimize to train-set, apply to test-set) is not novel. \n-  P-value is given for test-set only. However, in many real settings (especially in areas such as Medicine, Sociology, Psychology etc), splitting the dataset ONCE and giving a p-value for the half of it is just 'non acceptable'. Except for the splitting problem, there is also a significant diminution of power. Therefore, I have serious doubts in terms of applicability in medium-sized datasets. It is probably the reason why the authors choose to test their framework to sample sizes >200. \n- Interpretability: Authors claim that interpretability is achieved but I did not see any experiment concerning this fact and how this works. (handling colinearities of predictors etc)\n- No experiments of Type II error control\n\nMinor Weaknesses\n- No experiments about no mean shift but different variances case.\n- authors claim AUTOML but users should worry about 1. splitting properly 2.weighting properly in case of unbalanced datasets. To be fair, both are mentioned clearly as limitations of their work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IaAymn1JTrq", "reviewer": "Reviewer_22yK", "summary": "This paper proposes to leverage existing AutoML frameworks to build two-sample tests, based on training a witness function on a train set and using it to compute the mean discrepancy on the test set.\nPrevious work has shown that minimizing the signal-to-noise ratio (SNR) is equivalent to maximizing the test power. However, the SNR is not commonly implemented in AutoML frameworks and is not amenable to mini-batching which rules out ML methods based on it. \nIn order to circumvent this, the authors prove that minimizing the squared loss implies minimizing the signal-to-noise ratio.\nSince the squared loss is commonly available in AutoML frameworks, this result allows to use them to build two-sample tests.\n", "questions": "With respect to the permutation method to compute p-values I suggest to cite and discuss the cases considered in:\n*Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical applications in genetics and molecular biology, 9(1).*\n\nLet B denote the number of permutations and N the number of permutation statistics as least as extreme as the statistic $\\tau$ obtained with the original labels.\n\nIn a nutshell, that work discusses the following:\n* using an unbiased estimator of the p-value $N/B$ can make the type I error rate above $\\alpha$, which can be especially dangerous in a multiple testing context\n* Instead of **estimating** the p-values, these can be computed under the permutation distribution:\n  * when permutations are sampled without replacement, $\\frac{N+1}{B+1}$ yields an exact p-value i.e. $P(p\\leq\\alpha)=  \\alpha$\n  * when permutations are sampled with replacement, $\\frac{N+1}{B+1}$ yields a valid p-value i.e. $P(p\\leq\\alpha)\\leq \\alpha$\n  * these formulas are valid for any $B$ (even small ones) but, for better power, a large $B$ is desired\n\n\nThe first work that considered to limit the computation to a subset of permutations and to give an exact formula for the case with replacement :\nDwass, M. (1957). Modified randomization tests for nonparametric hypotheses. The Annals of Mathematical Statistics, 181-187.\n\n\nOther comments:\n\nLine 93: converges to a constant as $n,m\\to\\infty$ ?\n\nLine 107: can you explicitly say why an estimate of the SNR based on a mini-batch would be biased?\n\nLine 121 and equation after line 122 : $L$ should be $L_{P,Q,c}$\n\nLine 124: For any function $h$ ?\n\nLine 125: Why is the loss strictly positive?\n\nLine 130: $\\bar{h}^*_P\\geq \\bar{h}^*_Q$:\n- deserves a justification\n- the notation $\\bar{h}$ is defined in the appendix but not in the main text\n\nLine 177: can you define and explain the square brackets operator? \n\nLine just after 469: I suggest to use $\\equiv$ for the shorthand definition \n\nLine 496: maximize -> maximizing", "limitations": "The authors have properly discussed some limitations and the potential negative societal impact of their work.\nI suggest to discuss consistency in relationship to properties of the regression methods considered in the AutoML frameworks.\n   ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the paper is clearly written and properly positioned with respect to prior work.\n- the proposed method is of interest to a wide audience of practitioners\n- the method is properly justified by the theoretical results\n\nWeaknesses\n- two-sample tests based on witness optimization were already proposed in the literature, so the originality is limited\n- the paper lacks a discussion on the basic property expected for a two-sample test, that is consistency (test power $\\to 1$ as the number of observations $\\to \\infty$). This should be related to the properties of the ML models considered by the AutoML frameworks.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3vFmgJ5IQpW", "reviewer": "Reviewer_6vMQ", "summary": "The paper proposed to use AutoML to perform two-sample tests to detect distribution shifts. The paper shows both theoretical proof and practical experiments on distribution shift detection tasks.", "questions": "- Same as in weakness, how other AutoML systems might impact the robustness of the two-sample tests in distribution shift detection?\n- Is it possible for the AutoML system to benefit from the proposed two-sample tests by detecting potential distribution shift between training and testing input spaces?", "limitations": "Yes. The authors addressed the limitations and usage considerations in part 6. Discussion. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The paper is well written with clear problem construction, thorough related work references, experiment settings and results discussion.\n- The idea of using AutoML as a tool to detect distribution shift of complex datasets is novel and interesting. \nWeakness:\n- Besides AutoGluon, there are also other AutoML systems such as AutoSklearn, H2O AutoML, etc., how does the choice of AutoML systems impact the distribution shift detection using the proposed two-sample tests setting?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept