AutoML Two-Sample Test

Abstract: Two-sample tests are important in statistics and machine learning, both as tools for scientific discovery as well as to detect distribution shifts.
This led to the development of many sophisticated test procedures going beyond the standard supervised learning frameworks, whose usage can require specialized knowledge about two-sample testing. We use a simple test that takes the mean discrepancy of a witness function as the test statistic and prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Without any user input about the problems at hand, and using the same method for all our experiments, our AutoML two-sample test achieves competitive performance on a diverse distribution shift benchmark as well as on challenging two-sample testing problems. 

Reviews and author responses:

 We thank the reviewers for their time, their participation in the discussion, and their helpful suggestions. We will use the additional page for the camera-ready version to discuss the topics raised in the reviews as we outlined in our rebuttal.

The main additions will be:

- Consistency of our method, relating it to nonparametric methods for theoretical guarantees and discussing the idea of switching models.

- Approaches to overcome the simple data splitting we use (sequential approach or combining p-values over multiple train/test splits).

- Clarification of the permutation approach we use.

- Presentation of the Python package.

- Additional experimental results as suggested by the reviewers and reported.


We cannot fit this into the existing 9 pages and hence are not able to provide an updated version now (which is still constraint to 9 pages).
Hence, if some reviewers agree that we can address their concerns in the camera-ready version, may we ask them to kindly adjust their score accordingly, please?

Thank you very much!

 Generally, I don’t quite buy that because this is a “AutoML” issue, and therefore the proposed two-sample test would just neglect this issue. For the most of cases, a classification problem such as image classification only needs to do prediction instead of inference, and this lack of consistency issue in automl for the classification problem might be fine. However, a two-sample test essentially is doing inference for a hypothesis, and a reliable model is expected to use. A good practice of two-sample test design should starts from controlling Type I error (which you already have) and then proving the optimality of this test asymptotically, and lastly empirically validates the proposed test in a finite-sample case. Solely showing “good empirical results” does not guarantee  any reliability  of a two-sample test. 

 Thanks for clarifying your implementation on permutations. I think it is important to inform the users about this and stress that this approach  provides type I error control (i.e. valid p-values) for any sample size.   

 Thanks for addressing my concern on consistency. I think the paper would benefit from this discussion, in order to let the users know what to expect in terms of consistency. Moreover, I think it is worth including the suggestion of reviewer R3Sj of combining AutoML with nonparametric pointwise consistent regressors via switching.  This would allow to combine the power of AutoML with the consistency guarantees of nonparametric methods.

 Type-I error : I apologize for my "TYPE error" error. Indeed I meant Type-I error control here as you correctly understood. 

Generally: Thank you for your answer. I am waiting to read the updated version, which will discuss all the above limitations properly. I insist that these comments, especially regarding the splitting issue, the lack of power, possible solutions etc, should be presented in the main manuscript and not in the appendix, since such important limitations consist essential elements of the authors' current work.


 Dear reviewer, 
I accidentally included this part of the answer to your review in the answer to another reviewer. Please take this into account and excuse the inconvenience.

> 'I didn't see the comparison with model selection based on cross-validation. I think it is a very important baseline. Did I miss something here?'

In our experiments on Blobs and Higgs we compare against kfda-witness of Kübler et al (2022). We missed mentioning this explicitly, but their results are obtained using cross validation (for kernel choice and regularization), which seems to be a reason it performs so well. We will add this and generally emphasize that using cross-validation instead of a full AutoML framework can be a good alternative. 

 Dear Reviewers,

We would like to touch base with you to see whether you had a chance to look at our rebuttal. We do hope that it has resolved all the concerns you raised in your review. However, if there remain concerns or if you have more questions, we will be more than happy to provide additional clarification.

Thank you so much for your time.

Best wishes, Authors

 Dear Reviewer, you asked for a comparison with other AutoML frameworks. We obtained the following results for the distribution shift benchmark running AutoSklearn with a 10 minute time limit.

| sample size | Test Power |
| --- | --- | 
| 10  | 0.10 |
| 20 | 0.18 |
| 50 | 0.28 |
| 100 | 0.38 |
| 200 | 0.43 |
| 500 | 0.38|
| 1,000 | 0.43 |
| 10,000 | 0.49 |

We observe that the results obtained via AutoGluon outperform the new experiments with AutoSklearn. The main reason might be, that AutoSklearn takes longer to train and hence the ten-minute time limit is too short. We will try to run the whole benchmark experiment also with a longer limit; however, this will take some more time. 
We will add the results to the final version of the paper.



 We thank you for your review and hope that we can mitigate your criticism regarding the consistency and usage of AutoML with our general author response (above). Much of your criticism is about the use of AutoML generally and not specific to our paper. According to our empirical results, AutoML can produce witness functions that are powerful enough to be used for two-sample test in practice. 


> Work of Lheritier et al (2018)


We thank you for pointing out the work of Lheritier et al, which we will discuss as a potential extension.  Indeed, adding adaptive data splitting strategies to our test could be interesting for future research, in particular since we (like many other works) somewhat arbitrarily fix a 50/50 splitting ratio. Nevertheless, there are also good reasons that many popular tests rely on data splitting [Lopez-Paz & Oquab (2017), Liu et al (2020), ...]. For example, it simplifies the control of type-I errors and allows us to use any engineering tricks to maximize predictive performance in the first stage. As you say, our approach is 'straightforward yet relatively effective'; this is precisely what we want to focus on with our work.

We would be thankful to incorporate any other comments from you that help us to improve the manuscript.

 Thank you for your review and for emphasizing the importance of a discussion of consistency, which we will add (see our general response above). We hope that the practical relevance of our paper together with the open-source package convinces you that our paper is indeed an important contribution.
Thank you also for studying our paper so attentively and providing a list of minor comments, which we will thankfully incorporate in our paper.

> Regarding your comment on *permutations and p-value estimation*:

Thank you for pointing out the work of Phipson & Smyth (2010) which is an important reference to include here. After submission, we also realized that using the biased estimates of the p-value is the safer way to go. This is in fact how we implemented it in our Python package `XXX`, where we use the approach with replacement. We generally use quite many permutations to have the best performance and reduce the amount of additional randomness. We will add a few lines to the manuscript and refer to the paper to make sure users are aware of this topic.

----
Update from August 08: Please ignore below, which belongs to the answer to R3Sj and was accidentally copied here.

> 'I didn't see the comparison with model selection based on cross-validation. I think it is a very important baseline. Did I miss something here?'

In our experiments on Blobs and Higgs we compare against kfda-witness of Kübler et al (2022). We missed mentioning this explicitly, but their results are obtained using cross validation (for kernel choice and regularization), which seems to be a reason it performs so well. We will add this and generally emphasize that using cross-validation instead of a full AutoML framework can be a good alternative. 


 We thank you for your valuable review and your comments regarding data splitting and testing only on held-out data. Below, we address your suggestions and questions.
> P-value is given for the test set only & related work of Bargiotas et al (2021) and Clemencon et al. 

Thank you for pointing out the paper of Bargiotas et al. We will discuss this and the ts-AUC approach of Clemencon et al in the updated version. Interestingly, Bargiotas et al conclude that future work 'should investigate the statistical metrics that would be theoretically suitable to be used as optimization criteria'. This is precisely what we do, albeit for a witness test, which in turns is related to the optimized t-test mentioned in Remark 1 of Clemencon et al.

Furthermore, an approach similar to the one used by Bargiotas et al that allows to use all the data in the testing phase should in principle be feasible with the witness test. However, it is presently not clear to us whether any additional correction needs to be done to guarantee Type-I error control. We contacted the authors of Bargiotas et al and hope to be able to clarify this. If you are aware of any reference that explicitly proofs correct Type-I error control of this approach, kindly let us know.

> Interpretability

We discuss the interpretability of our results in Section 3.3 and Figure 2. The latter corresponds to an actual experiment. After performing the test, the witness is evaluated and the inputs with maximal/minimal value are inspected. For image data, this easily allows analyzing the differences. This is  similar to the interpretability given by Lopez-Paz & Oquab (2017). Another approach is to interpret the results via feature importance (again similar to Bargiotas et al). Here, AutoML is useful because it directly provides methods to estimate feature importance.

> No experiments on Type II error control.

In all our experiments in the main paper, we report rejection rates, i.e., the test power, which is exactly *(1 - Type-II error rate)*. 
Did you perhaps mean Type-I error control here? In fact, we include Type-I error experiments in the Appendix B.1 for Higgs and Blobs and show that our test correctly controls Type-I error at finite data size. Please let us know if we misunderstand your question, so we can clarify this.

> 'users should worry about 1. splitting properly 2. weighting properly in case of unbalanced datasets.'

Our provided package `XXX` automatically splits the data and also does the weighting automatically, so users do not need to worry about this. By default, we split the data 50/50. We hope that, similarly to Bargiotas et al, we can add a discussion on how to overcome the fact that only 50% of the data are used for testing.

> When should we use AutoML or not? - Experiments with difference in variance.

As we mention in our discussion, if one can make strong assumptions about the data, using a classic test can be beneficial. Following your suggestion, we ran some additional toy experiments, where we test for equality in variance for two Gaussians with equal mean (and variance 1.0 and 1.5 respectively). We run our AutoML test and compare the performance to an F-test of equal variance (which uses the assumption of normality!). 
For sample sizes 50, 100, 500 we obtain test power of AutoML as 0.15, 0.43, 0.97 and for the F-test as 0.88, 0.97, 1.0 (estimated over 100 trials, at significance level 5%). Obviously both tests are consistent but the F-test has a clear advantage because it is specific to the problem. Conversely, the AutoML test is particularly suited for situations where we have complex datasets and do not know how the distributions might differ (if they do) and as you correctly point out have rather large datasets. We will extend the discussion in the paper and include the experiment as an explicit example.



We hope that our reply clarified some aspects, and are confident that we can improve the manuscript with your suggestions.

 Thank you for your review and positive assessment, we address your two open questions below. 

> How does the choice of AutoML system impact the [...] proposed tests?

Please see our comment to all reviewers for a description of the Python package we released. This allows to easily incorporate other AutoML frameworks. We will also run the distribution shift benchmark with AutoSklearn and report the results in the camera-ready version.

> Is it possible that AutoML benefits from the proposed two-sample test.

In principle, it could be a good idea that an AutoML framework automatically tests for distribution shift between the training and test data. Our method would be an obvious candidate to do so. However, this is not the focus of our paper.


 We thank all the reviewers for their time and their constructive comments. 
We want to emphasize that the main goal of our paper is to provide an ML based two-sample test that is **easy to apply in practice**. Reviewers 6vMQ, k88M, and 22yK deem the paper above-the-acceptance-threshold saying it is 'novel', 'well written', 'of interest to a wide audience of practitioners', and 'justified by theoretical results'. There remained some reservations pertaining to the following topics: consistency of the procedure, applicability with other (Auto)ML frameworks, and novelty. We address these points here in a general response to all reviewers, and address other questions and report additional experiments in individual comments. Should further questions come up, we will be happy to address them during the discussion period. 

### Novelty and applicability to other (Auto)ML frameworks

Reviewers k88M and 22yK correctly note that the used general framework (data splitting, learning a witness function, and testing on held-out data) is in itself not novel. The novel theoretical contribution lies in the proof that optimizing a squared loss is equivalent to maximizing the test power, enabling us to leverage recent advances in AutoML. This novelty is acknowledged explicitly by Reviewers k88M and 22kY. Furthermore, we put our effort into ensuring not only that the proposed method is theoretically sound, but also that it is easy-to-use for practitioners. To this end, we released an open-source package (after submission), called `XXX` here to preserve anonymity. 

Our provided package is simply installable via `pip install XXX`. With the default setting, one can compute  p-values as follows, where `sample_P` and `sample_Q` are the (possibly unbalanced) samples:
```
import XXX
tst = XXX.AutoTST(sample_P, sample_Q, model=AutoGluonTabularPredictor)
p_value = tst.p_value()
```
Our package is modularized such that it is easy for practitioners to use other AutoML frameworks (Rev 6vMQ) when learning the witness functions or use methods that enjoy stronger theoretical guarantees (Rev R3Sj). All they have to do is to call AutoTST with `model=CustomModel` where `CustomModel` is a wrapper around their custom ML framework. To emphasize this flexibility, we will also run the distribution shift benchmark with other AutoML frameworks, as suggested by 6vMQ, and will include the results in the camera-ready version. This open-source package is an important addition to our paper, strengthening its contribution, practical relevance, and impact. We kindly ask the reviewers to take this into account.


### Consistency

We thank reviewer 22yK and R3Sj for requesting an explicit discussion of consistency. Although this property essentially follows from prior work [Lopez-Paz & Oquab (2017), Kübler et al. (2022)], we will add a discussion to make the paper more self-contained. The consistency follows from Equation (4): The test is consistent if in Stage I  we find a witness function with SNR > ϵ >0 with probability going to 1 as the training data set size grows. Whenever SNR > ϵ, Eq.(4) implies that the probability of rejection will go to 1, as the test set size grows to infinity. Thus, overall  to ensure consistency of the test, we need to ensure that the ML framework consistently learns a function that discriminates $P$ and $Q$. 

As reviewer R3Sj correctly points out, AutoML does not provide formal guarantees on finding the optimal witness function. Generally, we want to stress that this discussion is not specific to our work, but rather of general nature. For a theoretical analysis, relying on a non-parametric method seems indeed more adequate and standard results from regression and classification can be re-used.



Nevertheless, *in practice* what matters is to learn a model with the best predictive performance in the first stage. Our evaluation on a large distribution shift benchmark in Table 1 demonstrates the effectiveness of the proposed method and that AutoML can produce powerful witness functions. If some users, on the other hand, want to work with methods that have stronger theoretical guarantee, they can simply wrap their preferred method and still use our package `XXX` (see above). 


 The paper proposed to use AutoML to perform two-sample tests to detect distribution shifts. The paper shows both theoretical proof and practical experiments on distribution shift detection tasks. Strength:
- The paper is well written with clear problem construction, thorough related work references, experiment settings and results discussion.
- The idea of using AutoML as a tool to detect distribution shift of complex datasets is novel and interesting. 
Weakness:
- Besides AutoGluon, there are also other AutoML systems such as AutoSklearn, H2O AutoML, etc., how does the choice of AutoML systems impact the distribution shift detection using the proposed two-sample tests setting?
 - Same as in weakness, how other AutoML systems might impact the robustness of the two-sample tests in distribution shift detection?
- Is it possible for the AutoML system to benefit from the proposed two-sample tests by detecting potential distribution shift between training and testing input spaces? Yes. The authors addressed the limitations and usage considerations in part 6. Discussion. 

 This work proposes a framework in order to perform a multivariate two-sample test using classification accuracy as a proxy to two sample problem. They prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML. Strengths
- Well written article
- Interesting theory that maximizes the statistic of the newly proposed witness two sample test. However, the reader should read also the authors work (Kubler et al 2022) in order to fully understand. 
- The authors try to secure the direct link between the optimized criterion of 1st stage and statistic of 2nd stage.

Weaknesses
-  The framework  (split dataset, optimize to train-set, apply to test-set) is not novel. 
-  P-value is given for test-set only. However, in many real settings (especially in areas such as Medicine, Sociology, Psychology etc), splitting the dataset ONCE and giving a p-value for the half of it is just 'non acceptable'. Except for the splitting problem, there is also a significant diminution of power. Therefore, I have serious doubts in terms of applicability in medium-sized datasets. It is probably the reason why the authors choose to test their framework to sample sizes >200. 
- Interpretability: Authors claim that interpretability is achieved but I did not see any experiment concerning this fact and how this works. (handling colinearities of predictors etc)
- No experiments of Type II error control

Minor Weaknesses
- No experiments about no mean shift but different variances case.
- authors claim AUTOML but users should worry about 1. splitting properly 2.weighting properly in case of unbalanced datasets. To be fair, both are mentioned clearly as limitations of their work. - Due to the fact that the witness two-sample test is a newly proposed test, and we do not fully understand (currently) its limitations, I would recommend the authors to carefully mention also in this work their intuition about the cases (datasets) where such a test might be appropriate or not appropriate. 
- I kindly refer the authors to the work of  Clemencon et al AUC optimization and the two-sample problem (2009). They offer exactly the same framework and additionally, they take advantage of the strong theory between AUC and U-statistic of Mann Whitney hypothesis test. Therefore in a way, their work is very close to theirs. Moreover, linked to the above publication's theory, I kindly recommend the authors to see and discuss also the recent publication of Bargiotas et al 'Revealing posturographic profile of patients with Parkinsonian syndromes through a novel hypothesis testing framework based on machine learning (2021)' where many of the weaknesses of the initial framework (and the framework of others that are already mentioned (Liu 2020, Lopez-Paz 2017)) are addressed and tested in a "real-life" situation.
   The authors mentioned clearly some limitations. However, it is not clear to me (although slightly mentioned) in which cases the auto-ML should be preferred or not. I believe that there are some more limitations/discussions that should be addressed (see weaknesses), especially concerning sample sizes, non-parametric cases, unbalanced datasets, type of datasets and ways of using such a test. I do not obviously ask for extended experimental results. However, It is useful to be discussed to some extent.  

 This paper proposes to leverage existing AutoML frameworks to build two-sample tests, based on training a witness function on a train set and using it to compute the mean discrepancy on the test set.
Previous work has shown that minimizing the signal-to-noise ratio (SNR) is equivalent to maximizing the test power. However, the SNR is not commonly implemented in AutoML frameworks and is not amenable to mini-batching which rules out ML methods based on it. 
In order to circumvent this, the authors prove that minimizing the squared loss implies minimizing the signal-to-noise ratio.
Since the squared loss is commonly available in AutoML frameworks, this result allows to use them to build two-sample tests.
 Strengths:
- the paper is clearly written and properly positioned with respect to prior work.
- the proposed method is of interest to a wide audience of practitioners
- the method is properly justified by the theoretical results

Weaknesses
- two-sample tests based on witness optimization were already proposed in the literature, so the originality is limited
- the paper lacks a discussion on the basic property expected for a two-sample test, that is consistency (test power $\to 1$ as the number of observations $\to \infty$). This should be related to the properties of the ML models considered by the AutoML frameworks.


 With respect to the permutation method to compute p-values I suggest to cite and discuss the cases considered in:
*Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical applications in genetics and molecular biology, 9(1).*

Let B denote the number of permutations and N the number of permutation statistics as least as extreme as the statistic $\tau$ obtained with the original labels.

In a nutshell, that work discusses the following:
* using an unbiased estimator of the p-value $N/B$ can make the type I error rate above $\alpha$, which can be especially dangerous in a multiple testing context
* Instead of **estimating** the p-values, these can be computed under the permutation distribution:
  * when permutations are sampled without replacement, $\frac{N+1}{B+1}$ yields an exact p-value i.e. $P(p\leq\alpha)=  \alpha$
  * when permutations are sampled with replacement, $\frac{N+1}{B+1}$ yields a valid p-value i.e. $P(p\leq\alpha)\leq \alpha$
  * these formulas are valid for any $B$ (even small ones) but, for better power, a large $B$ is desired


The first work that considered to limit the computation to a subset of permutations and to give an exact formula for the case with replacement :
Dwass, M. (1957). Modified randomization tests for nonparametric hypotheses. The Annals of Mathematical Statistics, 181-187.


Other comments:

Line 93: converges to a constant as $n,m\to\infty$ ?

Line 107: can you explicitly say why an estimate of the SNR based on a mini-batch would be biased?

Line 121 and equation after line 122 : $L$ should be $L_{P,Q,c}$

Line 124: For any function $h$ ?

Line 125: Why is the loss strictly positive?

Line 130: $\bar{h}^*_P\geq \bar{h}^*_Q$:
- deserves a justification
- the notation $\bar{h}$ is defined in the appendix but not in the main text

Line 177: can you define and explain the square brackets operator? 

Line just after 469: I suggest to use $\equiv$ for the shorthand definition 

Line 496: maximize -> maximizing The authors have properly discussed some limitations and the potential negative societal impact of their work.
I suggest to discuss consistency in relationship to properties of the regression methods considered in the AutoML frameworks.
   

 This paper uses AutoML to optimize a classification function for a training set, and then the optimized function is used to perform a two-sample test in the test set. The proposed approach can be used in the MMD test and classifier two-sample test which needs a proper kernel/classifier.  Strengths: finding a good classification function to reveal the discrepancy for the two-sample test is important, and this work is inspired to solve this problem. The proposed approach to using AutoML is straightforward yet relatively effective as shown in related datasets. 

Weakness:  a developer designs a two-sample test with the following guidance: make sure the Type I error is controlled and decrease the Type II error as much as possible. As Type II error in the finite-sample case is usually intractable, one usually resorts to proving that a two-sample test is consistent or a statistic is optimal asymptotically. Although the test with AutoML can be proved to be consistent, there is no theoretical evidence to show that the AutoML asymptotically captures an optimal classification function (described in eq (7) and this is the so-called Bayes Optimal Classifier.), which makes the biggest weaknesses of this work. But this is fixable by using the model switch proposed by [Lhéritier, 2018] and [Erven, 2012]. For example, Automl can be used in the early stage of training, and then switch to some point-wise consistent classifier (e.g. knn, please see sec. 25.6 in [Györfi , 2002]) in the late stage of training. This is based on the fact that a non-parametric regression asymptotically approaches true posterior probabilities.  

Reference:
Lhéritier, Alix, and Frédéric Cazals. "A sequential non-parametric multivariate two-sample test." IEEE Transactions on Information Theory 64.5 (2018): 3361-3370.
Erven, Tim van, Peter Grünwald, and Steven De Rooij. "Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC–BIC dilemma." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74.3 (2012): 361-417.
L. Györfi and A. Krzyzak. A distribution-free theory of nonparametric regression. Springer, 2002.
 I didn't see the comparison with model selection based on cross-validation. I think this is a very important baseline. Did I miss something here?   Please see "Strengths And Weaknesses".

Meta-review: This paper proposes a two-sample test based on AutoML. The paper is interesting and it proposes what promises to be a practical method. However, the reviewers have noted several discussions that need to be made explicit in the paper. I will recommend this for acceptance with a strong encouragement for the authors to incorporate the reviewer comments into crafting the final manuscript. 

Decision: Accept