Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed to use AutoML to perform two-sample tests to detect distribution shifts. The paper shows both theoretical proof and practical experiments on distribution shift detection tasks.
strengths_and_weaknesses: Strength:
- The paper is well written with clear problem construction, thorough related work references, experiment settings and results discussion.
- The idea of using AutoML as a tool to detect distribution shift of complex datasets is novel and interesting. 
Weakness:
- Besides AutoGluon, there are also other AutoML systems such as AutoSklearn, H2O AutoML, etc., how does the choice of AutoML systems impact the distribution shift detection using the proposed two-sample tests setting?

questions: - Same as in weakness, how other AutoML systems might impact the robustness of the two-sample tests in distribution shift detection?
- Is it possible for the AutoML system to benefit from the proposed two-sample tests by detecting potential distribution shift between training and testing input spaces?
limitations: Yes. The authors addressed the limitations and usage considerations in part 6. Discussion.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a framework in order to perform a multivariate two-sample test using classification accuracy as a proxy to two sample problem. They prove that minimizing a squared loss leads to a witness with optimal testing power. This allows us to leverage recent advancements in AutoML.
strengths_and_weaknesses: Strengths
- Well written article
- Interesting theory that maximizes the statistic of the newly proposed witness two sample test. However, the reader should read also the authors work (Kubler et al 2022) in order to fully understand. 
- The authors try to secure the direct link between the optimized criterion of 1st stage and statistic of 2nd stage.

Weaknesses
-  The framework  (split dataset, optimize to train-set, apply to test-set) is not novel. 
-  P-value is given for test-set only. However, in many real settings (especially in areas such as Medicine, Sociology, Psychology etc), splitting the dataset ONCE and giving a p-value for the half of it is just 'non acceptable'. Except for the splitting problem, there is also a significant diminution of power. Therefore, I have serious doubts in terms of applicability in medium-sized datasets. It is probably the reason why the authors choose to test their framework to sample sizes >200. 
- Interpretability: Authors claim that interpretability is achieved but I did not see any experiment concerning this fact and how this works. (handling colinearities of predictors etc)
- No experiments of Type II error control

Minor Weaknesses
- No experiments about no mean shift but different variances case.
- authors claim AUTOML but users should worry about 1. splitting properly 2.weighting properly in case of unbalanced datasets. To be fair, both are mentioned clearly as limitations of their work.
questions: - Due to the fact that the witness two-sample test is a newly proposed test, and we do not fully understand (currently) its limitations, I would recommend the authors to carefully mention also in this work their intuition about the cases (datasets) where such a test might be appropriate or not appropriate. 
- I kindly refer the authors to the work of  Clemencon et al AUC optimization and the two-sample problem (2009). They offer exactly the same framework and additionally, they take advantage of the strong theory between AUC and U-statistic of Mann Whitney hypothesis test. Therefore in a way, their work is very close to theirs. Moreover, linked to the above publication's theory, I kindly recommend the authors to see and discuss also the recent publication of Bargiotas et al 'Revealing posturographic profile of patients with Parkinsonian syndromes through a novel hypothesis testing framework based on machine learning (2021)' where many of the weaknesses of the initial framework (and the framework of others that are already mentioned (Liu 2020, Lopez-Paz 2017)) are addressed and tested in a "real-life" situation.
  
limitations: The authors mentioned clearly some limitations. However, it is not clear to me (although slightly mentioned) in which cases the auto-ML should be preferred or not. I believe that there are some more limitations/discussions that should be addressed (see weaknesses), especially concerning sample sizes, non-parametric cases, unbalanced datasets, type of datasets and ways of using such a test. I do not obviously ask for extended experimental results. However, It is useful to be discussed to some extent.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to leverage existing AutoML frameworks to build two-sample tests, based on training a witness function on a train set and using it to compute the mean discrepancy on the test set.
Previous work has shown that minimizing the signal-to-noise ratio (SNR) is equivalent to maximizing the test power. However, the SNR is not commonly implemented in AutoML frameworks and is not amenable to mini-batching which rules out ML methods based on it. 
In order to circumvent this, the authors prove that minimizing the squared loss implies minimizing the signal-to-noise ratio.
Since the squared loss is commonly available in AutoML frameworks, this result allows to use them to build two-sample tests.

strengths_and_weaknesses: Strengths:
- the paper is clearly written and properly positioned with respect to prior work.
- the proposed method is of interest to a wide audience of practitioners
- the method is properly justified by the theoretical results

Weaknesses
- two-sample tests based on witness optimization were already proposed in the literature, so the originality is limited
- the paper lacks a discussion on the basic property expected for a two-sample test, that is consistency (test power $\to 1$ as the number of observations $\to \infty$). This should be related to the properties of the ML models considered by the AutoML frameworks.



questions: With respect to the permutation method to compute p-values I suggest to cite and discuss the cases considered in:
*Phipson, B., & Smyth, G. K. (2010). Permutation P-values should never be zero: calculating exact P-values when permutations are randomly drawn. Statistical applications in genetics and molecular biology, 9(1).*

Let B denote the number of permutations and N the number of permutation statistics as least as extreme as the statistic $\tau$ obtained with the original labels.

In a nutshell, that work discusses the following:
* using an unbiased estimator of the p-value $N/B$ can make the type I error rate above $\alpha$, which can be especially dangerous in a multiple testing context
* Instead of **estimating** the p-values, these can be computed under the permutation distribution:
  * when permutations are sampled without replacement, $\frac{N+1}{B+1}$ yields an exact p-value i.e. $P(p\leq\alpha)=  \alpha$
  * when permutations are sampled with replacement, $\frac{N+1}{B+1}$ yields a valid p-value i.e. $P(p\leq\alpha)\leq \alpha$
  * these formulas are valid for any $B$ (even small ones) but, for better power, a large $B$ is desired


The first work that considered to limit the computation to a subset of permutations and to give an exact formula for the case with replacement :
Dwass, M. (1957). Modified randomization tests for nonparametric hypotheses. The Annals of Mathematical Statistics, 181-187.


Other comments:

Line 93: converges to a constant as $n,m\to\infty$ ?

Line 107: can you explicitly say why an estimate of the SNR based on a mini-batch would be biased?

Line 121 and equation after line 122 : $L$ should be $L_{P,Q,c}$

Line 124: For any function $h$ ?

Line 125: Why is the loss strictly positive?

Line 130: $\bar{h}^*_P\geq \bar{h}^*_Q$:
- deserves a justification
- the notation $\bar{h}$ is defined in the appendix but not in the main text

Line 177: can you define and explain the square brackets operator? 

Line just after 469: I suggest to use $\equiv$ for the shorthand definition 

Line 496: maximize -> maximizing
limitations: The authors have properly discussed some limitations and the potential negative societal impact of their work.
I suggest to discuss consistency in relationship to properties of the regression methods considered in the AutoML frameworks.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper uses AutoML to optimize a classification function for a training set, and then the optimized function is used to perform a two-sample test in the test set. The proposed approach can be used in the MMD test and classifier two-sample test which needs a proper kernel/classifier. 
strengths_and_weaknesses: Strengths: finding a good classification function to reveal the discrepancy for the two-sample test is important, and this work is inspired to solve this problem. The proposed approach to using AutoML is straightforward yet relatively effective as shown in related datasets. 

Weakness:  a developer designs a two-sample test with the following guidance: make sure the Type I error is controlled and decrease the Type II error as much as possible. As Type II error in the finite-sample case is usually intractable, one usually resorts to proving that a two-sample test is consistent or a statistic is optimal asymptotically. Although the test with AutoML can be proved to be consistent, there is no theoretical evidence to show that the AutoML asymptotically captures an optimal classification function (described in eq (7) and this is the so-called Bayes Optimal Classifier.), which makes the biggest weaknesses of this work. But this is fixable by using the model switch proposed by [Lhéritier, 2018] and [Erven, 2012]. For example, Automl can be used in the early stage of training, and then switch to some point-wise consistent classifier (e.g. knn, please see sec. 25.6 in [Györfi , 2002]) in the late stage of training. This is based on the fact that a non-parametric regression asymptotically approaches true posterior probabilities.  

Reference:
Lhéritier, Alix, and Frédéric Cazals. "A sequential non-parametric multivariate two-sample test." IEEE Transactions on Information Theory 64.5 (2018): 3361-3370.
Erven, Tim van, Peter Grünwald, and Steven De Rooij. "Catching up faster by switching sooner: A predictive approach to adaptive estimation with an application to the AIC–BIC dilemma." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 74.3 (2012): 361-417.
L. Györfi and A. Krzyzak. A distribution-free theory of nonparametric regression. Springer, 2002.

questions: I didn't see the comparison with model selection based on cross-validation. I think this is a very important baseline. Did I miss something here?  
limitations: Please see "Strengths And Weaknesses".
review_title: null

Meta-review: metareview: This paper proposes a two-sample test based on AutoML. The paper is interesting and it proposes what promises to be a practical method. However, the reviewers have noted several discussions that need to be made explicit in the paper. I will recommend this for acceptance with a strong encouragement for the authors to incorporate the reviewer comments into crafting the final manuscript. 
confidence: Less certain
Decision: Accept