Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: It is stated that a comparison is performed "against the linear-time estimate of the Maximum Mean Discrepancy (MMD)"
- what kernel and perameters were used for this?
- why the linear MMD?  That is a higher variance, lower-power estimate than the quadratic MMD at a fixed sample size.

The MMD is itself a kind of discriminant function, which in the 2 sample literature has recently been assumed to be constrained to be in a unit ball of a RKHS.  Classifier 2-sample tests also require the assumption of a specific function space (effect of regularization also a bit complex?), so are there still advantages if the function space is a RKHS, too?  That should be provable, possibly leading to an improved kernel 2-sample test.

Could you optimize the MMD definition with your proposed classifier function spaces (k-nn & neural network)?  Would that have provably worse properties than the proposed test?

Missing relevant citations using MMD for model selection are:
- A Test of Relative Similarity For Model Selection in Generative Models
- Generative Models and Model Criticism via Optimized Maximum Mean Discrepancy
review_title: comparison to MMD

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The submission considers the setting of 2-sample testing from the perspective of evaluating a classifier.  For a classifier between two samples from the same distribution, the distribution of the classification accuracy follows a simple form under the null hypothesis.  As such, a straightforward threshold can be derived for any classifier.  Finding a more powerful test then amounts to training a better classifier.  One may then focus efforts, e.g. on deep neural networks, for which statistics such as the MMD may be very difficult to characterize.

+ The approach is sound and very general
+ The paper is timely in that deep learning has had huge impacts in classification and other prediction settings, but has not had as big an impact on statistical hypothesis testing as kernel methods have

- The discussion of the relationship to kernel-MMD has not always been as realistic as it could have been.  For example, the kernel-MMD can also be seen as a classifier based approach, so a more fair discussion could be provided.  Also, the form of kernel-MMD used in the comparisons is a bit contradictory to the discussion as well
 * The linear kernel-MMD is used which is less powerful than the quadradic kernel-MMD (the authors have justified this from the perspective of computation time)
 * The kernel-MMD is argued against due to its unwieldy distribution under the null, but the linear time kernel-MMD (see also Zaremba et al., NIPS 2013) has a Gaussian distribution under the null.

Arthur Gretton's comment from Dec 14 during the discussion period was very insightful and helpful.  If these insights and additional experiments comparing the kernel-MMD to the classifier threshold on the blobs dataset could be included, that would be very helpful for understanding the paper.  The open review format gives an excellent opportunity to assign proper credit for these experiments and insights by citing the comment.
review_title: A generally useful and interesting approach to 2-sample tests

final_score: rating: null
review_content: This is still a pre-review phase. At this point, I only skimmed through the paper. 

1. You mentioned that ".. two-sample tests can measure statistical dependence..." on page 2. This is intuitively easy to understand since a two-sample test relies on a distance between distributions P and Q. One can set P to be P_xy and Q to be P_x*P_y to turn a distance into a dependence measure. What I do not understand is that in practice we do not have sample from P_x*P_y. As mentioned in the paper that a permutation to break the xy pairs is needed to simulate a sample from P_x*P_y. But what can we say theoretically about this approach? Say, assume that the two-sample test is consistent, is the independence test done in this way consistent as well?

2. On page 4, "... maximizing the size of the test ...", what is "the size"? Type-I error?

3. The chosen toy examples chosen do not seem to illustrate well the advantages of the proposed method (Fig. 1). Why are these examples considered? Have you considered other toy problems?




review_title: Theoretical questions on classifier two-sample test

final_score: rating: 7: Good paper, accept
review_content: 

## Paper summary

The paper reconsiders the idea of using a binary classifier to do two-sample testing. The idea is to split the sample into two disjoint training and test sets, train a classifier on the training set, and use the accuracy on the test set as the test statistic. If the accuracy is above chance level, one concludes that the two samples are from different distributions i.e., reject H0.

A theoretical result on an asymptotic approximate test power is provided. One implication is that the test is consistent, assuming that the classifier is better than coin tossing. Experiments on toy problems, evaluation of GANs, and causal discovery verify the effectiveness of the test. In addition, when the classifier is a neural net, examining the first linear filter layer allows one to see features which are most activated. The result is an interpretable visual indicator of how the two samples differ.

## Review summary 

The paper is well written and easy to follow. The idea of using a binary classifier for a two-sample testing is not new, as made clear in the paper. The main contributions are the analysis of the asymptotic test power, the use of modern deep nets as the classifier in this context, and the empirical studies on various tasks. The empirical results are satisfactorily convincing.  Although not much discussion is made on why the method works well in practice, overall contributions have a potential to start a new direction of research on model criticisms of generative models, as well as visualization of where a model fails. I vote for an acceptance.

## Major comments / questions 

My main concern is on Theorem 1 (asymptotic test power) and its assumptions.  But, I understand that these can be fixed as discussed below.

* Under H0, the distribution of the test statistic (i.e., sum of 0-1 classification results) follows Binomial(nte, 1/2) as stated.  However, under H1, terms in the sum are independent but *not* identical Bernoulli random variable. This is because each term depends on a data point z_i, which can be from either P or Q. So, in the paragraph in Sec3.1: "... the random variable n_te \hat{t} follows a Binomial(nte, p)..." is not correct. Essentially p depends on z_i. It should follow a Poisson binomial distribution.

* In the same paragraph, for the same reason, the alternative distribution of Binomial(nte, p=p_{risk}) is probably not correct. I guess you mention it to use Moivre-Laplace to get the asymptotic normality. 

Anyway, I see no reason why you would need this statement as the Binomial is not required in the proof, but only its asymptotic normality. A variant of the central limit theorem (instead of the Moivre-Laplace theorem) for independent, non-identical variables would still allow you to conclude the asymptotic normality of the Poisson binomial (with some conditions). See for example http://stats.stackexchange.com/questions/5347/how-can-i-efficiently-model-the-sum-of-bernoulli-random-variables

* The statement in Theorem 1 should be more precise. For instance, "Assume the assumptions on the classifier given in the previous paragraph. Then, for large n, the power of C2ST is approximate \Phi(..)." 

The current version is "The power of C2ST is \Phi(..)."


* The proof of Theorem 1 should be more precise regarding which quantities are exact, which are approximate. Both the null and alternative normal distributions are approximate for finite nte, for instance.

* It is unclear to me why the paper includes independence tests in the experiments. It does imply that the proposed test can be used to do an independence test. But, isn't this also true for other two-sample tests where x,y can be stacked together? This seems ad-hoc and raises further questions regarding the consistency, what type of dependency can be detected, etc. These points are not discussed.

* Comment: By using classification accuracy as a proxy, one should expect the test power, for a given sample size n, to be lower than a direct statistic like MMD. A vague analogy would be the t-test (based on the values of the data) vs. the sign test (based on only whether x_i < y_j, not the actual values). The classifier test is in some sense reminiscent of the sign test i.e., passing data points through a classifier and binarizing the output. For sufficiently large n, the test can still correctly detect the difference as shown empirically.

## Minor comments / questions

* Section 2, in the paragraph on the four steps of testing, the random variable for the statistic T is undefined.

* In the last paragraph of Sec.2: "kernel two-sample tests require the prescription of a manually-engineered representation of the data under study, and return values in units that are difficult to interpret." This is too vague. Manually-engineered representation? Wouldn't a neural net require even more of the explicitness of the representation?

* In sec.3.2, it is better to also state the assumptions on the classifier in a slightly less technical way. Specifically, under H0, you assume that the classifier is not biased, and under H1 you assume that it can learn well (better than coin tossing).






review_title: a review

final_score: rating: 7: Good paper, accept
review_content: I would like first to apologize for the delay.

Summary: A framework for two-samples statistical test using binary
classification is proposed. It allows multi-dimensional sample testing and
an interpretability that other tests lack. A theoritical analysis is
provided and various empirical tests reported.

A very interesting approach. I have however two main concerns.

The clarity of the presentation is obscured by too much content. It would
be more interesting if the presentation could be somewhat
self-contained. You could consider making 2 papers out of this paper.

Seriously, you cram a lot of experiments in this paper. But the setting
of the experiments is not really explained. We are supposed to have read
Jitkrittum et al., 2016, Radford et al., 2016, Yu et al., 2015, etc. All 
this is okay but reduces your public to a very few.

For example, if I am not mistaken, you never explained what SCF is, despite
the fact that its performances are reported. 

As a second point, given also that the number of submissions to this conference are exploding,
I would like to challenge you with the following question:

Why is this work significant to the representation learning community?

review_title: a very interesting framework; could improve clarity; significance to ICLR?

final_score: rating: null
review_content: - How do you choose the value of the hyper-parameters ? The ones related to the optimal classifier (K for KNN, the size of the hidden layer)? The ones related to the test (n_te)? Maybe you explained in somewhere but I missed the explanation.

- In the appendix B, *n* refers to something different than the size of learning examples in one class.

- In section 3.1, I think there is an *I* missing in the expression in the 2nd line. 

review_title: Hyper-parameters and more

Meta-review: The reviewers agree that the paper is a valuable contribution to the literature.
Decision: Accept (Poster)