Task: Language Identification
Input (Spanish, English): ## Paper summary  The paper reconsiders the idea of using a binary classifier to do two-sample testing. The idea is to split the sample into two disjoint training and test sets, train a classifier on the training set, and use the accuracy on the test set as the test statistic. If the accuracy is above chance level, one concludes that the two samples are from different distributions i.e., reject H0.  A theoretical result on an asymptotic approximate test power is provided. One implication is that the test is consistent, assuming that the classifier is better than coin tossing. Experiments on toy problems, evaluation of GANs, and causal discovery verify the effectiveness of the test. In addition, when the classifier is a neural net, examining the first linear filter layer allows one to see features which are most activated. The result is an interpretable visual indicator of how the two samples differ.  ## Review summary   The paper is well written and easy to follow. The idea of using a binary classifier for a two-sample testing is not new, as made clear in the paper. The main contributions are the analysis of the asymptotic test power, the use of modern deep nets as the classifier in this context, and the empirical studies on various tasks. The empirical results are satisfactorily convincing.  Although not much discussion is made on why the method works well in practice, overall contributions have a potential to start a new direction of research on model criticisms of generative models, as well as visualization of where a model fails. I vote for an acceptance.  ## Major comments / questions   My main concern is on Theorem 1 (asymptotic test power) and its assumptions.  But, I understand that these can be fixed as discussed below.  * Under H0, the distribution of the test statistic (i.e., sum of 0-1 classification results) follows Binomial(nte, 1/2) as stated.  However, under H1, terms in the sum are independent but *not* identical Bernoulli random variable. This is because each term depends on a data point z_i, which can be from either P or Q. So, in the paragraph in Sec3.1: "... the random variable n_te \hat{t} follows a Binomial(nte, p)..." is not correct. Essentially p depends on z_i. It should follow a Poisson binomial distribution.  * In the same paragraph, for the same reason, the alternative distribution of Binomial(nte, p=p_{risk}) is probably not correct. I guess you mention it to use Moivre-Laplace to get the asymptotic normality.   Anyway, I see no reason why you would need this statement as the Binomial is not required in the proof, but only its asymptotic normality. A variant of the central limit theorem (instead of the Moivre-Laplace theorem) for independent, non-identical variables would still allow you to conclude the asymptotic normality of the Poisson binomial (with some conditions). See for example
Output (English): en