POST SELECTION INFERENCE WITH INCOMPLETE MAXIMUM MEAN DISCREPANCY ESTIMATOR
Year: 2019

Abstract:
Measuring divergence between two distributions is essential in machine learning and statistics and has various applications including binary classification, change point detection, and two-sample test. Furthermore, in the era of big data, designing divergence measure that is interpretable and can handle high-dimensional and complex data becomes extremely important. In this paper, we propose a post selection inference (PSI) framework for divergence measure, which can select a set of statistically significant features that discriminate two distributions. Specifically, we employ an additive variant of maximum mean discrepancy (MMD) for features and introduce a general hypothesis test for PSI. A novel MMD estimator using the incomplete U-statistics, which has an asymptotically normal distribution (under mild assumptions) and gives high detection power in PSI, is also proposed and analyzed theoretically. Through synthetic and real-world feature selection experiments, we show that the proposed framework can successfully detect statistically significant features. Last, we propose a sample selection framework for analyzing different members in the Generative Adversarial Networks (GANs) family.

heading: 1 INTRODUCTION
text: Computing the divergence between two probability distributions is fundamental to machine learning and has many important applications such as binary classification (Friedman et al., 2001), change point detection (Yamada et al., 2013a; Liu et al., 2013), two-sample test (Gretton et al., 2012; Yamada et al., 2013b), and generative models such as generative adversarial networks (GANs) (Goodfellow et al., 2014; Li et al., 2015b; Nowozin et al., 2016), to name a few. Recently, interpreting the difference between distributions has become an important task in applied machine learning (Mueller & Jaakkola, 2015; Jitkrittum et al., 2016) since it can facilitate scientific discovery. For instance, in biomedical binary classification tasks, it is common to analyze which variables or features are different between two different distributions (classes).
The simplest approach to measure divergence between two probability densities would be parametric methods. For example, t-test can be used if the distributions to be compared are known and well-defined (Anderson, 2001). However, in many real-world problems, the property of distribution is not known a priori, and therefore model assumptions are likely to be violated. In contrast, non-parametric methods can be applied to any distributions without prior assumptions. The maximum mean discrepancy (MMD) (Gretton et al., 2012) is an example of non-parametric discrepancy measures and is defined as the difference between the mean embeddings of two distributions in a reproducing kernel Hilbert space (RKHS). For RKHSs associated with characteristic kernels (Sriperumbudur et al., 2011), the difference in mean embeddings defines a proper metric, and thus MMD can capture potentially non-linear difference between distributions (i.e., higher order moments).
∗Equal contribution
However, since MMD considers the entire d dimensional vector, it is hard to interpret how individual features contribute to the discrepancy.
To deal with the interpretability issue, divergence measure with feature selection has been actively studied (Yamada et al., 2013a; Mueller & Jaakkola, 2015; Jitkrittum et al., 2016). For instance, in Mueller & Jaakkola (2015), the Wasserstein divergence is employed as a divergence measure and `1 regularizer is used for feature selection. However, these approaches focus on detecting a set of features that discriminate two distributions. But for scientific discovery applications such as biomarker discovery, it might be preferable to test the significance of each selected feature (e.g., one biomarker). A naive approach would be to select features from one dataset and then test the selected features using the same data. However, in such case, selection bias is included and thus the false positive rate cannot be controlled. Therefore, it is crucial in hypothesis testing that the selection event should be taken into account to correct the bias. To the best of our knowledge, there is not yet an existing framework that tests the significance of selected features that distinguish between two different distributions on the same dataset.
In this paper, we propose mmdInf, a post selection inference (PSI) algorithm for distribution comparison, which finds a set of statistically significant features that discriminate between two distributions. mmdInf enjoys several compelling properties. First, it is a non-parametric method based on kernels, and thus it can detect features that distinguish various types of distributions. Second, the proposed framework is general and can be applied to not only feature selection but also other distribution comparison problems, such as dataset comparison. In mmdInf, we employ the recently developed PSI algorithm (Lee et al., 2016) and use MMD as a divergence measure. However, the standard empirical squared MMD estimator has a degenerated null distribution, which violates the requirement of a proper Normal distribution for the current PSI algorithm. To address this issue, we apply the block estimate (Zaremba et al., 2013) and the linear estimate (Gretton et al., 2012) of MMD. Furthermore, we propose a new empirical estimate of MMD based on the incomplete U-statistics (incomplete MMD estimator) (Blom, 1976; Janson, 1984; Lee, 1990) and show that it asymptotically follows the normal distribution, and has much greater detection power (compared to the existing estimators of MMD) in PSI. Finally, we propose a framework to analyze different members in the GAN (Goodfellow et al., 2014) family based on mmdInf. We elucidate the theoretical properties of the incomplete U-statistics estimate of MMD and show that mmdInf can successfully detect significant features through feature selection experiments.
The contributions of our paper are summarized as follows: (1) We propose a non-parametric PSI algorithm mmdInf for distribution comparison. (2) We propose the incomplete MMD estimator and investigate its theoretical properties (see Figure 1). (3) We propose a sample selection framework based on mmdInf that can be used for analyzing generative models.

heading: 2 RELATED WORK
text: Divergence measures in general can be divided into two categories: f -divergence (Ali & Silvey, 1966), such as the Kullback-Leibler divergence (Cover & Thomas, 2012) or the α-divergence (Rényi et al., 1961; Póczos & Schneider, 2011), and integral probability metric (Müller, 1997), including
the total variation distance (Shorack & Shorack, 2000) and the Wasserstein distance in its dual (Villani, 2008). In this work we consider the maximum mean discrepancy (MMD) (Gretton et al., 2012), which falls into the second category. Although these divergence measures can be used for testing the discrepancy between p(x) and q(x), it is hard to test the significance of one of k selected features from the entire d features, where the setup is useful for scientific discovery tasks such as biomarker discovery. Recently, a MMD-based change point detection algorithm, which can compute the p-value of the largest score, has been proposed (Li et al., 2015a). However, this method can only test the largest score (i.e., one-feature). Thus, it is not clear whether the approach can be extended to feature selection in general.
A novel testing framework for the post selection inference (PSI) has been recently proposed (Lee et al., 2016), in which statistical inference after feature selection with Lasso (Tibshirani, 1996) is investigated. This work shows that statistical inference conditioned on the selection event can be done for linear regression models with Gaussian noise if the selection event can be written as a set of linear constraints. However, the PSI algorithm needs to assume a Gaussian response, which is a relatively strong assumption, and consequently it cannot be directly applied for non-Gaussian output problems such as classification. To deal with this issue, a kernel based PSI algorithm (hsicInf) for independence test has been proposed (Yamada et al., 2018), in which the Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005) is employed to measure the independence between input and its output, and thus significance test can be performed on non-Gaussian data. In this paper, we propose an alternative kernel based inference algorithm for distribution comparison called mmdInf, which can be used for both feature selection and binary classification.
In Sections 3.4 and 5.3, we manifest how we analyze different members in the GANs (Goodfellow et al., 2014) family. Here, we discuss several evaluation metrics that have been proposed to compare generative models. For example, the inception scores (Salimans et al., 2016) and the mode scores (Che et al., 2016) measure the quality and diversity of the generated samples, but they were not able to detect overfitting and mode dropping / collapsing for generated samples. The Frechet inception distance (FID) (Heusel et al., 2017) defines a score using the first two moments of the real and generated distributions, whereas the classifier two-sample tests (Lopez-Paz & Oquab, 2016) considers the classification accuracy of a binary classifier as a statistic for two-sample test. Although the above metrics are reasonable in terms of discriminability, robustness, and efficiency, the distances between samples are required to be computed in a suitable feature space. We can also use the kernel density estimation (KDE); or more recently, Wu et al. (2016) proposed to apply the annealed importance sampling (AIS) to estimate the likelihood of the decoder-based generative models. Nevertheless, these approaches need the access to the generative model for computing the likelihood, which are less favorable comparing to the model agnostic approaches which rely only on a finite generated sample set. On the other hand, the maximum mean discrepancy (MMD) (Gretton et al., 2012) is preferred against its competitors (Sutherland et al., 2016; Huang et al., 2018). Therefore, we propose the mmdInf based GANs analysis framework.
3 PROPOSED METHOD (MMDINF)
In this section, we introduce a PSI algorithm with MMD (Gretton et al., 2012).

heading: 3.1 POST SELECTION INFERENCE (PSI)
text: Suppose we are given independent and identically distributed (i.i.d.) samples X = {xi}mi=1 ∈ Rd×m from a d-dimensional distribution p and i.i.d. samples Y = {yj}nj=1 ∈ Rd×n from another d-dimensional distribution q. Our goal is to find k < d features that differentiate betweenX and Y and test whether each selected feature is statistically significant.
Let S be a set of selected features, we consider the following hypothesis test:
• H0: ∑d s=1 ηsD̂(X (s),Y (s)) = 0 | S was selected,
• H1: ∑d s=1 ηsD̂(X (s),Y (s)) 6= 0 | S was selected,
where D̂(X(s),Y (s)) is the estimated discrepancy measure for the selected feature s and η = [η1, . . . , ηd]
> ∈ Rd is an arbitrary pre-defined parameter. To test the s-th feature, we can set η as a unit vector whose s-th position is 1 and zero otherwise.
We employ the post-selection inference (PSI) framework (Lee et al., 2016) to test the hypothesis. Thanks to the cumulative distribution of Theorem 1, we can compute the p-value of∑d s=1 ηsD̂(X (s),Y (s)) under feature selection.
Theorem 1 (Lee et al., 2016) Suppose that z ∼ N (µ,Σ), and the feature selection event can be expressed as Az ≤ b for some matrix A and vector b, then for any given feature represented by η ∈ Rn we have
F [V −(A,b),V +(A,b)]
η>µ,η>Ση (η>z) | Az ≤ b ∼ Unif(0, 1),
where F [a,b]µ,σ2(x) is the cumulative distribution function (CDF) of a truncated normal distribution
with mean µ and variance σ2 truncated at [a,b]. Given that α = A Ση η>Ση , the lower and upper truncation points are given by
V −(A, b) = max j:αj<0 bj − (Az)j αj + η>z, V +(A, b) = min j:αj>0 bj − (Az)j αj + η>z.
Feature Selection with Discrepancy Measure: Assume we have an estimate of a discrepancy measure for each feature: z = [D̂(X(1),Y (1)), . . . , D̂(X(d),Y (d))]> ∈ Rd ∼ N (µ,Σ), where D̂(·, ·) has large positive value when two distribution are different. We select the top-k features with largest discrepancy scores. We denote the index set of the selected k features by S, and that of the unselected k̄ = d− k features by S̄. This feature selection event can be characterized by
D̂(X(s),Y (s)) ≥ D̂(X(`),Y (`)), for all (s, `) ∈ S × S̄.
Note that we have in total k · k̄ constraints. The selection event can be rewritten as As,`z ≤ 0, for all (s, `) ∈ S × S̄, where As,` = [0 · · · 0 −1︸︷︷︸
s 0 · · · 0 1︸︷︷︸ ` 0 · · · 0] and A>s,` ∈ Rd is a row vector of A ∈ R(k·k̄)×d. Under such
construction,Az ≤ b can be satisfied by setting b = 0.

heading: 3.2 MAXIMUM MEAN DISCREPANCY (MMD)
text: We employ the maximum mean discrepancy (MMD) (Gretton et al., 2012) as divergence measure.
LetF be the unit ball in a reproducing kernel Hilbert space (RKHS) andK(x,x′) the corresponding positive definite kernel with Ex,x′∼p[ √ K(x,x)] < ∞,Ex,x′∼q[ √ K(x,x)] < ∞, the squared population MMD is defined as
MMD2[F , p, q] = Ex,x′ [K(x,x′)]− 2Ex,y[K(x,y)] + Ey,y′ [K(y,y′)],
where Ex,y denotes the expectation over independent random variables x with distribution p and y with distribution q. It has been shown that if the kernel function K(x,x′) is characteristic, then MMD[F ,X,Y ] = 0 if and only if p = q (Gretton et al., 2012). In the following, we introduce two existing MMD estimators and then propose the incomplete U-statistics MMD estimator.
(Complete) U-statistics estimator (Gretton et al., 2012): We use the Gaussian kernel: K(x,x′) = exp ( −‖x−x
′‖22 2σ2x
) , where σx > 0 is the Gaussian width. When m = n, the complete U-statistics of
MMD is defined as
MMD2u[F ,X,Y ] = 1 m(m− 1) ∑ i 6=j h(ui,uj),
where h(u,u′)=K(x,x′)+K(y,y′)−K(x,y′)−K(x′,y) is the U-statistics kernel for MMD and u = [x> y>]> ∈ R2d. However, since the complete U-statistics estimator of MMD is degenerated under p = q and does not follow normal distribution, this estimator cannot be used in PSI.
Algorithm 1 mmdInf (Feature Selection) H0 : ∑d s=1 ηsMMD 2 inc(X
(s),Y (s)) = 0 | S was selected Input: Data matricesX = [Xsel Xc] ∈ Rd×m and Y = [Ysel Yc] ∈ Rd×n. Params: η ∈ Rd and k. 1: Compute ẑ = [MMD2inc(X (1) sel ,Y (1) sel ), . . . ,MMD 2 inc(X (d) sel ,Y (d) sel )]
> ∈ Rd. 2: Select k features (i.e., k = |S|) and computeA using ẑ and Σ̂ usingXc and Yc. Set b = 0. 3: Compute p-value as p = 1− F [V −(A,b),V +(A,b)]
0,η>Σ̂η (η>ẑ).
Block estimator (Zaremba et al., 2013): The block estimate of MMD is given by
MMD2b [F ,X,Y ] = B1 n n/B1∑ i=1 MMD2u[F ,Xi,Yi],
where X = [X1, . . . ,Xm/B1 ], Y = [Y1, . . . ,Yn/B2 ], and Xi ∈ Rd×B1 and Yi ∈ Rd×B2 are i-th partitioned data. Here, we assume that the number of blocks m/B1 = n/B2 is an integer and B = B1 = B2. A commonly used heuristic for the block size is to set B = √ n. This estimator asymptotically follows the normal distribution whenB1 andB2 are finite andm and n go to infinity. The block estimator can be used for PSI, but the variance and normality depends on the partition of X and Y . Specifically, when the total number of samples is small, then a small block size would result in high variance, whereas larger block size tends to result in non-Gaussian response.
Incomplete U-statistics estimator: The described problems of the block-estimator motivated us to design a new MMD estimator that is normally distributed and has smaller variance. We therefore propose an MMD estimator based on the incomplete U-statistics (Blom, 1976; Janson, 1984; Lee, 1990). The incomplete U-statistics estimator of MMD is given by
MMD2inc[F ,X,Y ] = 1
` ∑ (i,j)∈D h(ui,uj),
where D is a subset of Sn = {(i, j)}i 6=j , and ` = |D|. D can be fixed design or random design. In particular, if we design D as D = {(1, 2), (3, 4), . . . , (n− 1, n)} then the incomplete U-statistic corresponds to the linear-time MMD estimator (Gretton et al., 2012).

heading: 3.3 COVARIANCE MATRIX ESTIMATION
text: For PSI, we need to estimate the covariance matrix Σ from data.
Block estimator To estimate Σ, we first compute Hb ∈ Rd×(n/B) whose elements are the meansubtracted MMDs. Then, we regard Hb as the data matrix and use a standard covariance estimator for estimating Σ̂b. Note that for small n, the number of blocks n/B tends to be smaller than the dimension d, and thus, the estimation accuracy of covariance matrix is low and affects the detection accuracy. To handle this issue, we employ the POET algorithm (Fan et al., 2013).
Incomplete U-statistics estimator To estimate Σ, we first compute Hinc ∈ Rd×` whose elements are the mean-subtracted U-stat kernels. Then, we regard Hinc as the data matrix and can use a standard covariance estimator to compute Σ̂inc.
The estimation performance of the covariance estimation for the block estimate heavily depends on the block size B. That is, if B is large, we need to estimate Σ̂b from a small number of samples. On the other hand, in the incomplete U-statistics estimator, we can set ` = rn n/B, where r > 0 is a constant (See Section 4). Thus, in practice, the estimated covariance matrix tends to be more accurate than the block estimator counterpart. Figure 6 in the supplementary material shows the MSEs between the true and the estimated covariance matrices of the incomplete estimate and the block estimate, respectively. As can be seen, the error of the incomplete estimate is smaller than that of the block estimate, which is an advantage of the incomplete MMD over the block MMD in PSI.

heading: 3.4 ADDITIONAL APPLICATIONS (GANS ANALYSIS)
text: The proposed PSI framework is general and can be used for not only feature selection but also sample selection. In generative modeling, the generated distribution should match the data distribution;
in other words, the discrepancy between the generated data and real data should be small. We can therefore apply the selective inference algorithm and use the significance value to evaluate the generation quality. In this paper we apply mmdInf to compare the performance of GANs. We first select the model whose generated samples has the smallest MMD score with the real data and then perform the hypothesis test.
Let x(s)i ∈ Rp be a feature vector generated by s-th GAN model with random seed i and yj ∈ Rp is a feature vector of an original image. Image features can be extracted by pre-trained Resnet (He et al., 2016) or auto-encoders. The hypothesis test can be written as
• H0: MMD2inc[F ,X(s),Y ]=0 | s generates samples closest to the real distribution, • H1: MMD2inc[F ,X(s),Y ] 6= 0| s generates samples closest to the real distribution.
Since we want to test the best generator that minimizes the discrepancy between generated and true samples (e.g., low MMD score), this sample selection event can be characterized by
MMD2inc[F ,X(s),Y ] ≤ MMD2inc[F ,X(`),Y ], (s, `) ∈ S × S̄.

heading: 4 THEORETICAL ANALYSIS OF INCOMPLETE MMD
text: We investigate the theoretical properties of the incomplete MMD estimator under the random design with replacement. For simplicity, we denote MMD2inc[F ,X,Y ] = MMD2inc, MMD2[F , p, q] = MMD2, MMD2inc[F ,X(s),Y (s)] = MMD 2,(s) inc , and MMD
2[F , p(s), q(s)] = MMD2,(s), respectively. We introduce the conditional expectations for the U-statistic kernel h(u,u′) as h1 = Eu′ [h(u,u′)] and h2 = h(u,u′). See the supplementary material for proof.
Theorem 2 Let c be the smallest integer such that hc 6= MMD2, and let n and ` tend to infinity such that γ = limn,`→∞ n−c`, 0 ≤ γ ≤ ∞. For sampling with replacement, we have ` 1 2 (MMD2inc −MMD2) D−→ N (0, σ2), if γ = 0. n c 2 (MMD2inc −MMD2)
D−→ V, if γ =∞. `
1 2 (MMD2inc−MMD2) D−→γ 12V + T, if 0 < γ <∞,
where V is the random variable of the limit distribution of n c 2 (MMD2u −MMD2), T is the random variable of N (0, σ2), σ2 = V ar(h(u,u′)), and T and V are independent.
Corollary 3 Assume limn,`→∞ n−2` = 0 and 0 < γ = limn,`→∞ n−1` < ∞. For sampling with replacement, incomplete U-statistics estimator of MMD is asymptotically normally distributed as{ `
1 2 MMD2inc D−→ N (0, σ2), if p = q. `
1 2 (MMD2inc −MMD2) D−→ N (0, σ2 + γσ2u), if p 6= q.
where σ2 = V ar(h(u,u′)) and σ2u = 4(Eu[(Eu′ [h(u,u′)]]− Eu,u′ [h(u,u′)])2).
Corollary 4 Assume limn,`→∞ n−1` = 0. For sampling with replacement, the incomplete Ustatistics estimator of MMD is asymptotically normally distributed as
` 1 2 (MMD2inc −MMD2) D−→ N (0, σ2).
Thus, in practice, by setting ` n2, the incomplete estimator is asymptotically normal and therefore can be applied in PSI. More specifically, we can set ` = rn n2, where r is a small constant. In practice, we found that r = 10 works well in general.
Figure 2 shows the empirical distribution under p = q and p 6= q for the complete estimator, the block estimator and the incomplete estimator. As can be observed, the empirical distribution of the incomplete estimator is normal for small sampling parameter r, and becomes similar to its complete counterpart if r is large; this is supported by Theorem 2 (γ =∞). Finally, the following theorem assure the joint normality of the z vector in Theorem 1.
Theorem 5 Suppose 0 ≤ γ <∞. Then, `1/2 {[
MMD2,(1)inc , . . . ,MMD 2,(d) inc
] − [ MMD2,(1), . . . ,MMD2,(d) ]} converges to multivariate normal distribution.

heading: 5 EXPERIMENTS
text: We compared mmdInf with a naive testing baseline (mmd), which first selects features using MMD and estimates corresponding p-values with the same data of feature selection without adjustment for the selection event. For mmdInf, we used the three MMD estimators: the linear-time MMD (Gretton et al., 2012), the block MMD (Zaremba et al., 2013), and the incomplete MMD. We used 1/2 of data to calculate the covariance matrix of MMD and the rest to perform feature selection and inference. We fixed the number of selected features (prior to PSI) k to 30. In PSI the significance of each of the 30 selected features (from ranking MMD) is computed and features with p-value lower than the significance level α = 0.05 are selected as statistically significant features.
For block MMD, in each experiment we set the candidate of block size as B = {10, 20, 50} . For incomplete MMD, in each experiment the ratio between number of pairs (i, j) sampled to compute incomplete MMD score and sample size is fixed at r = `n ∈ {0.5, 5, 10}. We reported the true positive rate (TPR) k ′
k∗ where k ′ is the number of true features selected by mmdInf or mmd and
k∗ is the total number of true features in synthetic data. We further computed the false positive rate (FPR) k ′′
k−k∗ where k ′′ is the number of non-true features reported as positives. We ran all
experiments over 200 times, and reported the average TPR and FPR.

heading: 5.1 SYNTHETIC EXPERIMENTS (PSI)
text: The number of features d is fixed to 50, and for each feature, data is randomly generated following a Gaussian distribution with set mean and variance. 10 out of the 50 features are set to be significantly different by shifting the distribution of one class away from the other (mean or variance). More specifically, we generate the synthetic data as
(a) Mean shift x ∼ N (050, I50), y ∼ N (µ, I), µ = [1>10 0>40]> ∈ R50, (b) Variance shift x ∼ N (050, I50), y ∼ N (0,Σ), Σ = diag([1.51>10 1>40]>), whereN (µ,Σ) is a multivariate normal distribution with mean µ ∈ Rd and covariance Σ ∈ Rd×d, 1p ∈ Rd is a vector whose elements are all one, 0d ∈ Rd is a vector whose elements are all zero, and diag(a) ∈ Rd×d is a diagonal matrix whose diagonal elements are a ∈ Rd.
Figure 3(a) and (b) show the FPRs of linear MMD, block MMD and incomplete MMD with or without PSI. As can be clearly seen, PSI successfully controls FPR with significance level α = 0.05 for all the three estimators, whereas the naive approach tends to have higher FPRs. Figures 3 shows the TPRs of the synthetic data. In both cases, the TPR of incomplete MMD converges to 1 significantly faster than the the other two empirical estimates.

heading: 5.2 REAL-WORLD DATA (BENCHMARK)
text: We compared the proposed algorithm by using real-world datasets. Since it is difficult to decide what is a ”true feature” in real-world data, we choose a few datasets for binary classification with small amount of features, and regard all the original features as true. We then concatenated random features to the true features (the total number of features d = 100). Table 1 shows TPRs and FPRs of mmdInf with different MMD estimators. It can be observed that the incomplete estimator significantly outperforms the other empirical estimates. Note that a higher TPR can be achieved with higher r, while the FPR is still controlled at 0.05 with r = 5. Additional comparison of different empirical estimates on real dataset (with varying sample size) can be found in Figures 8.

heading: 5.3 GANS ANALYSIS (SAMPLE SELECTION)
text: We also applied mmdInf for evaluating the generation quality of GANs. We trained BEGAN (Berthelot et al., 2017), DCGAN (Radford et al., 2015), STDGAN (Miyato et al., 2017), Cramer GAN (Bellemare et al., 2017), DFM (Warde-Farley & Bengio, 2016), DRAGAN (Kodali et al., 2017), Minibatch Discrimination GAN (Salimans et al., 2016), and WGAN-GP (Gulrajani et al., 2017), generated 5000 images (using Chainer GAN package 1 with CIFAR10 datasets), and extracted 512 dimensional features by pre-trained Resnet18 (He et al., 2016). For the true image sets, we subsampled 5000 images from CIFAR10 datasets and computed the 512 dimensional features using the same Resnet18. We then tested the difference between the generated images and the real images using mmdInf on extracted features (see Sec. 3.4).
We found that for all the members in the GAN family, the null hypothesis was rejected, i.e., the generated distribution and the real distribution are different. This result is consistent with the findings in (Sutherland et al., 2016), which demonstrate that optimized MMD has perfect discriminative power in GANs evaluation. As sanity check, we evaluated mmdInf by constructing an ”oracle” generative model that generates real images from CIFAR10. Next, we randomly selected 5000 images (a disjoint set from the oracle generative images) from CIFAR10 in each trial, and set the sampling ratio to r = 5. Figure 4(a) shows the distribution of p-values computed by our algorithm. We can see that the p-values are distributed uniformly in the tests for the ”oracle” generative model, which matches the theoretical result in Theorem 1. Thus the algorithm is able to detect the distribution difference and control the false positive rate. In other words, if the generated samples do not follow the original distribution, we can safely reject the null hypothesis with a given significance level α.
1https://github.com/pfnet-research/chainer-gan-lib
Figure 4(b) shows the estimated MMD scores of each model. Based on the results, we could tell that DFM was the best model and DCGAN was the second best model to approximate the true distribution. However, the difference between various members is not obvious. Developing a validation pipeline based on mmdInf for GANs analysis would be one interesting line of future work.

heading: 6 CONCLUSION
text: In this paper, we proposed a novel statistical testing framework mmdInf, which can find a set of statistically significant features that can discriminate two distributions. Through synthetic and realworld experiments, we demonstrated that mmdInf can successfully find important features and/or datasets. We also proposed a method for sample selection based on mmdInf and applied it in the evaluation of generative models.

heading: 7 ACKNOWLEDGEMENT
text: We would like to thank Dr. Kazuki Yoshizoe, Prof. Yuta Umezu , and Prof. Suriya Gunasekar for fruitful discussions and valuable suggestions. MY was supported by the JST PRESTO program JPMJPR165A and partly supported by MEXT KAKENHI 16K16114. IT was partially supported by MEXT KAKENHI (17H00758, 16H06538), JST CREST (JPMJCR1302, JPMJCR1502), RIKEN Center for Advanced Intelligence Project, and JST support program for starting up innovation-hub on materials research by information integration initiative. KF was supported partly by JSPS KAKENHI 18K19793. YHHT and RS were supported in part by the DARPA grants D17AP00001 and FA875018C0150.

Review summaries:
The authors focus on the selection problem of k statistically significant features discriminating 2 probability distributions accessible via samples. They propose a non-parametric approach under the PSI (post selection inference) umbrella using MMD (maximum mean discrepancy) as a discrepancy measure between probability distributions. The idea is to apply (asymptotically) normal MMD estimators, rephrase the top-k selection problem as a linear constraint, and reduce the problem to Lee et al., 2016. The efficiency of the approach is illustrated on toy examples and in GAN (generative adversarial network) context. The technique complements the PSI-based independence testing approach recently proposed by Yamada et al., 2018.