Post Selection Inference with Incomplete Maximum Mean Discrepancy Estimator

1 INTRODUCTION . Computing the divergence between two probability distributions is fundamental to machine learning and has many important applications such as binary classification ( Friedman et al. , 2001 ) , change point detection ( Yamada et al. , 2013a ; Liu et al. , 2013 ) , two-sample test ( Gretton et al. , 2012 ; Yamada et al. , 2013b ) , and generative models such as generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ; Li et al. , 2015b ; Nowozin et al. , 2016 ) , to name a few . Recently , interpreting the difference between distributions has become an important task in applied machine learning ( Mueller & Jaakkola , 2015 ; Jitkrittum et al. , 2016 ) since it can facilitate scientific discovery . For instance , in biomedical binary classification tasks , it is common to analyze which variables or features are different between two different distributions ( classes ) . The simplest approach to measure divergence between two probability densities would be parametric methods . For example , t-test can be used if the distributions to be compared are known and well-defined ( Anderson , 2001 ) . However , in many real-world problems , the property of distribution is not known a priori , and therefore model assumptions are likely to be violated . In contrast , non-parametric methods can be applied to any distributions without prior assumptions . The maximum mean discrepancy ( MMD ) ( Gretton et al. , 2012 ) is an example of non-parametric discrepancy measures and is defined as the difference between the mean embeddings of two distributions in a reproducing kernel Hilbert space ( RKHS ) . For RKHSs associated with characteristic kernels ( Sriperumbudur et al. , 2011 ) , the difference in mean embeddings defines a proper metric , and thus MMD can capture potentially non-linear difference between distributions ( i.e. , higher order moments ) . ∗Equal contribution However , since MMD considers the entire d dimensional vector , it is hard to interpret how individual features contribute to the discrepancy . To deal with the interpretability issue , divergence measure with feature selection has been actively studied ( Yamada et al. , 2013a ; Mueller & Jaakkola , 2015 ; Jitkrittum et al. , 2016 ) . For instance , in Mueller & Jaakkola ( 2015 ) , the Wasserstein divergence is employed as a divergence measure and ` 1 regularizer is used for feature selection . However , these approaches focus on detecting a set of features that discriminate two distributions . But for scientific discovery applications such as biomarker discovery , it might be preferable to test the significance of each selected feature ( e.g. , one biomarker ) . A naive approach would be to select features from one dataset and then test the selected features using the same data . However , in such case , selection bias is included and thus the false positive rate can not be controlled . Therefore , it is crucial in hypothesis testing that the selection event should be taken into account to correct the bias . To the best of our knowledge , there is not yet an existing framework that tests the significance of selected features that distinguish between two different distributions on the same dataset . In this paper , we propose mmdInf , a post selection inference ( PSI ) algorithm for distribution comparison , which finds a set of statistically significant features that discriminate between two distributions . mmdInf enjoys several compelling properties . First , it is a non-parametric method based on kernels , and thus it can detect features that distinguish various types of distributions . Second , the proposed framework is general and can be applied to not only feature selection but also other distribution comparison problems , such as dataset comparison . In mmdInf , we employ the recently developed PSI algorithm ( Lee et al. , 2016 ) and use MMD as a divergence measure . However , the standard empirical squared MMD estimator has a degenerated null distribution , which violates the requirement of a proper Normal distribution for the current PSI algorithm . To address this issue , we apply the block estimate ( Zaremba et al. , 2013 ) and the linear estimate ( Gretton et al. , 2012 ) of MMD . Furthermore , we propose a new empirical estimate of MMD based on the incomplete U-statistics ( incomplete MMD estimator ) ( Blom , 1976 ; Janson , 1984 ; Lee , 1990 ) and show that it asymptotically follows the normal distribution , and has much greater detection power ( compared to the existing estimators of MMD ) in PSI . Finally , we propose a framework to analyze different members in the GAN ( Goodfellow et al. , 2014 ) family based on mmdInf . We elucidate the theoretical properties of the incomplete U-statistics estimate of MMD and show that mmdInf can successfully detect significant features through feature selection experiments . The contributions of our paper are summarized as follows : ( 1 ) We propose a non-parametric PSI algorithm mmdInf for distribution comparison . ( 2 ) We propose the incomplete MMD estimator and investigate its theoretical properties ( see Figure 1 ) . ( 3 ) We propose a sample selection framework based on mmdInf that can be used for analyzing generative models . 2 RELATED WORK . Divergence measures in general can be divided into two categories : f -divergence ( Ali & Silvey , 1966 ) , such as the Kullback-Leibler divergence ( Cover & Thomas , 2012 ) or the α-divergence ( Rényi et al. , 1961 ; Póczos & Schneider , 2011 ) , and integral probability metric ( Müller , 1997 ) , including the total variation distance ( Shorack & Shorack , 2000 ) and the Wasserstein distance in its dual ( Villani , 2008 ) . In this work we consider the maximum mean discrepancy ( MMD ) ( Gretton et al. , 2012 ) , which falls into the second category . Although these divergence measures can be used for testing the discrepancy between p ( x ) and q ( x ) , it is hard to test the significance of one of k selected features from the entire d features , where the setup is useful for scientific discovery tasks such as biomarker discovery . Recently , a MMD-based change point detection algorithm , which can compute the p-value of the largest score , has been proposed ( Li et al. , 2015a ) . However , this method can only test the largest score ( i.e. , one-feature ) . Thus , it is not clear whether the approach can be extended to feature selection in general . A novel testing framework for the post selection inference ( PSI ) has been recently proposed ( Lee et al. , 2016 ) , in which statistical inference after feature selection with Lasso ( Tibshirani , 1996 ) is investigated . This work shows that statistical inference conditioned on the selection event can be done for linear regression models with Gaussian noise if the selection event can be written as a set of linear constraints . However , the PSI algorithm needs to assume a Gaussian response , which is a relatively strong assumption , and consequently it can not be directly applied for non-Gaussian output problems such as classification . To deal with this issue , a kernel based PSI algorithm ( hsicInf ) for independence test has been proposed ( Yamada et al. , 2018 ) , in which the Hilbert-Schmidt Independence Criterion ( HSIC ) ( Gretton et al. , 2005 ) is employed to measure the independence between input and its output , and thus significance test can be performed on non-Gaussian data . In this paper , we propose an alternative kernel based inference algorithm for distribution comparison called mmdInf , which can be used for both feature selection and binary classification . In Sections 3.4 and 5.3 , we manifest how we analyze different members in the GANs ( Goodfellow et al. , 2014 ) family . Here , we discuss several evaluation metrics that have been proposed to compare generative models . For example , the inception scores ( Salimans et al. , 2016 ) and the mode scores ( Che et al. , 2016 ) measure the quality and diversity of the generated samples , but they were not able to detect overfitting and mode dropping / collapsing for generated samples . The Frechet inception distance ( FID ) ( Heusel et al. , 2017 ) defines a score using the first two moments of the real and generated distributions , whereas the classifier two-sample tests ( Lopez-Paz & Oquab , 2016 ) considers the classification accuracy of a binary classifier as a statistic for two-sample test . Although the above metrics are reasonable in terms of discriminability , robustness , and efficiency , the distances between samples are required to be computed in a suitable feature space . We can also use the kernel density estimation ( KDE ) ; or more recently , Wu et al . ( 2016 ) proposed to apply the annealed importance sampling ( AIS ) to estimate the likelihood of the decoder-based generative models . Nevertheless , these approaches need the access to the generative model for computing the likelihood , which are less favorable comparing to the model agnostic approaches which rely only on a finite generated sample set . On the other hand , the maximum mean discrepancy ( MMD ) ( Gretton et al. , 2012 ) is preferred against its competitors ( Sutherland et al. , 2016 ; Huang et al. , 2018 ) . Therefore , we propose the mmdInf based GANs analysis framework . 3 PROPOSED METHOD ( MMDINF ) In this section , we introduce a PSI algorithm with MMD ( Gretton et al. , 2012 ) . 3.1 POST SELECTION INFERENCE ( PSI ) . Suppose we are given independent and identically distributed ( i.i.d . ) samples X = { xi } mi=1 ∈ Rd×m from a d-dimensional distribution p and i.i.d . samples Y = { yj } nj=1 ∈ Rd×n from another d-dimensional distribution q . Our goal is to find k < d features that differentiate betweenX and Y and test whether each selected feature is statistically significant . Let S be a set of selected features , we consider the following hypothesis test : • H0 : ∑d s=1 ηsD̂ ( X ( s ) , Y ( s ) ) = 0 | S was selected , • H1 : ∑d s=1 ηsD̂ ( X ( s ) , Y ( s ) ) 6= 0 | S was selected , where D̂ ( X ( s ) , Y ( s ) ) is the estimated discrepancy measure for the selected feature s and η = [ η1 , . . . , ηd ] > ∈ Rd is an arbitrary pre-defined parameter . To test the s-th feature , we can set η as a unit vector whose s-th position is 1 and zero otherwise . We employ the post-selection inference ( PSI ) framework ( Lee et al. , 2016 ) to test the hypothesis . Thanks to the cumulative distribution of Theorem 1 , we can compute the p-value of∑d s=1 ηsD̂ ( X ( s ) , Y ( s ) ) under feature selection . Theorem 1 ( Lee et al. , 2016 ) Suppose that z ∼ N ( µ , Σ ) , and the feature selection event can be expressed as Az ≤ b for some matrix A and vector b , then for any given feature represented by η ∈ Rn we have F [ V − ( A , b ) , V + ( A , b ) ] η > µ , η > Ση ( η > z ) | Az ≤ b ∼ Unif ( 0 , 1 ) , where F [ a , b ] µ , σ2 ( x ) is the cumulative distribution function ( CDF ) of a truncated normal distribution with mean µ and variance σ2 truncated at [ a , b ] . Given that α = A Ση η > Ση , the lower and upper truncation points are given by V − ( A , b ) = max j : αj < 0 bj − ( Az ) j αj + η > z , V + ( A , b ) = min j : αj > 0 bj − ( Az ) j αj + η > z . Feature Selection with Discrepancy Measure : Assume we have an estimate of a discrepancy measure for each feature : z = [ D̂ ( X ( 1 ) , Y ( 1 ) ) , . . . , D̂ ( X ( d ) , Y ( d ) ) ] > ∈ Rd ∼ N ( µ , Σ ) , where D̂ ( · , · ) has large positive value when two distribution are different . We select the top-k features with largest discrepancy scores . We denote the index set of the selected k features by S , and that of the unselected k̄ = d− k features by S̄ . This feature selection event can be characterized by D̂ ( X ( s ) , Y ( s ) ) ≥ D̂ ( X ( ` ) , Y ( ` ) ) , for all ( s , ` ) ∈ S × S̄ . Note that we have in total k · k̄ constraints . The selection event can be rewritten as As , ` z ≤ 0 , for all ( s , ` ) ∈ S × S̄ , where As , ` = [ 0 · · · 0 −1︸︷︷︸ s 0 · · · 0 1︸︷︷︸ ` 0 · · · 0 ] and A > s , ` ∈ Rd is a row vector of A ∈ R ( k·k̄ ) ×d . Under such construction , Az ≤ b can be satisfied by setting b = 0 .

Summary: The authors focus on the selection problem of k statistically significant features discriminating 2 probability distributions accessible via samples. They propose a non-parametric approach under the PSI (post selection inference) umbrella using MMD (maximum mean discrepancy) as a discrepancy measure between probability distributions. The idea is to apply (asymptotically) normal MMD estimators, rephrase the top-k selection problem as a linear constraint, and reduce the problem to Lee et al., 2016. The efficiency of the approach is illustrated on toy examples and in GAN (generative adversarial network) context. The technique complements the PSI-based independence testing approach recently proposed by Yamada et al., 2018.