Quantifying Statistical Significance of Neural Network Representation-Driven Hypotheses by Selective Inference

1 INTRODUCTION . The remarkable predictive performance of deep neural networks ( DNNs ) stems from their ability to learn appropriate representations from data . In order to understand the decision-making process of DNNs , it is thus important to be able to explain and interpret DNN representations . For example , in image classification tasks , knowing the attention region from DNN representation allows us to understand the reason for the classification . In the past few years , several methods have been developed to explain and interpret DNN representations ( Ribeiro et al. , 2016 ; Bach et al. , 2015 ; Doshi-Velez & Kim , 2017 ; Lundberg & Lee , 2017 ; Zhou et al. , 2016 ; Selvaraju et al. , 2017 ) ; however , some of them have turned out to be unstable and not reproducible ( Kindermans et al. , 2017 ; Ghorbani et al. , 2019 ; Melis & Jaakkola , 2018 ; Zhang et al. , 2020 ; Dombrowski et al. , 2019 ; Heo et al. , 2019 ) . Therefore , it is crucially important to develop a method to quantify the reliability of DNN representations . In this paper , we interpret these representations as hypotheses that are driven by DNN ( called DNNdriven hypotheses ) and employ statistical hypothesis testing framework to quantify the reliability of DNN representations . For example , in an image classification task , the reliability of an attention region can be quantified based on the statistical significance of the difference between the attention region and the rest of the image . Unfortunately , however , traditional statistical test can not be applied to this problem because the hypothesis ( attention region in the above example ) itself is selected by the data . Traditional statistical test is valid only when the hypothesis is non-random . Roughly speaking , if a hypothesis is selected by the data , the hypothesis will over-fit to the data and the bias needs to be corrected when assessing the reliability of the hypothesis . Our main contribution in this paper is to introduce Selective Inference ( SI ) approach for testing the reliability of DNN representations . The basic idea of SI is to perform statistical inference under the condition that the hypothesis is selected . SI approach has been demonstrated to be effective in the context of feature selections such as Lasso . In this paper , in order to introduce SI for DNN representations , we develop a novel SI algorithm based on homotopy method , which enables us to derive the exact ( non-asymptotic ) conditional sampling distribution of the DNN-driven hypothesis . We use p-value as a criterion to quantify the reliability of DNN representation . In the literature , pvalues are often misinterpreted and there are various source of mis-interpretation has been discussed ( Wasserstein & Lazar , 2016 ) . In this paper , by using SI , we address one of the sources of misinterpreted p-values ; the p-values are biased when the hypothesis is selected after looking at the data ( often called double-dipping or data dredging ) . We believe our approach is a first significant step to provide valid p-values for assessing the reliability of DNN representations . Figure 1 shows an example that illustrates the importance of our method . Related works . Several recent approaches have been developed to visualize and understand a trained DNN . Many of these post-hoc approaches ( Mahendran & Vedaldi , 2015 ; Zeiler & Fergus , 2014 ; Dosovitskiy & Brox , 2016 ; Simonyan et al. , 2013 ) have focused on developing visualization tools for the activation maps and/or the filter weights within trained networks . Others have aimed to identify the discriminative regions in an input image , given a trained network ( Selvaraju et al. , 2017 ; Fong & Vedaldi , 2017 ; Zhou et al. , 2016 ; Lundberg & Lee , 2017 ) . In parallel , some recent studies have showed that many popular methods for explanation and interpretation are not stable with respect to the perturbation or the adversarial attack on the input data and the model ( Kindermans et al. , 2017 ; Ghorbani et al. , 2019 ; Melis & Jaakkola , 2018 ; Zhang et al. , 2020 ; Dombrowski et al. , 2019 ; Heo et al. , 2019 ) . However , there are no previous studies that quantitatively evaluate the stability and reproducibility of DNN representations with a rigorous statistical inference framework . In the past few years , SI has been actively studied for inference on the features of linear models selected by several feature selection methods , e.g. , Lasso ( Lee et al. , 2016 ; Liu et al. , 2018 ; Duy & Takeuchi , 2020 ) . The basic idea of SI is to make inference conditional on the selection event , which allows us to derive the exact ( non-asymptotic ) sampling distribution of the test statistic . Besides , SI has also been applied to various problems ( Bachoc et al. , 2014 ; Fithian et al. , 2015 ; Choi et al. , 2017 ; Tian et al. , 2018 ; Chen & Bien , 2019 ; Hyun et al. , 2018 ; Bachoc et al. , 2018 ; Loftus & Taylor , 2014 ; Loftus , 2015 ; Panigrahi et al. , 2016 ; Tibshirani et al. , 2016 ; Yang et al. , 2016 ; Suzumura et al. , 2017 ; Duy et al. , 2020 ) . However , to the best of our knowledge , there is no existing study that provides SI for DNNs , which is technically challenging . This study is partly motivated by Tanizaki et al . ( 2020 ) where the authors provide a framework to compute p-values for image segmentation results provided by graph cut and threshold-based segmentation algorithms . As we demonstrate in this paper , our method can be also used to assess the reliability of DNN-based segmentation results . Contribution . To our knowledge , this is the first study that provides an exact ( non-asymptotic ) inference method for statistically quantifying the reliability of data-driven hypotheses that are discovered from DNN representation . We propose a novel SI homotopy method , inspired by Duy & Takeuchi ( 2020 ) , for conducting powerful and efficient SI for DNN representations . We conduct experiments on both synthetic and real-world datasets , through which we offer evidence that our proposed method can successfully control the false positive rate , has decent performance in terms of computational efficiency , and provides good results in practical applications . We provide our implementation in the supplementary document and it will be released when this paper is published . 2 PROBLEM STATEMENT . To formulate the problem , we denote an image with n pixels corrupted with Gaussian noise as X = ( X1 , ... , Xn ) > = µ+ ε , ε ∼ N ( 0 , Σ ) , ( 1 ) where µ ∈ Rn is an unknown mean pixel intensity vector and ε ∈ Rn is a vector of Normally distributed noise with the covariance matrix Σ that is known or able to be estimated from external data . We note that we do not assume that the pixel intensities in an image follow Normal distribution in Equation ( 1 ) . Instead , we only assume that the vector of noises added to the true pixel values follows a multivariate Normal distribution . For an image X and a trained DNN , the main target is to identify an attention region ( discriminative/informative region ) in the input image X based on a DNN representation . A pixel is assigned to the attention region if its corresponding value in the representation layer is greater than a pre-defined threshold . We denote the set of pixels ofX divided into attention region and non-attention region as C+X and C − X , respectively . Definition 1 . We define A ( X ) as the event that the result of dividing pixels of image X into two sets of pixels C+X and C − X is obtained by applying a DNN onX , i.e. , A ( X ) = { C+X , C − X } . ( 2 ) Quantifying the statistical significance of DNN-driven hypotheses . Given an observed image xobs ∈ Rn sampled from the model ( 1 ) , we can obtain C+ xobs and C− xobs by applying DNN on xobs . Let us consider a score ∆ that represents the degree to which the attention region differs from the non-attention region . In general , we can define any score as long as it is written in the form ∆ = η > xobs . For example , we can define ∆ as the difference in average pixel values between the attention region and the non-attention region , i.e. , ∆ = mC+ xobs −mC− xobs = 1 |C+ xobs | ∑ i∈C+ xobs xobsi − 1 |C− xobs | ∑ i∈C− xobs xobsi = η > xobs , where η = 1|C+ xobs |1 n C+ xobs − 1|C− xobs |1 n C− xobs , and 1nC ∈ Rn is a vector whose elements belonging to a set C are 1 , and 0 otherwise . If the value of |∆| is sufficiently large , the difference between C+ xobs and C− xobs is significant and the attention region is reliable . To quantify the statistical significance , we consider a statistical hypothesis testing with the following null hypothesis H0 and alternative hypothesis H1 : H0 : µC+ xobs = µC− xobs vs. H1 : µC+ xobs 6= µC− xobs , ( 3 ) where µC+ xobs and µC− xobs are the true means of the pixel values in the attention region and nonattention region , respectively . Given a significance level α ( e.g. , 0.05 ) , we reject H0 if the p-value is smaller than α , which indicates the attention region differs from the non-attention region . Otherwise , we can not say that the difference is significant . In a standard ( naive ) statistical test , the hypotheses in ( 3 ) are assumed to be fixed , i.e. , non-random . Then , the naive ( two-sided ) p-value is simply given as pnaive = PH0 ( |η > X| ≥ |∆| ) = PH0 ( |η > X| ≥ |η > xobs| ) . ( 4 ) However , since the hypotheses in ( 3 ) are actually not fixed in advance , the naive p-value is not valid in the sense that , if we reject H0 with a significance level α , the false detection rate ( type-I error ) can not be controlled at level α , which indicates that pnaive is unreliable . This is due to the fact that the hypotheses ( the attention region ) in ( 3 ) are selected by looking at the data ( the input image ) , and thus selection bias exists . This selection bias is sometimes called data dredging , data snooping or p-hacking ( Ioannidis , 2005 ; Head et al. , 2015 ) . Selective inference ( SI ) for computing valid p-values . The basic idea of SI is to make inference conditional on the selection event , which allows us to derive the exact ( non-asymptotic ) sampling distribution of the test statistic η > X in an attempt to avoid the selection bias . Thus , we employ the following conditional p-value pselective = PH0 ( |η > X| ≥ |η > xobs| | A ( X ) = A ( xobs ) , q ( X ) = q ( xobs ) ) , ( 5 ) where q ( X ) = ( In − cη > ) X with c = Ση ( η > Ση ) −1 . The first condition A ( X ) = A ( xobs ) indicates the event that the result of dividing pixels into an attention region and non-attention region for a random image X is the same as that of the observed image xobs , i.e. , C+X = C + xobs and C−X = C − xobs . The second condition q ( X ) = q ( xobs ) indicates the component that is independent of the test statistic forX is the same as the one for xobs . The q ( X ) corresponds to the component z in the seminal SI paper of Lee et al . ( 2016 ) ( Sec 5 , Eq 5.2 and Theorem 5.2 ) . The p-value in ( 5 ) , which is called selective type I error or selective p-values in the SI literature ( Fithian et al. , 2014 ) , is valid in the sense that PH0 ( pselective < α ) = α , ∀α ∈ [ 0 , 1 ] , i.e. , the false detection rate is theoretically controlled at level α indicating the selective p-value is reliable . To calculate the selective p-value in ( 5 ) , we need to identify the conditional data space . Let us define the set of x ∈ Rn that satisfies the conditions in ( 5 ) as X = { x ∈ Rn | A ( x ) = A ( xobs ) , q ( x ) = q ( xobs ) } . ( 6 ) According to the second condition , the data in X are restricted to a line ( Sec 6 in Liu et al . ( 2018 ) , and Fithian et al . ( 2014 ) ) . Therefore , the set X can be re-written , using a scalar parameter z ∈ R , as X = { x ( z ) = a+ bz | z ∈ Z } , ( 7 ) where a = q ( xobs ) , b = Ση ` ( η > ` Ση ` ) −1 , and Z = { z ∈ R | A ( x ( z ) ) = A ( xobs ) } . ( 8 ) Now , let us consider a random variable Z ∈ R and its observation zobs ∈ R that satisfyX = a+bZ and xobs = a+ bzobs . Then , the selective p-value in ( 5 ) is re-written as pselective = PH0 ( |η > X| ≥ |η > xobs| |X ∈ X ) = PH0 ( |Z| ≥ |zobs| | Z ∈ Z ) . ( 9 ) Since the variable Z ∼ N ( 0 , η > Ση ) under the null hypothesis , the law of Z | Z ∈ Z follows a truncated Normal distribution . Once the truncation region Z is identified , the selective p-value ( 9 ) can be computed as pselective = F Z 0 , η > Ση ( −|z obs| ) + 1− FZ0 , η > Ση ( |z obs| ) , ( 10 ) where F Em , s2 is the c.d.f . of the truncated normal distribution with mean m , variance s 2 and truncation region E . Therefore , the most important task is to identify Z . Extension of the problem setup to hypothesis driven from DNN-based image segmentation . We interpret the hypothesis driven from image segmentation result as the one obtained from the representation at output layer instead of internal representation . Our problem setup is general and can be directly applied to this case . For example , we can consider the attention region as the object region and the non-attention region as the background region . Then , we can conduct SI to quantify the significance of the difference between object and background regions . We note that we consider the case where the image is segmented into two regions—object and background—to simplify the problem and notations . The extension to more than two regions is straightforward .

Summary: This paper proposed a novel method which to quantify the reliability of DNN-driven hypotheses in a statistical hypothesis testing framework. Naive statistical testings are not appropriate for the DNN-driven hypotheses, where the hypotheses are selected by looking at the data(i.e. The selection bias exists). To address this problem, the authors developed a novel homotopy method under the Selective-Inference(SI) framework, which can derive the exact sampling distribution of the DNN-driven hypotheses. In this paper,  the authors mainly focus on DNNs which consist of affine operations, max-operations, and piecewise-linear activation. As described by Lee et al. (2016), the main idea of SI is to make the inference conditional on the selection event. Specifically to the DNN-driven hypotheses, the authors proposed a novel method that consists of two steps, 1) Adding extra conditioning to make the problem traceable. 2) Combining multiple over-conditioning cases by homotopy method to solve the over-conditioning problem. The experimental results on both synthetic and real-world datasets illustrate the proposed method can successfully control the FP error rate.