Quantifying Statistical Significance of Neural Network Representation-Driven Hypotheses by Selective Inference
Authors: Vo Nguyen Le Duy
Shogo Iwazaki
Ichiro Takeuchi
Venue:  (2021)
Keywords: Deep Neural Network Representation
Reliability
Selective Inference
Statistical Hypothesis Testing
p-value

Abstract: In the past few years, various approaches have been developed to explain and interpret deep neural network (DNN) representations, but it has been pointed out that these representations are sometimes unstable and not reproducible. In this paper, we interpret these representations as hypotheses driven by DNN (called DNN-driven hypotheses) and propose a method to quantify the reliability of these hypotheses in statistical hypothesis testing framework. To this end, we introduce Selective Inference (SI) framework, which has received much attention in the past few years as a new statistical inference framework for data-driven hypotheses. The basic idea of SI is to make conditional inferences on the selected hypotheses under the condition that they are selected. In order to use SI framework for DNN representations, we develop a new SI algorithm based on homotopy method which enables us to derive the exact (non-asymptotic) conditional sampling distribution of the DNN-driven hypotheses. We conduct experiments on both synthetic and real-world datasets, through which we offer evidence that our proposed method can successfully control the false positive rate, has decent performance in terms of computational efficiency, and provides good results in practical applications.

Under review as a conference paper at ICLR 2021
QUANTIFYING STATISTICAL SIGNIFICANCE OF NEU- RAL NETWORK REPRESENTATIIN--DRIVNN HYPOTHE- SES BY SELECTIVE INFERENCE
Anonymous authors Paper under double-bling blind review
ABSTRACT
the past few years, various approaches have been developed to explain and in- terpret deep neural network (DNN) representations, but it has been pointed out that these representations are sometimes unstable and not reproducible. paper, we interpret these representations as hypotheses driven by DNN (called DNN-driven hypotheses) and propose a method to quantify the reliability of these hypotheses in statistical hypothesis testing framework. To this end, we introduce Selective Inference (SI) framework, which has received much attention in the past few years as new statistical inference framework for data-driven hypotheses. The basic idea of SI is to make conditional inferences on the selected hypothe- ses under the condition that they are selected. In order use SI framework for DNN representations, we develop a new SI algorithm based on homotopy method which enables to derive the exact (non-asymtootic) conditional sampling distri- bution of the DNN-driven hypotheses. In this paper, we demonstrate the proposed method in computer vision tasks as practical examples. We conduct experiments both synthetic and real-world datasets, through which offer evidence that our proposed method can successfully control the false positive rate, has decent performance in terms of computational efficiency, and provides good results in practical applications.
this
INTRODUCTION
The remarkable predictive performance of deep neural networks (DNNs) stems from their ability to learn appropriate representations from data. order to understand the decision-making process of DNNs, it is thus important to be able to explain and interpret DNN representations. For example, in image classification tasks, knowing the attention region from DNN representation allows us to un- derstand the reason for the classificatinn. In the past few years, several methods have been developed to explain and interpret DNN representations (Ribeiro et al. [016]] Bach et al. 2015: Doshi Velez & al. (017); however, some of them have turned out to be unstable and not reproducible (Kindermans et al. 2017 Ghorbani et al. 2019 Mclis & Jaakkola 2018 Zhang et al. 2020 Dombrowski al. [2019] Heo et al. 2019). Therefore, it is crucially important to develop method to quantily the reliability of DNN representations.
Kim 2017; Lundberg & Lee 2017 [Zhou et [016)
In this paper, we interpret these representations as hypotheses that are driven by DNN (called DNN- driven hypotheses) and employ statistical hypothesis testing framework to quantify the reliability of DNN representations. For example, in an image classification task, the reliability of an attention region can be quantified based on the statistical significance of the difference between the attention region and the rest of the image. Unfortunately, however, traditional statistical test cannot be applied to this problem because the hypothesis (attention region in the above example) itself is selected by the data. Traditional statistical test is valid only when the hypothesis is non-random. Roughly speaking, if a hypothesis is selected by the data, the hypothesis will over-fi fit to the data and the bias needs to be corrected when assessing the reliability of the hypothesis.
Our main contribution in this paper is introduce Selective Inference (SI) approach for testing the reliability of DNN representations. The basic idea of SI is to perform statistical inference under the condition that the hypothesis is selected. SI approach has been demonstrated to be effective

---

Under review as a conference paper at ICLR 2021
Input Image
Attention Map
Attention Region
Input Image
Attention Map
Region
Image without region
Image with tumor region
(a) naive-p = 0.00 (false positive-wromg wrong detection) and selective-p = 0.94 (true negative)
(b) naive-p = 0.00 (true positive) and selective-p 0.02 (true positive)
Figure Examples of the proposed method on brain tumor image classification. Given a CNN trained to classify tumor versus non-tumor brain images in advance, our method provides the statis- tical significance of the attention region for each test image the form of p-values by comparing the pixel information in the attention and non-attention regions. Since the attention region is se- lected by the input image, the p-value obtained by the naive comparison of the two regions (naive y-value) is highly biased. In the left-hand side figure where there is no brain tumor, the naive p-value is nearly zero (indicating false Sesitive-incorres selective p-value is large (indicating true negative). On the other hand, in the right-hand hand side figure where there actually exist a brain tumor, both the naive p-value and the selective p-values are very small (indicating true positive). The proposed selective inference method can provide valid exact (non-asymptotic) p-values for DNN representations such as attentions.
identifying tumor region), while the proposed
in the context of feature selections such Lasso. In this paper, in order introduce SI for DNN representations, we develop a novel SI algorithm based on homotopy method, which enables us to derive the exact (non-asymptotic) conditional sampling distribution of the DNN-driven hypothesis. We use p-value as a criterion to quantify the reliability of DNN representation. In the literature, p- values are often misinterpreted and there are various source of mis-interpretation has been discussed Wasserstein & Lazar (016). In this paper, by using SI, we address one of the sources of mis- interpreted p-values; the p-values are biased when the hypothesis is selected after looking the data (often called double-dipping or data dredging). We believe our approach is a first significant step to provide valid p-va p-values for assessing the reliability of DNN representations. Figure shows an example that illustrates the importance of our method.
Related works. Several recent approaches have been developed visualize and understand trained DNN. Many of these post-hoc approaches (Mahendran & Vedaldi 2015) Zeiler & Fergus Brox] 2016]] [Smonyyan et al.) (2013) have focused on developing visualization tools for the activation maps and/or the filter weights within trained networks. Others have aimed to identify the discriminative regions in an input image, given trained network (Selvaraju et al.) 2017 Fong & Vedaldi, 2017 Zhou et al., 2016 Lundberg & Lee 2017). In parallel, some recent studies have showed that many popular methods for explanation and interpretation are not stable with respect the perturbation or the adversarial attack on the input data and the model (Kindermans let al. 2017] Ghorbani et af. 2219)) Melis & Jaakkola 2018 Zhang et al., 2020 [Dobbrowski et al.) 2019 Heo al. (0119). However, there are no previous studies that quantitatively evaluate the stability and reproducibility of DNN representations with a rigorous statistical inference framework.
2014 Dosovitskiy
In the past few years, SI has been actively studied for inference on the features of linear models selected by several feature selection methods, e.g., Lasso (Lee et al. 2016)] Liu et al. 2018 Duy & Takeuchi 2020). The basic idea of SI make inference conditional on the selection event, which allows us to derive the exact -somatyyttooic sampling distribution of the test statistic. Besides,
SI has also been applied to various problems (Bachoc et al. 2014 Fithian et af.
2015 Choi al.,
2017
Tian et al.| 2018 Chen & Bien 2019 Hyun et al. 2018 Bachoc et al. 2018 Lofuus & Taylor,
2014
Loftus 2015] Panigrahi et al. 2016 Tibshirani et al. 2016 Yang et al.
2016 Suzumura
ct al.
2017 Duy et al. 2020). However, to the best of our knowledge, there is existing study that
provides SI for DNNs, which is technically challenging. This study is partly motivated by Tanizaki
et al.
22000) where the authors provide framework to compute p-values for image segmentation results provided by graph cut and threshold based segmentation algorithms. As we demonstrate in this paper, method can be also used to assess the reliability of DNN-based segmentation results.
2

---

Under review as a conference paper at ICLR 2021
Contribution. To our knowledge, this is the first study that provides exact (non-asympiotic) inference method for statistically quantifying the reliability of data-driven hypotheses that are dis- covered from DNN representation. We propose novel SI homotopy method, inspired by Duy & "Takeuchi 2020), for conducting powerful and efficient SI for DNN representations. We conduct experiments on both synthetic and real-world datasets, through which offer evidence that our proposed method can successfully control the false positive rate, has decent performance in terms of computational efficiency, and provides good results in practical applications. We provide our implementation in the supplementary document and it will be released when this paper is published.
2 PROBLEM STATEMENT
To formulate the problem, we denote an image with n pixels corrupted with Gaussian noise as
X = -X1, Xa) 4 E, E~ N(0, E);
where 14 R is an unknown mean pixel intensity vector and & IR" is vector of Normally distributed noise with the covariance matrix Σ that is known or able to be estimated from external data. We note that we do not assume that the pixel intensities in an image follow Normal distribution in Equation T.. Instead, we only assume that the vector of noises added to the true pixel values follows a multivariate Normal distribution. For an image X and a trained DNN, the main target is to identify an attention region (diicriminnaiieeiifornma a DNN representation. pixel is assigned to the attention region if its corresponding value in the representation layer is greater than a pre-defined threshold. We denote the set of pixels of X divided into attention region and non-attention region as C and Cx respectively.
region) in the input image X based on
Definition 1. We define A(X) as the event that the result of dividing pixels of image X into two sets of pixels C* and Cx is obtained by applying a DNN on X, i.e.,
A(X) Cx)
Quantifying the statistical significance of DNN-driven hypotheses. Given an observed image ob Rn sampled from the model (1), we can obtain Ch and Cgabe by applying DNN on zb Let us consider a score A that represents the degree to which the attention region differs from the non-attention region. In general, we can define any score as long as it is written in the form A = 7 x obas For example, we can define A as the difference in average pixel values between the attention region and the non-attin attenttnnioon region, i.e.,
A mel gobs
me
obs
-bb 7) obe
AEC
iEC aulo
where 77 1e ohe oolo
1
and 12 E 18 is vector whose elements belonging to
set C are 1, and O otherwise.
If the value of Al is sufficiently large, the difference between C+ and C go is significant and the attention region is reliable. To quantify the statistical significance, we consider a statistical hypothesis testing with the following null hypothesis Ho and alternative hypothesis H,:
Ho
Pet -Mc-... gols
vs. H1 Me+ gobs
# Mc
where Mc- and Mc- goos attention region, respectively. Given significance level a (e.g., 0.05), we reject Ho the p-value is smaller than a, which indicates the attention region differs from the non-attention region. Otherwise, we cannot say that the difference is significant.
are the true means of the pixel values in the attention region and non-
In standard (naive) statistical test, the hypotheses in Then, the naive (two-sided) p-value is simply given as
assumed to be fixed, i.c., non-random.
(4) However, since the hypotheses in 3 are actually not fixed in advance, the naive paluu is not valid in the sense that, if we reject H, with significance level a, the false detection rate (type-I error) cannot be controlled at level a, which indicates that Pusive is unreliable. This is due to the fact that the hypotheses (the attention region) in (3) are selected by looking at the data (the input image), and thus selection bias exists. This selection bias is sometimes called data dredging, data snooping or p-hacking (Toannidis. 2005 Head et al. 2015).
Puaive Pn, (In x A = PHo (nn"x X |n T nhh
3

---

Under review as a conference paper at ICLR 2021
Selective inference (SI) for computing valid p-values. The basic idea of SI is make inference conditional on the selection event, which allows us to derive the exact (non-asymptotic) sampling distribution of the test statistic v X in attempt avoid the selection bias. Thus, employ the following conditional p-value
Pselective = PHo (17) [nnX]] x 2 nn x: bbs
A = A(xx ((X) = ((x
(5)
where ((X) (In cn )X with == Ty(n1 The first condition A(X) A A(xobo)
indicates the event that the result of dividing pixels into attention region and non-attention region for a random image X is the same as that of the observed image x i.e., CX = C and
Cx = C Coo The second condition (XX) = q(x indicates the component that is independent of the test statistic for X is the same the one for ob The ((X) corresponds to the component in the seminal SI paper of Lee al. (2016) (Sec 5, Eq 5.2 and Theorem 5.2). The p-value in ( which is called selective type I error selective p-values in the SI literature (Fithian et al.) 2014), is valid in the sense that H., (Pselective < a) = a, Va E [0, 1], i..., the false detection rate is theoretically controlled level indicating the selective p-value is reliable.
To calculate the selective p-value in (5). we need to identify the conditional data space. Let us define the set of x E R that satisfies the conditions in (5)
X = {æ E R" A(xx) A(x q((x)) - )x)
According to the second condition, the data in A' are restricted to line (Sec 6 in Liu et al. (2018) and Fithian et al. (2014)). Therefore, the set X' can be re-written, using a scalar parameter 2 E IR as
X {=<(z) a + bz Z}.
where a = q(xobs), b = Emp Enr ,,a and
2 {= TR. A(x(2))
A(m(z))=A(x<))))) A(
(8)
Now, let us consider a random variable Z E R and its observation obs E IR that satisfy X = a a-bb bZ and obs + bbs Then, the selective p-vuuue in (5) is re- re-written as
Pselective PHo (In7 T X 2 In obs X E X)) PHo (12) > 12 obs
I E Z).
(9)
Since the variable Z ~ N(0, n En) under the null hypothesis, the law of Z 2 E Z follows a truncated Normal distribution. Once the truncation region Z is identified, the selective p-value (9) can be computed as
Pselective
F En (-122be)) + 1 F 1.2--, En (1
(10)
where FË sa is the c.d.f. of the truncated normal distribution with mean m, variance 5² and trunca- tion region& E. Therefore, the most important task to identify Z.
Extension of the problem setup to hypothesis driven from DNN-based image segmentation. We interpret the hypothesis driven from image segmentation result as the one obtained from the representation at outpur layer instead of internal representation. Our problem setup is general and can be directly applied to this case. For example, we can consider the attention region as the object region and the non-attention region as the background region. Then, we can conduct SI quantify the significance of the difference between object and background regions. We note that we consider the case where the image segmented into two regions-obje and background simplify the problem and notations. The extension to more than two regions is straightforward.
3 PROPOSED METHOD
As we discussed $2, to calculate the selective p-value, the truncation region 2 in Equation (
must be identified. To construct Z, we have to 1) compute A(x(=)) for all E E, and 2) identify the set of intervals of 2 on which A(x(z)) = o(x However, seems intractable to obtain A(x(22) for infinitely many values of R.
Our first idea to develop SI for DNN is that we additionally condition on some extra event to make the problem tractable. We now focus piecewise-linewt c.g., ReLU, Leaky ReLU (the extension to general AFs is discussed later). Then, consider additionally conditioning on the selected piece of each piecewise-linen AF in the DNN.
class of DNNs whose activation functions (AFs) are
4

---

Under review as a conference paper at ICLR 2021
Observed Representation
DNN (xx
Will spn
of sprit
8ill Spril
8h sprl
Parametrized
Proposed Method
Conditional Data Space
Data Space
(x E RR (dxx af 0b5), q(x) = ((x)))
A schematic illustration of the proposed method. By applying DNN on the observed image ob.s, we obtain an representation. Then, we parametrize xobs with a scalar parameter z in the dimension of test-statistic to identify the subspace X' whose data has the same representation as ahs has. Finally, the valid statistical inference is conducted conditional on X' We introduce a homotopy method for efficiently characterizing the conditional data space X.
Figure 2:
Let (z(x) be "the selected piece" of piecewise-linear AF at the j-th unit in a DNN for given input image and let s(x) be the set of (x) for all the nodes in DNN
Definition 2.
For example, for a ReLU activation function, sz(x) takes either 0 or depending on whether the input the j-th unit located the flat part (inactive) the linear part (active) of the ReLU function. Using the notion of selected pieces (xr), instead of computing the selective p-value in (9) we consider the following reediionig (oc) conditional p-value
Psolective = PH, (1 2 obs Z 2o0),
oe
(11)
where 2o0 (= E E A(x(z)) {
= bbo) s s<x(2)) (hs
However, such over-
conditioning in SI leads to the loss of statistical power (Lee et al. 2016).
Our second idea is develop homotopy method resolve the over-conditioning problem, i.e., remove the conditioning of s(x(2)) = 8(xr With the homotopy method, we can efficiently compute A(xx(z) in finite number of operations without the need of considering infinitely many values of z E I. which is subsequently used to obtain truncation region Z in (8) The main idea to compute finite number of breakpoints which one node of the network going to change its status from active to inactive or vice versa. This concept is similar to the regularization path of Lasso where we can compute a finite number of breakpoints at which the active changes.
To this end, we introduce two-step iterative approach generally described as follows (see Fig. 2
Step (over-conditioning step). Considering over-conditioning case by additionally conditioning on the selected pieces of all the hidden nodes in the DNN.
Step 2 (homotopy step). Combining multiple over-conditioning cases by homotopy method to obtain A(x(2)) for all 2 E R.
3.1 STEP1: CONDITIOITGG STEP
We now show that by conditioning on the selected pieces a (bx of all the hidden nodes, we can write the selection event of the DNN as set of linear inequalities.
Lemma 1. Consider class of DNN which consists of affine operations and piecewise-linear AFs. Then, the over-conditioning region is wrillen as
zor = E ((a(a Da())))))))))))))))))))))())))
x<2)) S 2h(a(w=le)))
for a matrix ( and a vector 2 ()(((()))) which depend only on the selected pieces s shoo))).
Proof. For the class DNN, by fixing the selected pieces all the piecewise- AFs, the input each AF is represented by affine function of an image a.. Therefore, the condition for selecting
5

---

Under review as a conference paper at ICLR 2021
piece in piecewise- linear AF, 8, (x(()) Sj (2 is written as linear inequality w.r.t x(z). Similarly, the value of each unit in the representation layer is also written as an affine function of <<2). Since the attention region is selected if the value is greater than threshold, the choice of attention region A(x(=)) = A(x is characterized by a set of linear inequalities w.r.t. ar2). IT
Furthermore, let consider max-operation, an operation to select the max one from finite number of candidates. A max max-operation is characterized by a set of comparison operators, i.e, inequali- ties. Let consider DNN which contains max-operators, and denote $(x) be the set of selected candidates of all the max-operators for an input image x.
Corollary 1. Consider a class of DNN which consists of affine operations, mux-operations and piecewise-linear AFs. Then, region Zo defined as goe= == {= E 200 ((<(2)) = ((xxbb)) is characterized by set of linear inequalities w.r.t x²2)).
The proof is shown in Appendix A.1
Remark 1. In this work, we mainly focus on the trained DNN where the activation functions used at hidden layers are piecewise linear, e-g., ReLU, Leaky ReLU, which is commonly used in CNN. Oth- erwise, if there is any specific demand to use non-piecewise linear functions such as sigmoid or tanh at hidden layers, we can apply some piecewise-linea approximation approach to these functions. We provided examples about the approximation for this case in Appendix A.5
Remark Most of the basic operations in a trained neural network are written as affine operations. In the traditional neural network, the multiplication results between the weight matrix and the output CNN, the main
of the previous layer and its summation with bias vector is affine operation. convolution operation is obviously an affine operation. Upsampling operation is also affine.
Remark 3. Although the max- max-pooling operation is not affine operation, it can be written set of linear inequalities. For instance, 8n = max U1, U2 0/} can be written as a set {e v S (v1, 12; vy) and e; is a standard basis vector with
position Remark 4. In Remark I we mentioned that we need to perform piecewise linear approximation for non- oin-piceeess linear activations. However, if these functions are used at output layer, we do not need to perform the approximation task because we can define the set of linear inequalities based on the values before doing activation. See the next example for the case of sigmoid function.
e2 v, e & Bg v), where
Example 1. Let us consider a 3-layer neural network with n input nodes, h hidden nodes and ouput nodes. Let W(1) E Rhxn and (1) R respectively be the weight matrix and bias vector between input layer and hidden layer, and (2)) E ga and (2) E R" respectively be the weight matrix and bias vector between hidden layer and output layer. The activation function at hidden layer
is ReLU, and we use sigmoid function output layer. At the hidden layer, for any node [h), the selection event is written as
( x + w (1) 2 0, W
if the output of ReLU function at jth node 2 0,
11
x + wj < O, otherwise. J:
Let (1) E th and (11) E R be the vectors in which "j((k] 1, (1)
the output of ReLU
function at the jth node 2 0, and af") =0, 0, = -1 otherwise. Then we have the linear inequality
system OΙæ S 1/1 where O = (-s (1) w ..... - sn ( and 1/1 (s1 (1) "("" ..... sh
( w/ (
Next, for any output node E [n]. the selection event-a linear inequality is written as
2 0. if the output of sigmoid function at th node 2 0.5, w (w + (()) o a (2) + (2) ( ((W() + w)()) o (1) + (2 0, otherwise,
where o is the element-wiss product. Similar to the hidden layer, we can also construct the linear inequality system Ozæ 222 at the output layer. Finally, the whole linear inequality system is written as
Ox v ab (O; Q2) S (1b1 4b22
(12)

---

Under review as a conference paper at ICLR 2021
Algorithm 1 compute.soluti
Input: a, b, [amin: Fmax
1: Initialization: f 1, At Zmin: T
2: while 22 < Amax do
3: Obtain A(x(2)) by applying trained DNN to a<2)) a bz:
4: Compute the next breakpoint *++1 Equation 13
Then assign T T (2441), and =
5: end while
Output: (A(m(Z))ZEET
3.2 STEP 2: HOMOTOPY STEP
We now introduce homotopy method to compute A(æ(z)) based over-conditioning step.
Lemma 2. Consider real value zt. By applying a trained DNN to a=(2). we obtain set of linear inequalities ((s(m(=))) ((z)) S *b(a(a(=))) Then, the next breakpoint 2441 > at which the status of one node going to be changed from active inactive or vice versa, iee. the sign of one linear inequality is going to be changed, is calculated by
u (=(a(()))) ((((((())))))))
min 2++1 (1((
(13)
(O(e(a(z))))
The proof is shown in Appendix A.2 Algorithm shows our solution to efficiently identify A(x(2)). In this algorithm, multiple breakpoints 21 < 22 ... < 2|T| are computed one by one. Each breakpoint 2,, & [[T]], indicates a point at which the sign of one linear inequality is changed, i.e. the status of one node in the network is going to change from active to inactive or vice versa. By identifying all these breakpoints {2) [e(TT))) the solution path is given by A(x(2)) A(x(2)) if E ( E [[ For the choice of [zmin; zmax]; see Appendix A.3
EXPERIMENT
We highlight the main results. Several additional results and details can be found in Appendix A.6
Numerical Experiments. We demonstrate the performances of two versions of the proposed method: proposed methhod (homotopy) and proposed-methode The p-values in these two versions were computed by (5) and (1I), respectively. Besides, we also compared the proposed methods with the naive p-value in (4) and the permutation test. The details of permutation test procedure is de- generated 120 null images in which 2; E [n]] NTO, 1) for each n E {64, 256, 1024, 4096). To test the power, we generated images (I1, .... [n, with
scribed in Appendix A.6 To test the FPR control,
(x1, ..., I.,)
256 for each true average difference in the underlying model Mct Pc- = A, E {0.5, 1.0, 1.5, 2.0}. For each case, we ran 120 trials. We chose the signifi- cance level a = 0.05. For more information about the setup as well as the the structure of a neural network, see the experimental setup paragraph in Appendix A.6 The results of FPR control shown in the first part of Fig. 3 The proposed methods could successfully control the FPR under 0.05 while the naive method can not. Since the naive method fails to control FPR, did not consider the power anymore. the second part of Fig, 3 we see that the ever-conditioning option has lower power than the homotopy method. It because the truncation region in proposed-method oc is shorter than the one in proposed-method (homotopy), which is demonstrated in the third part of Fig. 3 The last part of Fig. 3 shows the reason why the proposed homotopy method is efficient. With the homotopy method, we only need to consider the number of encountered intervals on the line along the direction of test statistic which is almost linearly increasing practice.
Real-data examples. We performed comparison on real-world brain image dataset, which in- cludes 939 images with tumor and 941 images without tumor. We first compared our method with permutation test in terms of FPR control. The results test could not control the FPR properly, we did not compare the power. The comparisons between naive p-value and selective y-value shown in Figs. 6nd The naive p-value was still small
shown in Table I Since the permutation

---

Under review as a conference paper at ICLR 2021
Palse Postiiv Rate (FPR)
Power
Interval Length
proptaci
Parcourted Intenak
2.
preposel michooao
00:
8
602
24
20
13
64
255
1034
ec pipposs--mahhed
15
259
1026
data
Figure 3: Results of false positive rate (FPR), power, length of interval, and encountered interval.
Table 1: FPR and power comparisons on real-world brain image dataset.
Attention Detection Task
Segmentation Task
FPR
Power
FPR
Power
Proposed Method
0.056
0.669
0.057
0.683
Permutation Test
0.850
0.640
Image
Attention Map
Region
Input Image
Attention
Attention
(a) naive-p = 0.00 and selective-p = 0.87
(b) naive-p 0.00 and selective-p 0.29
Figure 4: Inference on hypotheses obtained from internal representation (without tumor region).
Inout Image
Attention Map
Region
Attention
Attention
(a) naive-p = 0.00 and selective-p = 0.01
(b) naive-p- 0.00 and selective-p = 2.17e-5
Figure 5: Inference hypotheses obtained from internal representation (with tumor region).
Input Image
Segmentation
Input Image
Segmentation
Input Image
Segmentation Result
Pusive 0.00,
Prelective
Poaive 0.00, Psolective 0.45
Pusive 0.00, Prelective 0.16
Figure 6: Inference on hypotheses obtained from output representation (without tumor region).
Imput Segmentation Result
Segmentation
Input Segmentation Recult
Puaive 0.00, Pselective 8.15e-5 Pnaive 0.00, Pselective 9.20e-3 Pnaive 0.00, Pselective 1191e-2
Figure 7: Inference hypotheses obtained from output representation (with tumor region).
when the image has tumor region, which indicates that the naive p-values cannot be used for quantifying the reliability of DNN-driven hypotheses. The proposed method could successfully identify false positive detections well as true positive detections.
8

---

Under review as a conference paper at ICLR 2021
5 CONCLUSION
We proposed a novel method to conduct statistical inference on the significance of the data-driven hypotheses driven from neural network representation based the concept of selective inference. In the context of explainable DNN or interpretable DNN, we are primarily interested in the reliability of the trained network when given new imputs (not training inputs). Therefore, the validity of our proposed method does not depend on how the DNN is trained.
In regard of the generality, the proposed method can be applied any kind of network as long as the network operation is characterized by a set of linear incqualities (or approximated by piecewise- linear functions) because all the algorithms and theories $2 and $3 only depend on the property of each component and does not depend on the entire structure of the network.
We believe that this paper provides significant step toward reliable artificial intelligence (AI) and open several directions for stutistically evaluating the reliability of DNN representation-driven hy- potheses. Although it is not necessary to account
impact of training in this paper because the validity of our proposed method does not depend on how the DNN is trained, defining new prob- lem setup and providing solution for the case in which the training process needs to be considered is potential direction. Moreover, widening the practical applicability of the proposed method in other fields such as NLP and signal processing would also represent a valuable contribution.
REFERENCES
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus- Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer- wise relevance propagation. PloS one, 10(7)=e01301 301110314, 40, 2015.
François Bachoc, Hannes Leeb, and Benedikt M Pötscher. Valid confidence intervals for post- model-selcction predictors. arXiv preprint arXiv:1412.46600 2014.
François Bachoc, Gilles Blanchard, Pierre Neuvial, al. On the post selection inference constant under restricted isometry properties. Electronic Journal of Statistics, 12(2):3736 2018.
Shuxiao Chen and Jacob Bien. Valid inference corrected for outlier removal. Journal of Computa- tional and Graphical Statistics, pp. 1-12, 2019.
Yunjin Choi, Jonathan Taylor, Robert Tibshirani, et al. Selecting the number of principal compo- nents: Estimation of the true rank of noisy matrix. The Annals of Statistics, 45(6))2590-2617, 2017.
Ann-Ken Man-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations be manipulated and geometry is to blame. In Advances in Neural Information Processing Systems, pp. 13589-13600, 2019.
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning. arXiv preprint arXiv: 1702.08608 2017.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4829- 4837, 2016.
Vo Nguyen Le Duy and Ichiro Takeuchi. Parametric programming approach for powerful lasso selective inference without conditioning on signs. arXiv preprint arXiv:2004.09749, 2020.
Vo Nguyen Le Duy, Hiroki Toda, Ryota Sugiyama, and Ichiro Takeuchi. Computing valid p-value for optimal changepoint by selective inference using dynamic programming. arXiv preprint arXiv:2002.09132 2020.
William Fithian, Dennis Sun, and Jonathan Taylor. Optimal inference after model selection. arXiv preprint 4100:274 2014.
William Fithian, Jonathan Taylor, Robert Tibshirani, and Ryan Tibshirani. Selective sequential model selection, arXiv preprini arXiv:15/2.02534 2015.

---

Under review as a conference paper at ICLR 2021
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Megan I. Head, Luke Holman, Rob Lanfear, Andrew T Kahn, and Michael D Jennions. The extent and consequences of p-hacking in science. PLoS Biol, 13(3)e1002106 2015.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversar- ial model manipulation. In Advances in Neural Information Processing Systems, pp. 2925-2936, 2936, 2019.
Sangwon Hyun, Kevin Lin, Max G'Sell, and Ryan Tibshirani. Post-selection inference for changepoint detection algorithms with application copy number variation data. arXiv preprint arXiv: 2018.
John PA loannidis. Why most published research findings are false. PLoS medicine, 2(8):e 2005.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv preprint arXiv:1711.00877. 2017.
Jason D Lee, Dennis L Sun, Yuekai Sun, Jonathan E Taylor, et al. Exact post-selection inference, with application the lasso. The Annals of Statistics, 44(3)907-9977, 2016.
Keli Liu, Jelena Markovic, and Robert Tibshirani. More powerful post-selection inference, with application to the lasso. arXiv preprint arXiv:1801.00037. 2018.
Joshua R Loftus. Selective inference after cross-validation. arXiv preprint arXiv: 2015.
Joshua Loftus and Jonathan Taylor. significance test for forward stepwise model selection. 2014.
arXiv preprint
Scout M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Advances in neural information processing systems, pp. 4765-4774, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp-
5188-5196, 2015.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, pp. 7775-7784, 2018.
Snigdha Panigrahi, Jonathan Taylor, and Asaf Weinstein. Bayesian post-selection inference in the linear model. arXiv preprint arXiv:1605. 160.00000000884. 28, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. arXiv preprint /312.6631 2013.
Shinya Suzumura, Kazuya Nakagawa, Yuta Umezu, Koji Tsuda, and Ichiro Takeuchi. Selective inference for sparse high-order interaction models. In Proceedings of the 34th International Con- ference Machine Learning- Volume 70, pp. 3338-3347. JMLR. org, 2017.
10

---

Under review as a conference paper at ICLR 2021
Kosuke Tanizaki, Noriaki Hashimoto, Yu Inatsu, Hidekata Hontani, and Ichiro Takeuchi. Computing valid p-values for image segmentation by selective inference. 2020.
Xiaoying Tian, Jonathan Taylor, et al. Selective inference with a randomized response. The Annals of Statistics, 46(2):679-710, 2018.
Ryan Tibshirani, Jonathan Taylor, Richard Lockhart, and Robert Tibshirani. Exact post-selection inference for sequential regression procedures. Journal of the American Statistical Association, 111(514):600-622) 620, 2016.
Ronald L Wasserstein and Nicole Lazar. The asa statement on p-values: context, process, and purpose, 2016.
Fan Yang, Rina Foygel Barber, Prateek Jain, and John Lafferty. Selective inference for group-sparse linear models. In Advances in Neural Information Processing Systems, pp. 2469-2477, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 518-833. Springer, 2014.
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In 29th {USENIX) Security Symposiun ((USENIX) Security 20), 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.


Reviews:
[{"review_id": "iNXHjVd4-m1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a framework of selective inference to rigorously study the problem of finding significant attention areas of a neural network model (with certain constraints). \n\nPros\n- The proposed method yields good performance in power in practice.\n\n- The FDR of the proposed method is theoretically controlled.\n\n- The authors proposed an efficient algorithm to find certain kinds of attention areas. \n\nCons & Questions: \n\n- Is the assumption (1) necessary for the proposed non-asymptotic method? Please address. If it is necessary, please also address why this assumption is practical, or existing work where the same was assumed.\n\n- The proposed method has several assumptions on the neural network, such as the layers need to be affine / max pooling / piecewise linear. Given the design of popular neural networks, the method is widely applicable enough. However, it will be helpful if the authors can address the commonly used network structure where the proposed method cannot be used. \n\nFor example, can one uses this method to test the alternative hypothesis that a region found by BERT's internal attention mechanism is significant? Such a case study will help readers gain more clarity of the method.\n\nAlso, if the method is not applicable to softmax layer, does the score produced from the logit? Please clarify.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fX_rIKvYEW5", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nIn this paper, the authors propose a novel approach for quantifying the statistical significance of binary masks predicted by a subclass of deep neural network (DNN) models for image segmentation problems.\n\nIn brief, the manuscript considers the particular setting where a model has been pretrained to produce a binary output of the same dimension as the input, which can be interpreted as a binary attention mask / image segmentation output. In this scenario, the main contribution is an approach to test the null hypothesis that a linear contrast of the image, depending on the segmentation outputs, equals zero. In particular, the specific formulation put forward in the paper tests the null hypothesis that the average value of pixels classified as 1 (e.g. in the mask) is equal to the average value of pixels classified as 0 (e.g. not in the mask).\n\nSince the linear contrast is a function of the input, in this case, a nonlinear function implemented by the pre-trained DNN model, classical statistical inference (such as e.g. a two-sided t-test) does not apply. Instead, the authors propose to leverage recent advances in the field of selective inference to derive the null distribution of the test statistic conditioned on the DNN’s output (plus additional constraints to get rid of nuisance parameters).\n\nIn a nutshell, to accomplish this, they:\n\n1) Assume that images follow a multivariate normal distribution with known covariance matrix,\nand\n2) Restrict the class of DNN models to those whose internal representation in every layer can be expressed as a piecewise affine function of the input. Namely, this excludes activation functions other than piecewise linear functions (e.g. ReLU), which the authors propose to approximate using piecewise linear functions.\n\nThe modelling assumption (1) together with the model constraints (2) permit the problem to be cast in a form for which the celebrated Polyhedral lemma (Lee et al. 2016) applies.\n\nFinally, the authors also tackle the algorithmic problem of efficiently integrating over the resulting truncation regions. To this end, they propose two approaches: one based on over-conditioning, inspired by Lee et al. 2016, and another based on the homotopy method, possibly inspired by Liu et al. 2018.\n\nExperimental results are provided for a synthetic toy problem with “images” that follow a multivariate normal distribution with no correlation between pixels. The ground-truth attention mask is a large square region in the center of the image, and pixels within this region have a larger mean than those outside. DNN models consisting of 2 convolutional layers are pretrained on data generated under the same distribution, with the entire ground-truth binary segmentation masks used for training for each “image”. Results suggest that, under these conditions, the proposed approach succeeds in controlling the type I error while achieving non-trivial statistical power. The supplementary material shows that type I error remains controlled for other distributions such as Laplace or Student-t with 20 DoF. \n\nFinally, results are also shown for a real-world brain image dataset, where type I error appears to be only sensibly larger than the tower and power non-trivial as well.\n\n### High-level assessment\n\nFrom a methodological perspective, the authors have identified a novel, interesting application of existing work, namely, the now seminal work of Lee et al. 2016 and recent improvements over those ideas presented by Liu et al. 2018, to the problem of image segmentation using certain DNN models. Moreover, they also proposed a sound algorithmic implementation of these ideas targeting the application at hand and provided some nascent, preliminary results that suggest the proposed approach might be of use in certain applications such as medical imaging.\n\nAll in all, this represents an original contribution of relevance to the field that I wish to see published eventually. However, I believe the current version of the manuscript falls short in certain areas, and should be improved prior to publication. Mainly:\n\n1) I have strong reservations regarding the adequacy of the modelling assumptions required by the selective inference framework for this application.\n\n2) The experimental results are substantially lacking, both in terms of breadth and depth.\n\n3) The paper is also lacking in clarity. Many key, low-level aspects such as model architecture, (pre)-training procedure or assumptions between the relation of pre-training and testing data are either not clearly stated or mentioned without proper justification or discussion of its implications and impact on the overall results. Likewise, I feel that some of the limitations of the proposed approach are understated and that its generality is overstated, especially in the abstract and the introduction.\n\nBecause of this, I am as of now leaning towards recommending rejection, as I believe the paper would greatly benefit from a strong, non-incremental revision.\n\n### Major points / suggestions for improvement\n\n1) In order to frame the problem within the assumptions of the Polyhedral lemma, the authors postulate that images in the dataset follow a multivariate normal distribution, with unknown mean but known covariance matrix.\n\nUnfortunately, while selective inference is known to abound on strong parametric assumptions, I believe this assumption is just too strong and hard to justify in the context of image data relative to, say, interpreting the coefficients of a linear model as done in more “typical” applications of selective inference.\n\nWhile it might be unrealistic to expect the authors to overcome inherent limitations of the current state of the art in selective inference, I believe they should at least carry out a much more thorough investigation of the extent to which these assumptions are applicable to real-world data, and what are the consequences of violations for the inference process.\n\nThe robustness results provided in the supplementary material are a good step in this direction, but keep many unrealistic assumptions, such as unimodality or simplistic / inexistent correlation structures that remain far from representative of natural or medical images.\n\n2) The proposed approach relies on pre-training the DNN model on a separate dataset, since the authors have made no attempt to account for the impact of training itself on the null distribution. \n\nThis can be an important limitation, especially in applications where data does not abound, which unfortunately might have a strong overlap with applications for which statistical significance might be of special relevance. Moreover, the need to split the data into “development” and “inference” sets will introduce randomness of the inference results, apart from incurring a loss in statistical power.\n\nTo this end, I believe the authors should: (1) discuss in much greater clarity how the pre-training of the DNN models interacts with their proposed approach; (2) study the extent to which the resulting inferences are robust to data splitting & retraining of the DNN model and (3) the impact that data splitting has on the resulting statistical power.\n\n3) The paper is substantially lacking in clarity, leaving the reader to second-guess many key aspects of the proposed approach.\n\nOne such example is precisely the issue of pre-training. It is never clearly stated in the main manuscript how the DNN models are (pre)-trained, and what the relation between the training and inferences datasets is exactly.\n\nOther, related aspects are (i) which type of supervision the models require for pre-training and (ii) which type of output layers are required in order for the null hypothesis to be relevant. For example, I had to look at the code in order to know that, in their simulation experiments, models are trained with the segmentation masks as supervision targets. To the best of my knowledge, this is nowhere stated in the paper. Similarly, I am uncertain of how the models for the real-world brain image dataset were trained. Given the change in architecture reported in the supplementary material, I could guess that in this case only tumor/no tumor labels were provided, but this is neither stated nor, as far as I could see, shown in the code provided. Also, in the latter case, I imagine that in order to keep the null hypothesis relevant, the model must rely on a global average pooling layer prior to classification since, otherwise, a null hypothesis defined in terms of the difference of pixel intensity values would not capture well the behavior of the output layer(s). More generally, the choice of architecture for the models is not justified in any way.\n\nAs a final example, it is also never precisely stated how exactly the output representation is thresholded in order to define the binary attention mask, how the threshold parameter is supposed to be chosen by a user and what its impact is in overall performance.\n\nAll in all, in my opinion, at the moment the manuscript does not provide sufficient information for a reader to accurately reproduce its results from the text alone.\n\n4) In my opinion, the abstract and introduction currently overstate the generality of the proposed approach. I believe that claims such as “testing the reliability of DNN representations” implicitly imply the approach can be applied to any kind of DNN model/problem. However, the target application for the framework here developed is narrower and applies only to image segmentation problems and perhaps to other strongly related settings (e.g. problems where the target representation has the same size as the input and can be interpreted as an attention mask of sorts).\n\nTo clarify, I do not intend to imply the author’s contribution is insufficient. I believe proposing an approach to quantify the statistical significance of image segmentation masks is on its own a worthwhile contribution. However, I do believe it would be preferable that the abstract and introduction were written in a way that was more specific to what the manuscript truly proposes, implements and tests.\n\n5) The experimental results, while providing some encouraging preliminary results supporting the proposed approach, in my opinion fall short in multiple aspects.\n\nFirstly, as mentioned in point 1. above, I believe the experiments on synthetic data as they stand now are barely a “sanity-check” for the model. Instead, I suggest that the authors explore ways to create more challenging synthetic datasets capturing characteristics of real-world data, perhaps (but not necessarily) by exploiting deep generative models trained on natural / imaging data. In brief, the goal should be to exhaustively characterize (1) the extent to which the modelling assumptions required by selective inference are applicable to real-world data and (2) which failure modes / limitations the proposed approach might have.\n\nSecondly, the models used by the authors are too small and not representative of those deployed in the target applications. I would encourage the authors to show results pertaining larger models (e.g. ResNet-based architectures with > 10 layers).\n\nFinally, related to the previous point, it would be of interest to explore the effect of model size & architecture (depth, number of units per layer, presence of residual connections / normalization) on the number of intervals encountered, false positive rate and statistical power.\n\n### Minor points (not related to the manuscript’s rating)\n\n1) In my opinion, Example 1 would be much more informative if it included inactive units rather than the special, much easier case where all hidden units are active.\n\n2) Personally, I do not find the definition of power in the supplementary material to be sufficiently clear. I would encourage the authors to define mathematically what # detected and # rejected are in their context.\n\n3) To the best of my knowledge, the code does not currently set the seed for the PRNG, which might be detrimental for reproducibility.\n\n4) I would recommend having the submission proof-read for English style and grammar issues.", "extra_scores": {}, "extra_text": {}}, {"review_id": "bjeh-4wNBuD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present a new method to quantify the representational strength of the neural networks. In particular, they address a known issue about how to judge a network's result relative to a hypothesis if the source of both the hypothesis and the network training was the same dataset.\nI believe the goal and motivation of this paper are clear and very well useful. Having a robust and fair theoretical framework would be a good step in analyzing a lot of vision approaches and approaches in NLP, control, and others. They also give some promising experimental results.\nThe techniques used in this work (e.g., selection of extra conditioning for traceability, Merging results of condition sets) have a satisfactory amount of novelty and can be of independent interest to the ML community. Since their approach (at least partially) addresses the issue explained above, it may find more interest among researchers of downstream tasks for explainability and reliability evaluation.\nI suggest the authors address famous problems with frequentist statistics and commonly misinterpreted p-value. I want to see the author's opinion about the position of their approach relative to such issues. Perhaps having a discussion section or limitations of the study would help. Also, does this approach have a Bayesian counterpart so that it doesn't rely on p-values?", "extra_scores": {}, "extra_text": {}}, {"review_id": "QCjELwTsYjO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a novel method which to quantify the reliability of DNN-driven hypotheses in a statistical hypothesis testing framework. Naive statistical testings are not appropriate for the DNN-driven hypotheses, where the hypotheses are selected by looking at the data(i.e. The selection bias exists). To address this problem, the authors developed a novel homotopy method under the Selective-Inference(SI) framework, which can derive the exact sampling distribution of the DNN-driven hypotheses. In this paper,  the authors mainly focus on DNNs which consist of affine operations, max-operations, and piecewise-linear activation. As described by Lee et al. (2016), the main idea of SI is to make the inference conditional on the selection event. Specifically to the DNN-driven hypotheses, the authors proposed a novel method that consists of two steps, 1) Adding extra conditioning to make the problem traceable. 2) Combining multiple over-conditioning cases by homotopy method to solve the over-conditioning problem. The experimental results on both synthetic and real-world datasets illustrate the proposed method can successfully control the FP error rate.\n\nConsidering that there are more and more interests in the research on neural representation learning, the problem that this work is trying to solve is pretty important. The SI framework has been well studied on the lasso and other problems. To the best of my knowledge, this is the first work that deploys SI to test DNN representation driven hypotheses. Although the authors only demonstrate it on simple DNNs, I can see the potentials of this method to apply on more practical and complex DNNs. The community may benefit from it on understanding neural representations.\n\nPros:\n- The problem that this paper is trying to solve, is clearly defined and is essential in understanding the representations DNN learned.\n- This proposed SI algorithm based on the homotopy method can derive the exact conditional sampling distribution of DNN-driven hypotheses in an efficient way, and it is proved to be effective in practical by experimental results. It also shows its novelty in solving the problem.\n- The authors provided comprehensive supplementary materials to help readers to understand the proposed method as well as to reproduce the experiments.\n- The structure of the paper is well designed, and the writing is clear.\n\nCons:\n- While this paper claims that the proposed method can quantify the reliability of neural network representation-driven hypotheses, there could be more examples (more realistic models on more tasks) to demonstrate the method's effectiveness in more scenarios. Currently, the examples in this paper are almost all using basic NN components on image inputs. The performances are not obvious if the networks contain parts like residual connections or recurrent structures, either the inputs are sentences rather than images.\n- Related to the previous point, the boundary between where this method can be applied and can not be applied is not so clear. It would be better if the authors could give such guidance for people to use this method.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four reviewers evaluated your work and provided a detailed review with many suggestions. I also think that there is an interesting idea and encouraging results but there is a lack of numerical results and still some parts are still unclear and need to be polished. Consequently in its  current form, the paper can not be accepted for publication. I would advise you to carefully follow the remarks of reviewer 1 to improve  the paper. 


Decision: Reject