Valid P-Value for Deep Learning-driven Salient Region
Authors: Miwa Daiki
Vo Nguyen Le Duy
Ichiro Takeuchi
Venue: ICLR 2023 poster (2023)
Keywords: Saliency Map
Attention
Selective Inference
Uncertainty Quantification
P-value
Statistical Hypothesis Testing

Abstract: Various saliency map methods have been proposed to interpret and explain predictions of deep learning models. Saliency maps allow us to interpret which parts of the input signals have a strong influence on the prediction results. However, since a saliency map is obtained by complex computations in deep learning models, it is often difficult to know how reliable the saliency map itself is. In this study, we propose a method to quantify the reliability of a saliency region in the form of p-values. Our idea is to consider a saliency map as a selected hypothesis by the trained deep learning model and employ the selective inference framework. The proposed method provably provides a valid p-value for the detected salient region, i.e., we can provably control the false positive rate of the detected salient region. We demonstrate the validity of the proposed method through numerical examples in synthetic and real datasets. Furthermore, we develop a Keras-based framework for conducting the proposed selective inference for a wide class of CNNs without additional implementation cost.

Published as a conference paper at ICLR 2023
VALID P-VALUE FOR SALIENT REGION
DEEP LEARNING-DRIVEN
Daiki Miwa"
Vo Nguyen Le Duy RIKEN
Nagoya Institute of Technology mi wa daiki ml 1 ab. ni t.@ i com
v voer rikn..... ip
Ichiro Takeuchi
Nagoya University and RIKEN
ichiro. takeuchi émae. nagoya-u.ac.
ABSTRACT
Various saliency map methods have been proposed to interpret and explain pre- dictions of deep learning models. Saliency maps allow to interpret which parts of the input signals have a strong influence on the prediction results. However, since saliency map obtained by complex computations deep learning mod- els, it is often difficult to know how reliable the saliency map itself is. In this study, we propose method to quantify the reliability of a salient region the form of p-values. Our idea is to consider salient region as selected hypothesis by the trained deep learning model and employ the selective inference framework. The proposed method can provably control the probability of false positive de- tections of salient regions. We demonstrate the validity of the proposed method through numerical examples in synthetic and real datasets. Furthermore, we de- velop a Keras-based framework for conducting the proposed selective inference for wide class of CNNs without additional implementation cost.
INTRODUCTION
Deep neural networks (DNNs) have exhibited remarkable predictive performance in numerous prac- tical applications in various domains owing to their ability to automatically discover the representa- tions needed for prediction tasks from the provided data. To ensure that the decision-makinn process of DNNs is transparent and easy to understand, it is crucial to effectively explain and interpret DNN representations. For example, in image classification tasks, obtaining salient regions allows us to explain which parts of the input image strongly influence the classification results.
Several saliency map methods have been proposed to explain and interpret the predictions of DNN models (Ribeiro al., 2016 Bach et al. 2015) Doshi- Velez & Kim 2017: Lundberg & Lee 2017 Zhou et al.) [016] Selvaraju et al. 2017p. However, the results obtained from saliency methods fragile [Kindermans et al.) 2017| Ghorbani al. 2019 Melis & Jaakkola 2018 Zhang al. 2020 Dombrowski et al. 2019 Heo et al. 2019). Therefore, it is important to develop a method for quantifying the reliability of DNN-driven salient regions.
Our idea is to interpret salient regions as hypotheses driven by a trained DNN model and employ a statistical hypothesis testing framework. We use the p-value as a criterion quantify the statistical reliability of the DNN-driven hypotheses. Unfortunately, constructing a valid statistical test for ddiver salient regions challenging because of the selection bias. In other words, because the trained DNN selects the salient region based on the provided data, the post-selection assessment of importance is biased upwards.
To correct the selection bias and compute valid p-values for DNN-driven salient regions, we intro- duce a conditional selective inference (SI) approach. The selection bias is corrected by conditional
Equual contribution
Corresponding author

---

Published as a conference paper at ICLR 2023
Input Image
Saliency Map
Salient Region
Reference Image
Two-sample Test
Image without tumor region. The naive-p= = 0.00 (wrong detection) and selective-p- = 0.43 (true negative)
0.00
Input Image
Saliency Map
Salient Region
Image
Two-sample Test
(b) Image with tumor region. The naive-p 0.00 (true positive) and selective-p 0.00 (true positive)
Figure I: Examples of the problem setup and the proposed method on the brain tumor dataset. By applying saliency method called CAM (Zou al. 2016) on a query input image, we obtain the salient region. Our goal is to provide the statistical significance of the salient region in the form of p-value by considering two-sample test between the salient region and the corresponding region in the reference image. Note that, since the salient region is selected based on the data, the degree of saliency in the selected region is biased upward. In the upper image where there is no true brain tumor, the naive p-value which is obtained without caring about the selection bias is nearly zero, indicating the false positive finding of the salient region. On the other hand, the selective p-value which is obtained by the proposed conditional SI approach is 0.43, indicating that the selected saliency region is not statistically significant. In the lower image where there is a true brain tumor, both the naive p-value and the selective p-value are very small, which indicate These results illustrate that naive p-value cannot be used to quantify the reliability of DNN-based salient region. In contrast, with the selective p-values, we can successfully identify false positive and true positive detections with a desired error rate.
positive finding.
SI in which the test statistic conditional on the event that the hypotheses (salient regions) are selected using the trained DNNs considered. Our main technical contribution develop method for explicitly deriving the exact (non- mnon-asymptotic) conditional sampling distribution of the salient region for a wide class convolutional neural networks (CNNs), which enables us to conduct conditional SI and compute valid p-values. Figure presents an example of the problem setup.
Related works. In this study, we focus on statistical hypothesis testing for post-hoc analysis, i.e.. quantifying the statistical significance of the salient regions identified in trained DNN model when a test input instance fed into the model. Several methods have been developed to visualize and understand trained DNNs. Many of these post-hoc approaches (Mahendran & Vedaldi 2015 Zeiler
& Fergus 2014 Dosovitskiy & Brox 2016, [Simonyan et al. 2013) have focused on developing visualization tools for saliency maps given trained DNN. Other methods have aimed to identify
the discriminative regions in an input image given a trained network (Selvaraju et al.) 2017 Fong &
Vedaldi 2017 Zhou al. 2016 Lundberg & Lee 2017). However, some recent studies have shown that many of these saliency methods of these saliency methods are not stable against a perturbation
or adversarial attack on the input data and model (Kindermans al. 2017 Ghorbani et al. 2019
Melis & Jaakkola 2018 Zhang et al.] 2020 Dombrowski et al. 2019 Heo et al.) 2019) To the best of our knowledge, no study to date has succeeded quantitatively evaluating the reproducibility of
DNN-driven salient regions with rigorous statistical inference framework.
In recent years, conditional has emerged as a promising approach for evaluating the statistical re- liability of data-driven hypotheses. It has been actively studied for making inferences the features of linear models selected by various feature selection methods, such as Lasso (Lee et al. 2016). The main concept behind conditional SI is to make inference based on the sampling distribution of the test statistic conditional on a selection event. This approach allows us to derive the exact sampling distribution of the test statistic. After the seminal work of Lee al. (016)) conditional SI has also
2

---

Published as a conference paper at ICLR 2023
been applied to wide range of problems (Lottus
2015
Choi et al., 2017 Tian & Taylor 2018
Yang et al. 2016; Tibshirani et al.
2016; Fithian et al. 2014 Loftus & Taylor
2014
Panigrahi
et al.) 2016; Sugiyama al. 2021a a Hyun et al., [221]]
Duy & Takeuchi 202 la
Sugiyama al.
2021b Chen & Bien 2019 sukurimichi et al.
2021 Tanizaki et al. 2020 Duy et al.
2020 2022
The most relevant existing work is Duy al. (0022) where the authors provide framework to compute valid p-values for DNN-based image segmentation results. In Duy et al. (2022). the authors only considered the inference on the output of DNN in a segmentation task. In this paper, we address a more general problem in which the hypotheses characterized by any internal nodes of the DNN can be considered. This enables us quantify the statistical significance of salient regions. Furthermore, we introduce a deras-based implementation framework that enables us to conduct SI for a wide class of CNNs without additional implementation costs. This is in contrast to Duy et al. 2022 in which the selection event must be implemented when the network architecture is changed.
In another direction, et al. (2020) considered the black box model interpretability a multiple hypothesis testing problem. Their goal was to identify important features by testing the significance of the difference between the prediction and the expected outcome when certain features are re- placed with their counterfactuals. However, this approach faces a significant challenge: the number of hypotheses to be considered can be very large (e.g.. in the case of an image with n pixels, the number of possible salient regions is 2"). Multiple testing correction methods, such the Bon- ferroni correction, are highly conservative when the number of hypotheses is large. To address the challenge, they only considered tractable number of regions selected by a human expert object detector, which causes selection bias because the candidate regions are selected based on the data.
Contribution. Our main contributions are as follows:
We provide an exact (non-asymptotic) inference method for salient regions obtained by CAM based the SI concept. We introduce valid p-values to statistically quantify the reliability of the DNN-driven salient regions inspired by Duy et al. 2022).
We propose a novel algorithm and its implementation. Specifically, we propose Keras-based im- plementation which enables us to conduct conditional SI for a wide class of CNNs without additional implementation costs.
We conducted experiments on synthetic and real-world datasets, through which we show that our proposed method can control the false positive rate, has good performance in terms of computational efficiency, and provides good results in practical applications. Our code is available at
http://gtthbbccoo///cccuco//lllt inference dan salient region,
PROBLEM FORMULATION
In this paper, we consider the problem of quantifying the statistical significance of the salient regions identified by trained DNN model when test input instance is fed into the model. Consider an n--imennional query input vector
X
= E e~ C((,00=)
and an n-dimensional reference input vector,
X (Xiee Xxel) = stel + rel ~ N(0,00
where ree E IRt are the signals and E, re Th are the noises for query and reference input vectors, respectively. We assume that the signals, s and re are unknown, whereas the distribution of noises and gre are known (or can be estimated from external independent data) to follow N((0, Ln),, an n--dimensional normal distribution with a mean vector and covariance matrix o² Ins which mutually independent. In the illustrative example presented in $1, X is query brain image for a potential patient (we do not know whether she/he has a brain tumor), whereas re is a brain image of healthy person without brain tumors.
Consider a saliency method for a trained CNN. We denote the saliency method as a function A R R.Dh that takes query input vector X R and returns the saliency map A(X) E IR. We
3

---

Published as a conference paper at ICLR 2023
define salient region Mx for the query input vector X as the set of elements whose saliency map value is greater than a threshold
Mx E [r]] A 2 z
where 7 E R denotes the given threshold. In this study, consider CAM Zhou et al. 2016 as an example of saliency method and threshold hrrsshodd-based definition of the salient region. Our method can be applied to other saliency methods and other definitions of salient region.
Statistical inference. To quantify the statistical significance of the saliency region MMx, we con- sider nwo-sample test for the difference between the salient regions of the query input vector X MM and corresponding region of the reference input vector X ref .Mx where X MMx is a sub-vector of X indexed by X. As examples of the wwo-sample test, we consider the mean null test:
Ho
Σ iEMx
ref
V.S. HI
Σ MEMx
ref $
(2)
Mx
Mx
Mx
Mx
iEMx
sEMx
and global null test:
Ho : si MiEMx: v.s. :1: ooo Bi Mx.
(3)
In the mean null test in Eq. region MM are the same between X and X contrast, in the global null test in Eq. B we consider null hypothesis that all elements of the signals in the salient region Mx are the same between X and X The p-values for these two-sample tests can be used to quantify the statistical significance of the salient region Mx.
consider a null hypothesis that the average signals in the salient
Test-statistic. For a two-sample test conducted between XMx and X Mxx we consider a class of test statistics called conditionally linear test-statistic, which is expressed as
X
T(X, T(X,""f X -mxx X
(4)
and conditionally test-statistic, which is expressed as
T(X, X re - -o--0-1 -0-1 -0-1||
(5)
r
where 7Mx E R22n is vector and PMx E 2R is projection matrix that depends on Mx- The test statistics for the mean null tests and the global null test can be written in the form Eqs. and 5) respectively. For the mean null test in Eq. ② we consider the following test-statistic
-Tx
X xo
1
Σx, tEMx
1
Σ iEMx
X
[Mx
Mx
12x in
where 7Mx = Mxx
E R?" with 1 being the n-dimensional vector whose elements belongs the set are set to 1, and O otherwise. For the global null test in Eq. consider the following test-statistic
x = 0- PMx
X
)11
Σ iEMx
X;
X,eef 2
20
diag(1 1m)
-diag(11 (1 11'xx)
where PMx =
To obtain p-values for these two-sample tests
-idaag( 1 (1ag(11M))
we need know the sampling distribution the ttst-statistics. Unfortunately, it is challenging to derive the sampling distributions of test-statistics because they depend on the salient region Mx, which is obtained through complicated calculation in the trained CNN.
COMPUTING VALID VAVUL BY CONDITIONAL SELECTIVE INFERENCE
In this section, we introduce approach compute the valid valuues for the two-sample tests for the salient region Mx between the query input vector and the reference input vector ref based on the concept of conditional SI (Lee al. 2016
4

---

Published as a conference paper at ICLR 2023
3.1 CONDITIONAL DISTRIBUTION AND SELECTIVE p- P-VALUE
Conditional distribution. The basic idea of conditional SI is to consider the sampling distribution of the test-statistic conditional on a selection event. Specifically, we consider the sampling property
of the following conditional distribution
x(X,X') (Mx = -Mxmn
(6)
where Xobs is the observation (realization) of random vector X. The condition in Eq. indicates the randomness of X conditional on the event that the same salient region Mx as the observed M is obtained. By conditioning on the salient region Mx. derivation of the sampling distri- bution of the conditionally linear and X test-statistic T(X, X is reduced to derivation of the
distribution of linear function and quadratic function of (X, X respectively.
Selective p-value. After considering the conditional sampling distribution in (6). we introduce the following selective p-value:
PHo (I((X 2 (X ooss, x
Mx M @x.x 2obs)) (7)
Pselective
where
2x,xaa = Xx,
Qobs = 2 2xomx obx
with Qx xxx
Tzn
nx nig 11 11
R22 in the case of mean null test, and (xirt)
X
2x. = (Vx,xint, Ux Xx. Qobs 2x
Xtaf abo
x with Vx, xtt PMx (x:ee) / PMM (x
R2" Ux,x
PMx
2 in the
case of global null test. The 2x,x is the sufficient statistic of the nuisance parameter that needs to be conditioned on in order to tractably conduct the inference
|
The selective p-luuue in Eq. has the following desired sampling property
P Ho Pselective I a Mx Mx a, Ya [0, 1].
This means that the selective p-values Prelective can be used as valid statistical significance measure for the salient region Mx.
3.2 CHARACTERIZATION THE CONDITIONAL DATA SPACE
To compute the selective p-value in we need to characterize the conditional data space whose characterization is described and introduced in the next section. We define the of (X X T E R 2t that satisfies the conditions in Eq. 7 as
D D= f
x xref TT
E 2 [Mx = M 2x.x" = 2obs
(9)
According to the second condition, the data in D is restricted to line in 2 as stated in the following Lemma.
Lemma 1. Let us define and a = Ux....XIr and b Vx VXemxX in the case of global null test. The D in can be
St X.boxxiit and b Tamx!! 1!? E R2nn in the case of mean null test,
TMx
rewritten as (XT x I aabb 2} by using the scalar parameter z E R. where
= E Ma M..w
(10)
with 21.m representing a vector of elements 1 through n xf. x.
'This nuisance parameter Qx eet corresponds to the component in the seminal conditional SI paper in (Cen Bien 2019) Sec. 3, Theorem @x.x==t x is standard approach in the conditional SI literature
Lee al. 2016) (see Sec. 5, Eq. 5.2 and Theorem 5.2) and 3.7). We note that additional conditioning on Qx
and used almost all conditional S1-related studies. Here, we would like to note that the selective p-value depends on @x.x'=t xwwf, but the property in 8 is satisfied without this additional condition because we can marginalize over all values of Qx X (see the lower part of the proof of Theorem 5.2 in Lee al. 2016 and the proof of Theorem 3.7 Chen & Bien (019
5

---

Published as a conference paper at ICLR 2023
Proof. The proof is deferred to Appendix A.1
Lemma D indicates that do not need to consider the 2n-dimensional data space. Instead, only need to consider the one-dimensional projected data space Z in (TO. Now, let us consider
random variable Z. E R and its observation Zohs E R that satisfies (X X )T =a + bZ and
(X X ) + bZobs. The selective p-value is rewritten
Psolective
=PHo (1 Zobs 7 2).
2
(11)
Because (XT X T N TT steeT) T
T2n
due to the independence between X and
X the variable Z ~ N(0, o² [ the case of mean null test and Z ~ X (Trace(P). P)) in the case of global null test under the null hypothesis. Therefore, Z Z E Z follows truncated normal distribution and a truncated X distribution, respectively. Once the truncation region Z identified, computation of the selective p-value in 11 is straightforward. Therefore, the remaining task is to identify Z.
4 PIECEWISE LINEAR NETWORK
The problem of computing selective p-values for the selected salient region is cast into the problem of identifying of intervals Z {z E R Mx(=) MM J. Given the complexity of saliency computation in a trained DNN, it seems difficult to obtain 2. In this section, however, we show that this is feasible for a wide class of CNNs.
Piecewise linear components in CNN. The key idea is to note that most of basic operations and common activation functions used in a trained CNN can be represented as piecewise linear functions in the following form:
Definition 1. (Piecewise Linear Function) A piecewise linear function f R - I is written as:
V/4 X 1b/. V x + 2b
if X Pf- (X' ER" if x P == (X' ER"
A X' < 6/1 }
A < 8/1,
f(X)
w W k(n X + pkun
if X Pk == (X' X' S k)
where W/, 4/. 2/ and 6% for E [K()) are certain matrices and vectors with appropriate
dimensions, P/ == fx E R" 8/2 < 6%. is polytope in R" for k e [K()), and K(f) is the
number of polytopes for the function
Examples of piecewise linear components in trained CNN are shown in Appendix A.2
Piecewise Linear Network.
Definition 2. (Piecewise Linear Network) network obtained by concatenations and compositions of piecewise linear functions is called piecewise linear network.
Since the concatenation and the composition of piecewise linear functions is clearly piecewise linear function, the output of any node in the piecewise linear network written as piecewise linear function of an input vector X. This is also true for the saliency map function A: E [n] obtained by CAM. Furthermore, as discussed in $4, we can focus on the input vector in the form of X X() ain + bi..2 which is parametrized by scalar parameter E IR. Therefore, the saliency map value for each element is written as a piecewise linear function of the scalar parameter z, i.c.,
A.z z + p A di
if z E L UA1 if = E LL UUA,)
62 = + P2
Ax(((2))
(12)
<K(AA) + PIRCA
U. if 2 LRLA URiAnl

---

Published as a conference paper at ICLR 2023
Algorithm 1 1SI.DNN
Input: Xbbe, -.min: Smase.
1: Obiain Sahe-
Cobr-Compute compute 17 as well as and + Lemma D and initialize:
1, Rx. = Amnt
2: for 1 S do
3: Compute Z441 by Auto Conditioning (see
4: ire (.)) xxet(s)
Eoba in [++1] 1 (by using Eq
then
5:
end T T (0)
6:
7: (=1+1
8: end
9: Identily Z + U1e7
10: Psclective +- Eq.
Output: Puclective
where ((A;;) is the number of linear pieces of the piecewise linear function, AAA A are certain
scalar parameters, [ U are intervals for k E [K(): (note that a polytope in R is reduced to an interval when it projected onto one-dimensional space).
This means that, for each piece of the piecewise linear function, we can identify the interval of such that (; (z)) 2 as follows
max (LZ" (r (T- pk /xtht) Un! I. min (UEL (r- pz /
ZE {
if i if k 0
=E{
Ax(((z)) 2 T. (13)
With slight abuse of notation, let collectively denote the finite number of intervals on E IR that are defined by L As
U (T 10--p- p² /N for all i) E [K()) x [n] as
[=0 21), (21, 1 =c], [Ze, =++1), [,
where Zmin 20 and max 2r are defined such that the probability mass of a < Zmin and 2 > Emax are negligibly small.
Algorithm. Algorithm 1 shows how identify 2 {= RR Mx Moba)- We simply check the intervals of in the order of [20, 21 1. [21, 22]. [2T [2T-1, 2T] to whether Mx =Mx(= or not in the interval by using Eq. 13.. Then, the truncation region Z in Eq. is given as = xeee
for xE(2, [2e, 2t+1]. the literature of homo-
topy method (a.k.a. parametric programming). is known that the actual computational cost differs significantly from the worst case. A well-known application of the homotopy method in the ML community is the Lasso regularization path, which also has the worst-case computational cost on the exponential order of the number of features, but the actual cost is known to be nearly linear order. Empirically, this also applies to our proposed method.
IMPLEMENTATION: AUTO-CONDITIONIG
The bottleneck of our algorithm Line Algorithm 1 where 24+1 must be found by considering all relevant piecewise linear components in complicated trained CNN. The difficulty lies not only in the computational cost but also the implementation cost. To implement conditional SI in DNNs naively, it is necessary to characterize all operations at each layer of the network as selection events and implement each of them specifically (Duy et al., 2022). To circumvent this difficulty, introduce modular implementation scheme called auto- contiriondiinnnng which is similar auto- differentiation (Baydin et al. 2018) in concept. This enables us to conduct conditional SI for a wide class of CNNs without additional implementation costs.
The basic idea in auto-conditioning to add a mechanism to compute and maintain the interval E [L U for each piecewise linear component f in the network (e.g., layer API in the Keras
FFor simplicity, we omit description for case x In this case, Pk 2 т. then 2 [L U.A1) + Mx

---

Published as a conference paper at ICLR 2023
framework). This enables automatically compute the interval [Ly Ug11 of piecewise linear function f when is obtained as concatenation and/or composition of multiple piecewise linear components. If f is obtained by concatenating two piecewise linear functions fi and fz. we can easily obtain [L U = L (iAnnLE, U/e) However, if is obtained as a composition of two piecewise linear functions fi and Jz. the calculation of the interval is given by the following lemma.
Lemma 2. Consider the composition of two piecewise linear functions f(X (z)) (f2o fi (1)(()))))) (X (2)). Given a real value of z, the interval | = the input domain of fa can be computed as
(8%))s (4/2
(8%) - (4
12
U
max (412)44),00
min 3
(1
(A K2 (5)))
where Bhr + + is the output of f1 (i.e., the input of f2). Moreover, A and 6/2 are obtained
by verifying the value of Bh + -/12. Then, the interval of the composite function is obtained as follows: [ U = [ 1 n [L U/2) [
The proof is provided in Appendix puted through layers as
A.3 Here, the variables B and yth can be recursively com-
G f 3 + wbt and 7 Vi
Lemma 2 indicates that the intervals in which X()) falls in can be forwardly propagated through these layers. This means that the lower bound LA and upper bound U/A, of the current piece in the piecewise linear function in Eq. (12 can be automatically computed by forward propagation of the intervals of the relevant piecewise linear components.
6 EXPERIMENT
We only highlight the main results. More details (methods for comparison, network structure, etc.) can be found in the Appendix A.4
Experimental setup. We compared our proposed method with the naive method, over-conditioning (OC) method, and Bonferroni correction. To investigate the false positive rate (FPR), we considered 1000 null images X (X1, Xn) and 1000 reference images X (Xref ...... r.fd), ), where rf and E, rof N(0. In). for each E (64, 256, 1024, 4096). To investigate the true positive rate (TPR), we set n 256 and generated 1,000 images, in which s; = A for any E S, where is the "true" salient region whose location randomly determined, and 8; = 0 for any i & S. We set A E {1,2, 2, 3, 4). Reference images were generated in the same way as in the case of FPR. In all experiments, we set t = in the mean null test and 7 = 5 in the global null test. We set the significance level
0.05. We used CAM the saliency method in all experiments.
Numerical results. The results of FPR control properties are presented in Fig. 2 method, OC, and Bonferroni successfully controlled the FPR in both the mean and global null test cases, whereas the naive method could not. Because naive method failed to control the FPR, we no longer considered its TPR. The results of the TPR comparison method has the highest TPR in all cases. The Bonferroni method has the lowest TPR because is conservative owing considering the number of all possible hypotheses. The OC method also has a low TPR because it considers several extra conditions, which cause the loss of TPR.
The proposed
shown in Fig.
The proposed
Real data experiments. We examined the brain image dataset extracted from the dataset used in Buda et al. (2019), which included 939 and 941 images with and without tumors, respectively. The results of mean null test are presented in Figs. and The results of the global null test are presented in Figs. ( and 7 The naive p-value remains small even when the image has no tumor region, which indicates that naive p-values cannot be used to quantify the reliability of DNN-based salient regions. The proposed method successfully identified false and true positive detections.
7 CONCLUSION
In this study, we proposed a novel method to conduct statistical inference on the significance of DNN-driven salient regions based the concept of conditional SI. We provided novel algorithm
8

---

Published as a conference paper at ICLR 2023
Proposed
Pippated
0:30
6C
6.
Berferio
Boviencei
0.25
6.
0:20
Prspcoed
0.5
24
256
1224
+000
24
1324
4006
(a) Mcan null test
(b) Global null test
(a) Mcan null test
(b) Global null test
Figure 2: False Positive Rate (FPR) comparison.
Figure True Positive Rate (FPR) comparison.
Input Image
Saliency Map
Salient Region
Reference Region
Reference Image
Figure 4: Mean null test for image without tumor (puaive 0.00, Pselective
0.78).
Input Image
Saliency Map
Salient Region
Reference Region
Reference Image
Figure 5: Mean null test for image with tumor (puaive 0.00, Pselective 1.92 10 10-4).
Input Image
Saliency Map
Salient Region
Reference Region
Reference Image
Figure 6: Global null test for image without tumor (Pnaive 0.03, Pselective 0.46)
Input Image
Saliency Map
Region
Reference Region
Reference Image
Figure 7: Global null test for image with a tumor (Pnaive 0.00, Pselective 1.51 x 10-3)
for efficiently and flexibly conducting conditional for salient regions. We conducted experiments on both synthetic and real-world datasets to demonstrate the performance of the proposed method. In current setting, we have not considered the situations where there a misalignment between the input image and the reference image. A potential future improvement could be additionally per- forming a step to automatically find an appropriate region in the reference image before conducting be represented
statistical test. If the matching operations can be easily incorporated to the proposed method. If the matching operations can be represented as of linear inequalities, they be easily incorporated to the proposed method.
set of linear inequalities, they

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work was partially supported by MEXT KAKENHI (20H00601), JST CREST (JPMJCR21D3), JST Moonshot R&D (JPMJMS2033-055) JST AIP Acceleration Research (JPMICR2IU2), NEDO (JPNP18002, JPNP20006), and RIKEN Center for Advanced Intelligence Project.
REFERENCES
Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Robur-Robett Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7)=e0130 2015.
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1 2018
Mateusz Buda, Ashirbani Saha, and Macicj A Mazurowski. Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by deep learning algorithm. Computers in biology and medicine, 109:218-22 225,,222,2000 2019.
Collin Burns, Jesse Thomason, and Wesley Tansey. Interpreting black box models via hypothesis testing. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, pp- 47-57, 2020.
Shuxiao Chen and Jacob Bien. Valid inference corrected for outlier removal. Journal of Computa- fional and Gruphical Statistics, pp. 2019.
Yunjin Choi, Jonathan Taylor, and Robert Tibshirani. Selecting the number of principal components: Estimation of the true rank of noisy matrix. The Annals of Statistics, 45(6):2590-26177,, 2017.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. In Advances Neural Information Processing Systems, pp. 13589-13600, 2019.
Finale Doshi Velez and Been Kim. Towards rigorous science arXiv preprint arXiv: 7002 706.08 2017.
of interpretable machine learning.
Alexey Dosovitskiy and Thomas Brox. Inverting visual representations with convolutional net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1829-48311 2016.
Vo Nguyen Le Duy and Ichiro Takeuchi. More powerful conditional selective inference for gener- alized lasso by parametric programming. arXiv preprint arXiv:2105.04922 2021a
Vo Nguyen Le Duy and Ichiro Takeuchi. Parametric programming approach for more powerful and general lasso selective inference. In International Conference on Artificial Intelligence and Statistics, pp. 901-909. PMLR, 2021b.
Vo Nguyen Le Duy, Hiroki Toda, Ryota Sugiyama, and Ichiro Takeuchi. Computing valid p-value for optimal changepoint by selective inference using dynamic programming. In Advances in Neural Information Processing Systems, pp. 11356-1 1367, 2020.
Vo Nguyen Le Duy, Shogo Iwazaki, and Ichiro Takeuchi. Quantifying statistical significance of neural network-based image segmentation by selective inference. Advances Neural Information Processing Systems, 2022.
William Fithian, Dennis Sun, and Jonathan Taylor. Optimal inference after model selection. arXiv preprint 1100v21410 2014.
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3429-3437, 2017.
10

---

Published as a conference paper at ICLR 2023
Amirata Ghorbani, Abubakar Abid, and James Zou. Interpretation of neural networks is fragile. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3681-3688, 2019.
Juyeon Heo, Sunghwan Joo, and Taesup Moon. Fooling neural network interpretations via adversar- ial model manipulation. In Advances in Neural Information Processing Systems, pp. 2925-2936 2019.
Sangwon Hyun, Kevin Z Lin, Max G'Sell, and Ryan J Tibshirani. Post-selection inference for changepoint detection algorithms with application copy number variation data. Biometrics, 77 (3):1037-1049, 2021.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo. Maximilian Alber, Kristof Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. arXiv preprint 1rXiv:/111.0086/ 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL
https://arxiv. wttp//o/////////o//o 6980
Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference, with application to the lasso. The Annals of Statistics, 44(3):907-92 2016.
Joshua R Loftus. Selective inference after vrass-validation. arXiv preprint arXiv: 1111111111...1111 2015.
Joshua R Loftus and Jonathan Taylor. significance test for forward stepwise model selection.
arXiv preprint arXiv: 2014.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Advances in neural information processing systems, pp. 4765-4774, 2017.
Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference computer vision and pattern recognition, pp-
5188-5196, 2015.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. In Advances in Neural Information Processing Systems, pp. 7775-7784, 2018.
Snigdha Panigrahi. Jonathan Taylor, and Asaf Weinstein. Bayesian post-selection inference in the linear model. arXiv preprint arXiv: 08824, 28, 2016.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networsss Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Kazuya Sugiyama, Vo Nguyen Le Duy, and Ichiro Takeuchi. More powerful and general selective inference for stepwise feature selection using the homotopy continuation approach. In Proceed- ings of the 38th International Conference on Machine Learning, 2021a.
Ryota Sugiyama, Hiroki Toda, Vo Nguyen Le Duy, Yu Inatsu, and Ichiro Takeuchi. Valid and exact statistical inference for multi-dimensional multiple change-points by selective inference. arXiv preprint arXiv:2 10.08989, 2021
Kosuke Tanizaki. Noriaki Hashimoto, Yu Inatsu, Hidekata Hontani, and Ichiro Takeuchi. Computing valid p-values for image segmentation by selective inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9553-9562, 2020.
Xiaoying Tian and Jonathan Taylor. Selective inference with a randomized response. The Annals of Statisties, 46(2):679-710, 2018.
11

---

Published as a conference paper at ICLR 2023
Ryan J Tibshirani, Jonathan Taylor, Richard Lockhart, and Robert Tibshirani. Exact post-selection inference for sequential regression procedures. Journal of the American Statistical Association, 1116514);600-620, 2016.
Toshiaki Tsukurimichi, Yu Inatsu, Vo Nguyen Le Duy, and Ichiro Takeuchi. Conditional selective inference for robust regression and outlier detection using piecewise-linear homotopy continua- tion. arXiv preprint arXiv:2104.00840, 2021.
Fan Yang, Rina Foygel Barber, Prateek Jain, and John Lafferty. Selective inference for group-sparse linear models. In Advances in Neural Information Processing Systems, pp. 2469-2477, 2016.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. $18-833. Springer, 2014.
Xinyang Zhang, Ningfei Wang, Hua Shen, Shouling Ji, Xiapu Luo, and Ting Wang. Interpretable deep learning under fire. In 29th {USENIX) Security Symposium ({USENIX) Security 20), 2020.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2921-2929, 2016.
12


Reviews:
[{"review_id": "u3q4Z86obC", "reviewer": "Reviewer_Duyc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work attempts to shed some light on the reliability of saliency maps.\nSpecifically, the authors propose a conditional Selective Inference (SI) method. The proposed method can provide valid p-values for statistically quantifying the reliability of the salient regions. The proposed method is evaluated on a synthetic dataset and a brain image dataset.", "summary_of_the_review": "Overall, I think the proposed conditional Selective Inference method is interesting and insightful. However, my main concern lies in the empirical evidence. It would be better to see more experimental results on other datasets, e.g., a natural image dataset, like CIFAR.", "strength_and_weaknesses": "Strengths:\n\n1. This work looks technically sound.\n\n2. The proposed method is correlated to statistical hypothesis tests. This idea is interesting and could be practically useful in some real-world scenarios, e.g., medical applications.\n\nWeaknesses:\n\n1. In this work, the saliency map is presumed to be generated by CAM (Zhou et al., 2016). We don't know how well the proposed method works with other CAM-like saliency map methods (e.g., Grad-Cam). Correspondingly, in the theory part, how the saliency map model A(\\cdot) affects the proposed selective p-value is unclear.\n\n2. How the proposed selective p-value corresponds to the correlation between the salient region and the reference region is unknown. In other words, can the selective p-value be aligned with the visual difference/similarity between salient regions and reference regions? One way to measure the linear correlation coefficient between a model saliency map and an empirical saliency map is to use correlation coefficient (CC). Please refer to \\url{https://saliency.tuebingen.ai/results.html} for more details.  \n\n3. Last but not least, the experiments are less convincing as the proposed method is only evaluated on a synthetic dataset and a small-scale dataset. I cannot tell if the proposed method can work well on other general yet complicated real-world datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "r1snc1EVtYm", "reviewer": "Reviewer_XKd9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on saliency map of class activation. An inference method is proposed to quantify the reliability of a saliency region in the form of p-values based on the concept of SI. The experimental results show the improved performance.", "summary_of_the_review": "Considering the novelty and the rigorousness of the paper, I hold a slightly negative stance.", "strength_and_weaknesses": "Strength:\n\n1. The writing flow of the paper is very nice, including comprehensive background and detailed related work. The motivation is clearly stated.\n2. The theoretical results are well presented. The assumptions made for generating such results are clearly described and contextualized.\n3. Experimental results consider several reasonable baselines and demonstrate the improvement produced by the method.\n\nWeaknesses:\n\n1. The work relies largely on the previous work to prove an exact inference method for the salient regions. Specifically, relying on similar assumptions and derivation of [1], the paper extends image segmentation to saliency maps. The mathematical formulas, the two-sample test, the conditional distribution, the selective p-value, and the piecewise linear network are all similar to [1]. It looks like an application based on [1]. Thus, it seems that the algorithm itself is not very novel.\n\n2. Is it possible to provide some experimental evidence for the argument that the proposed selective inference works well for a wide class of CNNs without additional implementation cost? How does the method integrate with various CNNs? For this issue, the authors may report the performance of different CNNs. In addition, the proposed method seems computationally expensive. How efficient is the algorithm compared with the baseline methods?\n\n[1] Vo Nguyen Le Duy, Shogo Iwazaki,and Ichiro Takeuchi. Quantifying statistical significance of neural network-based image segmentation by selective inference. NeurIPS, 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "efmXJSRI-f", "reviewer": "Reviewer_R4SS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a method to quantify the reliability of the saliency region in the form of p-values by employing the selective inference (SI) framework. In the proposed method, the authors regard the salient region proposed by DNN explainability methods such as CAM as a selected hypothesis. Since the hypothesis proposed by the model is not independent of the data, instead of classical inference with selection bias, the authors proposed to use selective inference where the inference is conditioned on a selected event. The authors also demonstrated the validity of the method through numerical and real datasets. \n\nThe main contributions of this work:\n\n1. The authors employed the selective inference framework to compute a p-value to quantify the reliability of a salient region proposed by a saliency model to overcome the selective bias problem. \n\n2. The authors also claimed to have provided a Keras-based implementation to efficiently and flexibly conduct conditional SI for salient regions for a wide class of CNNs.\n\n3. The authors conducted experiments on both synthetic and brain image datasets and claimed the proposed method is able to control the false positive rate (FPR) and has potential to be applied to practical applications. \n", "summary_of_the_review": "This paper proposed to evaluate the reliability of a salient region with a p-value by employing the selective inference (SI) framework to address the selection bias problem with classical inference. The authors also claimed to have provided an efficient Keras-based implementation of the proposed method, and experiments have conducted to prove the validity of the proposed method. In my opinion, this work is well motivated, the solution is clearly explained, and the claims are also supported by empirical results. I suggest the authors to perform more experiments on natural images with additional analysis for deeper insights. ", "strength_and_weaknesses": "Strength:\n\n1. The paper in general clearly explained the motivation of the work by identifying the problem with existing statistical inference framework, i.e. selection bias. \n\n2. The authors employed the selective inference framework to address the issue of selection bias in the context of saliency region detection and reliability estimation.\n\n3. The authors claimed that the proposed method is applicable to a wide class of CNNs, demonstrated by an analysis of piecewise linear network. \n\n4. The authors claimed to have implemented the proposed method in an efficient way and provided the code for reproducibility.\n\n5. The experiments provided also supported the claims made by the authors regarding the validity and applicability of the proposed method on practical problems. \n\nWeaknesses:\n\n1. Some parts of the math derivation are not very clear to me and may need further explanation. For example, between equations (6) and (7), “… we consider the following selection event…” for the global null test, the sign matching part is not very obvious to readers. \n\n2. On the line after equation (12), it is claimed that the variable Z is normally distributed, which is an important assumption for subsequent derivations to calculate the p-value. It could be further explained why it is normally distributed. \n\n3. The authors claimed that their Keras implementation of Auto-Conditioning is efficient, but provides no asymptotic analysis of the computational complexity of their algorithm. This is important when applying this method to practical applications. \n\n4. When using the brain image dataset in the experiment, does it matter which image is selected as the reference for the proposed method to work? What is the effect of selecting different reference images? \n\n5. The authors performed experiments using a synthetic dataset and a real brain image datasets, and compared the False Positive Rate and True Positive Rate of the proposed method and some other methods for the synthetic dataset. I suggest the authors do the same analysis for the real dataset, and do some more analysis with another natural image dataset, which is usually used to compare different saliency methods.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dbMr59_FVtP", "reviewer": "Reviewer_TSgM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a conditional selective inference (SI) approach for obtaining p-values for saliency maps. By leveraging the piecewise linearity of ReLU networks, they develop a method for performing the necessary hypothesis testing in an automated fashion across arbitrary network architectures. In experiments, their method outperforms several baselines", "summary_of_the_review": "Overall, this seems like a solid technical contribution in an area that people care about. I don't know much about the area, so I have low confidence, but I'm leaning towards acceptance after spending 2 hours reading the paper. I may be missing something important, though, and would be willing to adjust up or down depending on other reviews and author feedback.\n\n--------------\n\nUpdate after rebuttal:\n\nThe authors have addressed my concerns. I'm increasing my confidence in the accept decision. I'm not confident enough to increase the score to an 8 (the next highest increment).", "strength_and_weaknesses": "Strengths:\n- The proposed auto-conditioning method seems quite novel (conceptually and technically)\n- Interpretability is an important problem area\n- Compared to baselines, the proposed method greatly reduces false positives.\n\nWeaknesses:\n- Since I am not very knowledgeable about selective inference, I'm not sure whether the proposed method actually allows one to \"quantitatively evaluate the stability and reproducibility of DNN-driven salient regions\". If the authors could include some commentary or clarification of how the proposed method actually improves the usability of saliency maps, that would make the paper much more approachable for the broader ICLR audience. For instance, is the practical impact that one doesn't need to worry about setting thresholds for detection (assuming knowledge of the noise distribution)?\n- It seems like the proposed method might require knowing the noise distribution. Is this a limitation?\n- The noise is pixel-wise, which doesn't seem to capture important forms of structural variation. For instance, the reference image and input image have slight misalignments, which could be a problem for the proposed method. The authors don't seem to discuss this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BLOIqFuyKs", "reviewer": "Reviewer_rBPS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for quantifying the reliability of saliency maps extracted from deep nets. Specifically, they perform a two-sample test, where the test statistic is based on the sum of differences between the given sample and a reference within salient regions. The main difficulty lies in how to compute the distribution of the test statistic, as the saliency depends on a deep net. To solve this challenge, they propose and prove an algorithm to compute this distribution for a family of piecewise linear networks. Empirical results are also provided on a synthetic dataset and a real-world brain image dataset. ", "summary_of_the_review": "I have some concerns about the usefulness of the developed statistical test and need some clarification regarding their model assumption. See the weaknesses section for more details.", "strength_and_weaknesses": "\n# A. Strengths\n### A1. The topic of quantifying reliability for saliency maps is a topic of interest and relevant to the community. \n### A2. The proposed approach is based on theoretical justification and overall the presentation is adequate. \n### A3. Code and scripts to reproduce the paper are provided in the supplemental materials. \n\n\n# B.  Weaknesses\n### B1. Discussion regarding Duy et al (2022) seems vague. The paper states\n> In this study, we docs on a more general problem setup.\n\nIn what ways? Empirical comparison in the same setup as Duy et al (2022)? The current description does not fully disclose this paper's contribution. Please clarify.\n\n\n### B2. Justification for the test statistic in Eq. (4). The test statistic is defined as\n> $\nT(\\mathbf{X}, \\mathbf{X}^{\\text{ref}}) = \\sum_{i \\in \\mathcal{M}_{X}} |X_i - X_i^{\\text{ref}}|\n$\n\nThis is pixel-wise comparison of the elements within the salient region. However, pixel-wise differences are not particularly meaningful, especially, for capturing higher-level concepts in an image. Can the author clarify the motivation for using such test statistics?\n\n\n### B3. What is the practical application for performing a two-sample test when the test-outcome depends on the provided reference? In particular, how is the reference image selected in practice, e.g., in real data experiments? \n\n\n### B4. The paper states that for piecewise linear network\n> This is also true for the saliency map function $\\mathcal{A}_i(\\mathbf{X})$.\n\nI believe this statement is inaccurate, it depends on which saliency method is used. \n\nAdditionally, is this assumption true in practice? Typically, saliency methods are applied to classifier networks which contain softmax non-linearities for CNNs; similar issues with, the recently popular, transformer achirectures. \n\nHence, the title “Deep learning-driven” salient region seems to be a bit of an overstatement.\n\n\n### B5. Some experimental details are not clear. \nThe network architecture is provided in the supplemental material, however, it is unclear how the network is trained and on what annotation.\n\n\n### B6. Have the authors' considered other datasets beyond the brain image provided? \n\nEvaluation of more general/large-scale computer vision datasets could strengthen the contribution of this paper. As the authors claimed contribution on\n> develop a Keras-based framework for conducting the proposed selective inference ... without additional implementation cost.\n\nIt seems like the method can be easily applied to other datasets, why not demonstrate some more results?\n\n### B7. Any thoughts on Bayesian approaches, e.g., [I],  to uncertainty? A discussion would be interesting to have.\n\n[I] Kendall, Alex, and Yarin Gal. \"What uncertainties do we need in bayesian deep learning for computer vision?.\" Advances in neural information processing systems 30 (2017).\n\n### B8. The notation for $\\tilde{\\Sigma}$ should be clarified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster