Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work attempts to shed some light on the reliability of saliency maps.
Specifically, the authors propose a conditional Selective Inference (SI) method. The proposed method can provide valid p-values for statistically quantifying the reliability of the salient regions. The proposed method is evaluated on a synthetic dataset and a brain image dataset.
strength_and_weaknesses: Strengths:

1. This work looks technically sound.

2. The proposed method is correlated to statistical hypothesis tests. This idea is interesting and could be practically useful in some real-world scenarios, e.g., medical applications.

Weaknesses:

1. In this work, the saliency map is presumed to be generated by CAM (Zhou et al., 2016). We don't know how well the proposed method works with other CAM-like saliency map methods (e.g., Grad-Cam). Correspondingly, in the theory part, how the saliency map model A(\cdot) affects the proposed selective p-value is unclear.

2. How the proposed selective p-value corresponds to the correlation between the salient region and the reference region is unknown. In other words, can the selective p-value be aligned with the visual difference/similarity between salient regions and reference regions? One way to measure the linear correlation coefficient between a model saliency map and an empirical saliency map is to use correlation coefficient (CC). Please refer to \url{https://saliency.tuebingen.ai/results.html} for more details.  

3. Last but not least, the experiments are less convincing as the proposed method is only evaluated on a synthetic dataset and a small-scale dataset. I cannot tell if the proposed method can work well on other general yet complicated real-world datasets.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written in my opinion. The proposed method is novel and provides new insights into saliency map detection. 

The authors discuss the implementation in the paper and provide the code for review. I haven't verified if the experiments in this paper can be reproduced though.
summary_of_the_review: Overall, I think the proposed conditional Selective Inference method is interesting and insightful. However, my main concern lies in the empirical evidence. It would be better to see more experimental results on other datasets, e.g., a natural image dataset, like CIFAR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a conditional selective inference (SI) approach for obtaining p-values for saliency maps. By leveraging the piecewise linearity of ReLU networks, they develop a method for performing the necessary hypothesis testing in an automated fashion across arbitrary network architectures. In experiments, their method outperforms several baselines
strength_and_weaknesses: Strengths:
- The proposed auto-conditioning method seems quite novel (conceptually and technically)
- Interpretability is an important problem area
- Compared to baselines, the proposed method greatly reduces false positives.

Weaknesses:
- Since I am not very knowledgeable about selective inference, I'm not sure whether the proposed method actually allows one to "quantitatively evaluate the stability and reproducibility of DNN-driven salient regions". If the authors could include some commentary or clarification of how the proposed method actually improves the usability of saliency maps, that would make the paper much more approachable for the broader ICLR audience. For instance, is the practical impact that one doesn't need to worry about setting thresholds for detection (assuming knowledge of the noise distribution)?
- It seems like the proposed method might require knowing the noise distribution. Is this a limitation?
- The noise is pixel-wise, which doesn't seem to capture important forms of structural variation. For instance, the reference image and input image have slight misalignments, which could be a problem for the proposed method. The authors don't seem to discuss this.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality:
- The quality of the writing is good. The codebase in the supplementary material is good.
- Just below equation (4), the authors mention eta_{M_x}, but this isn't defined anywhere nearby. What is this variable?
- The authors use unusual symbols for referencing figures. Clarity would be improved by using the hyperref or cleverref packages and typesetting references as "Figure 1", "Table 1", etc. I think this might actually be in the style guideline file.


Novelty:
The novelty is good overall. The technical contribution with auto-conditioning seems substantial.

A minor point: The authors say "To our knowledge, this is the first method that can provide valid p-values to statistically quantify the reliability of the DNN-driven salient regions", but Duy et al. (2022) (the closest work to this one) say pretty much the same thing in their introduction. The authors explain their contributions relative to Duy et al., but it might be worth qualifying this statement to reflect those contributions.
summary_of_the_review: Overall, this seems like a solid technical contribution in an area that people care about. I don't know much about the area, so I have low confidence, but I'm leaning towards acceptance after spending 2 hours reading the paper. I may be missing something important, though, and would be willing to adjust up or down depending on other reviews and author feedback.

--------------

Update after rebuttal:

The authors have addressed my concerns. I'm increasing my confidence in the accept decision. I'm not confident enough to increase the score to an 8 (the next highest increment).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method for quantifying the reliability of saliency maps extracted from deep nets. Specifically, they perform a two-sample test, where the test statistic is based on the sum of differences between the given sample and a reference within salient regions. The main difficulty lies in how to compute the distribution of the test statistic, as the saliency depends on a deep net. To solve this challenge, they propose and prove an algorithm to compute this distribution for a family of piecewise linear networks. Empirical results are also provided on a synthetic dataset and a real-world brain image dataset. 
strength_and_weaknesses: 
# A. Strengths
### A1. The topic of quantifying reliability for saliency maps is a topic of interest and relevant to the community. 
### A2. The proposed approach is based on theoretical justification and overall the presentation is adequate. 
### A3. Code and scripts to reproduce the paper are provided in the supplemental materials. 


# B.  Weaknesses
### B1. Discussion regarding Duy et al (2022) seems vague. The paper states
> In this study, we docs on a more general problem setup.

In what ways? Empirical comparison in the same setup as Duy et al (2022)? The current description does not fully disclose this paper's contribution. Please clarify.


### B2. Justification for the test statistic in Eq. (4). The test statistic is defined as
> $
T(\mathbf{X}, \mathbf{X}^{\text{ref}}) = \sum_{i \in \mathcal{M}_{X}} |X_i - X_i^{\text{ref}}|
$

This is pixel-wise comparison of the elements within the salient region. However, pixel-wise differences are not particularly meaningful, especially, for capturing higher-level concepts in an image. Can the author clarify the motivation for using such test statistics?


### B3. What is the practical application for performing a two-sample test when the test-outcome depends on the provided reference? In particular, how is the reference image selected in practice, e.g., in real data experiments? 


### B4. The paper states that for piecewise linear network
> This is also true for the saliency map function $\mathcal{A}_i(\mathbf{X})$.

I believe this statement is inaccurate, it depends on which saliency method is used. 

Additionally, is this assumption true in practice? Typically, saliency methods are applied to classifier networks which contain softmax non-linearities for CNNs; similar issues with, the recently popular, transformer achirectures. 

Hence, the title “Deep learning-driven” salient region seems to be a bit of an overstatement.


### B5. Some experimental details are not clear. 
The network architecture is provided in the supplemental material, however, it is unclear how the network is trained and on what annotation.


### B6. Have the authors' considered other datasets beyond the brain image provided? 

Evaluation of more general/large-scale computer vision datasets could strengthen the contribution of this paper. As the authors claimed contribution on
> develop a Keras-based framework for conducting the proposed selective inference ... without additional implementation cost.

It seems like the method can be easily applied to other datasets, why not demonstrate some more results?

### B7. Any thoughts on Bayesian approaches, e.g., [I],  to uncertainty? A discussion would be interesting to have.

[I] Kendall, Alex, and Yarin Gal. "What uncertainties do we need in bayesian deep learning for computer vision?." Advances in neural information processing systems 30 (2017).

### B8. The notation for $\tilde{\Sigma}$ should be clarified.

clarity,_quality,_novelty_and_reproducibility: Overall, the paper is clear and they also provided code for reproducibility. The code is commented and the quality and readable.
summary_of_the_review: I have some concerns about the usefulness of the developed statistical test and need some clarification regarding their model assumption. See the weaknesses section for more details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors proposed a method to quantify the reliability of the saliency region in the form of p-values by employing the selective inference (SI) framework. In the proposed method, the authors regard the salient region proposed by DNN explainability methods such as CAM as a selected hypothesis. Since the hypothesis proposed by the model is not independent of the data, instead of classical inference with selection bias, the authors proposed to use selective inference where the inference is conditioned on a selected event. The authors also demonstrated the validity of the method through numerical and real datasets. 

The main contributions of this work:

1. The authors employed the selective inference framework to compute a p-value to quantify the reliability of a salient region proposed by a saliency model to overcome the selective bias problem. 

2. The authors also claimed to have provided a Keras-based implementation to efficiently and flexibly conduct conditional SI for salient regions for a wide class of CNNs.

3. The authors conducted experiments on both synthetic and brain image datasets and claimed the proposed method is able to control the false positive rate (FPR) and has potential to be applied to practical applications. 

strength_and_weaknesses: Strength:

1. The paper in general clearly explained the motivation of the work by identifying the problem with existing statistical inference framework, i.e. selection bias. 

2. The authors employed the selective inference framework to address the issue of selection bias in the context of saliency region detection and reliability estimation.

3. The authors claimed that the proposed method is applicable to a wide class of CNNs, demonstrated by an analysis of piecewise linear network. 

4. The authors claimed to have implemented the proposed method in an efficient way and provided the code for reproducibility.

5. The experiments provided also supported the claims made by the authors regarding the validity and applicability of the proposed method on practical problems. 

Weaknesses:

1. Some parts of the math derivation are not very clear to me and may need further explanation. For example, between equations (6) and (7), “… we consider the following selection event…” for the global null test, the sign matching part is not very obvious to readers. 

2. On the line after equation (12), it is claimed that the variable Z is normally distributed, which is an important assumption for subsequent derivations to calculate the p-value. It could be further explained why it is normally distributed. 

3. The authors claimed that their Keras implementation of Auto-Conditioning is efficient, but provides no asymptotic analysis of the computational complexity of their algorithm. This is important when applying this method to practical applications. 

4. When using the brain image dataset in the experiment, does it matter which image is selected as the reference for the proposed method to work? What is the effect of selecting different reference images? 

5. The authors performed experiments using a synthetic dataset and a real brain image datasets, and compared the False Positive Rate and True Positive Rate of the proposed method and some other methods for the synthetic dataset. I suggest the authors do the same analysis for the real dataset, and do some more analysis with another natural image dataset, which is usually used to compare different saliency methods.  

clarity,_quality,_novelty_and_reproducibility: The paper is well written with high quality and clarity. The math derivation of SI for the salient region p-value calculation is also clear. The work is novel by applying SI for salient region detection. The code is also provided for reproducibility though I didn’t evaluate the code for reproducibility for the purpose of this review. 
summary_of_the_review: This paper proposed to evaluate the reliability of a salient region with a p-value by employing the selective inference (SI) framework to address the selection bias problem with classical inference. The authors also claimed to have provided an efficient Keras-based implementation of the proposed method, and experiments have conducted to prove the validity of the proposed method. In my opinion, this work is well motivated, the solution is clearly explained, and the claims are also supported by empirical results. I suggest the authors to perform more experiments on natural images with additional analysis for deeper insights.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on saliency map of class activation. An inference method is proposed to quantify the reliability of a saliency region in the form of p-values based on the concept of SI. The experimental results show the improved performance.
strength_and_weaknesses: Strength:

1. The writing flow of the paper is very nice, including comprehensive background and detailed related work. The motivation is clearly stated.
2. The theoretical results are well presented. The assumptions made for generating such results are clearly described and contextualized.
3. Experimental results consider several reasonable baselines and demonstrate the improvement produced by the method.

Weaknesses:

1. The work relies largely on the previous work to prove an exact inference method for the salient regions. Specifically, relying on similar assumptions and derivation of [1], the paper extends image segmentation to saliency maps. The mathematical formulas, the two-sample test, the conditional distribution, the selective p-value, and the piecewise linear network are all similar to [1]. It looks like an application based on [1]. Thus, it seems that the algorithm itself is not very novel.

2. Is it possible to provide some experimental evidence for the argument that the proposed selective inference works well for a wide class of CNNs without additional implementation cost? How does the method integrate with various CNNs? For this issue, the authors may report the performance of different CNNs. In addition, the proposed method seems computationally expensive. How efficient is the algorithm compared with the baseline methods?

[1] Vo Nguyen Le Duy, Shogo Iwazaki,and Ichiro Takeuchi. Quantifying statistical significance of neural network-based image segmentation by selective inference. NeurIPS, 2022
clarity,_quality,_novelty_and_reproducibility: The novelty of the paper is limited due to its similarity to the aforementioned NeurIPS paper.
summary_of_the_review: Considering the novelty and the rigorousness of the paper, I hold a slightly negative stance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method for quantifying statistical significance for saliency maps derived from deep models, in particular in the context of "explainability" evaluation. The focus of evaluation is on medical images. The work builds significantly upon prior work by Duy et al., but extended it in particular to allow for separate reference image. 

Besides the relationship to the prior work, which I think the authors have resolved (leaving the question about the significance of the "delta"), the main concern is the limited nature of the test statistic, which effectively assumes pixel-wise alignment between the images. In my view that is indeed an issue, and a big limitation of the proposed method. There authors acknowledge this but are, in my opinion, a bit too cavalier about the likely ease of overcoming it.

Overall, after further discussion with the reviewers, I believe this paper is just barely above threshold. If it's accepted the authors can 

justification_for_why_not_higher_score: Not enough novelty, significance or likely impact to justify further promotion.
justification_for_why_not_lower_score: This is the weakest paper among those I recommend for acceptance. I list the points in favor above. But I would not be upset if the paper was ultimately left out.
Decision: Accept: poster