Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors proposed an algorithm for interpreting deep models, which is an extension of Fel et al., 2021. The main tool it relies on is called Sobol first-order sensitivity. The experiments are demonstrated on different CNN variants, including VGG, ResNet and DenseNet. 
strength_and_weaknesses: - The main criticism is on the novelty. Most of the components have been used in Fel et al., 2021.   Until equation (11), most of the contents are covered by Fel et al 2021, and the new ingredients from this paper seems to be equation (12) only.  I would encourage authors to spend more space discussing the new components proposed by this paper instead of repeating what people have done before. Also, It would also be nice to provide literatures about Sobol methods,  as Fel et al., 2021 is not the one proposing this metric but proposing to reuse this tool. Some important literatures include
    * I. Sobol’, “Sensitivity estimates for non linear mathematical models,” Mathematical Modelling and Computational Experiments, vol. 1, pp.
407–414, 1993.
    * B. Li and C. Chen, “First-order sensitivity analysis for hidden neuron selection in layer-wise training of networks,” Neural Processing Letters, vol. 48, pp. 1105–1121, 2018.

- It is not clear to me how we construct the inverse function of H and G.  The inputs of H and G are with higher dimensions than their outputs, which the inverse functions "usually" don't exist. Also, the function includes some pooling operations (sum), it is counter-intuitive to be able to construct an exact inverse function. More descriptions would be helpful for readers to understand. 

- The proposed methods are based on comparing the variances of different components, such as equation (10) and equation (11). However, there are already a lot of literatures discussing the variance of features for interpretability. The oldest one might from linear regression literatures, such as the notion of R^2. There are also more recent one worths discussing in the literature. 

- Other minor points:
    - The authors criticize the limitation of GradCAM is heavily depending on the architecture, which I think it's not a proper statement, as it has been widely applied to many different architectures with practical success. I would like to know authors' opinion on it.
clarity,_quality,_novelty_and_reproducibility: The method is fairly simple and most of them are from the previous works, so I believe it should be easy to reproduce. On the other hand, the novelty is a concern as mentioned above.
summary_of_the_review: The main concern is the novelty. There are also some components which seems not right to me, and I would like to see authors' justification in the rebuttal.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors suggest a new method for interpreting decisions by deep learning models using a variance-based sensitivity analysis method. By such analysis of the input features the authors claim that they can analyse not only the contribution of individual features but their relation as well, for example, such image patches. The method is evaluated in an experimental setup using image net and some known CNNs.
strength_and_weaknesses: I like first, the questions asked and second the idea of exploiting known methods from feature relevance analysis to understand the output of black-box models. I am kind of disappointed about the implementation of this general, very promising idea. The details of combining Sobol with SBSA are insufficient to support their claims. For example, why shall the sum of pixels in a region be a proper semantic representation of an image(part)? Also, the generation of the low-dimensional feature is arbitrary by the selection of image patches. I am missing a detailed study and motivation for this step. Second, experimental results lack clarity and insight. When comparing with other methods to explain the output of a CNN, their metric is not always the best. Overall, the metrics used, for example, PIR, deliver interesting insight, but in my opinion not more. Figure 3 shows an observation for a very specific pipeline of SBSA and Sobol, and not even one that I find reasonable (see my comment on a feature that uses the sum of intensity values). In summary, the idea is very promising but the current result is premature and without a significant contribution to the area. 
clarity,_quality,_novelty_and_reproducibility: Novelty is high, and I am always in favour of not re-invent the wheel. Sobol is worth further study. However, the quality of the paper in terms of clarity is only average. Reproducibility is limited since no information can be found on which images of ImageNet have been tested.  
summary_of_the_review: This paper is not ready for publication. More needs to be done to arrive at significant statements about the development of CNN architectures and their impact on the representation capability in the model. Compared to other works on interpreting CNN output, this paper lacks theoretical contribution and experimental evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a sample-based approach for interpreting the decision-making of neural networks for an image classification task. An image is divided into multiple regions and the importance of a region is evaluated separately along with its interaction with other regions. A Sobol variance-based sensitivity metric is used to estimate the importance of regions by varying the region and noting the effect on the prediction. The approach is validated with Imagenet validation images with respect to commonly used image classification models such as AlexNet, CGG16, ResNet50, and DenseNet161. 
strength_and_weaknesses: Strengths:

1. The proposed sample-based approach to interpreting the decision-making of neural networks is interesting. 

2. The Sobol variance-based sensitivity metric is a simple yet effective approach to estimating the importance of regions. 

3. Authors conducted extensive experiments to evaluate various aspects of the approach. 

Weaknesses:

1. Dividing images into a few regions to interpret the decision-making is simplistic. The proposed approach sum the pixel values in a region which may squash important variance cues in a region. Additionally, a coarse grid such as 4X4 or 8X8 may not respect object boundaries. What are the authors' comments on these issues?

2. Authors claim to capture interactions between regions which is missing in the existing works. However, the way the interaction is measured is effectively based on the individual importance of the regions. I am missing what aspect of Eq 11 captures non-linear region dependencies for decision-making that are missing in Eq 9?

3. Authors mentioned the G^{-1} and H^{-1} functions. Where are these inverse functions used?

4. Can G and H be combined in a single encoder and G^{-1} and H^{-1} combined in a single decoder? For example, one can combine G and H in a single encoder such as VAE and consider the latent representation as the lower dimensional representation $r$. Is the proposed approach applicable to such representations? 

5. As in figure 3 (a), the interactions between the regions increase with a lower number of regions. This is counterintuitive as discussed in section 4.2.3. This raises the question of how the approach depends on the size of the regions. It will also be useful to visually inspect the results on a set of images while changing the size of the regions.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, the novelty is incremental, and the approach can be reproduced.
summary_of_the_review: The paper provides an interesting idea of sample-based interpretability estimation. However, there are many aspects that need to be addressed to justify the impact of the approach. Please address the issues in the weaknesses section and I will be happy to reconsider the decision.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a sample-based approach for interpreting neural net decisions. I find the main findings of the paper about region interactions and reliance on the background interesting. However, the agreement among the reviewers is that the technical/empirical novelty and significance of the paper is currently marginally below acceptance standards and is not ready for publication. Therefore, I recommend rejection. I want to recommend authors to take reviewers suggestions into account to improve the paper. In particular, I find reviewer phNs's suggestions to be very helpful for improving the quality of the paper. 
justification_for_why_not_higher_score: Because the technical/empirical novelty and significance of the paper is limited
Decision: Reject