Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose HeatFlow, a framework based on the heat equation on a Riemannian manifold, to interpreting a representation of givn deep learning model. Especially, they focus to analyze the multi-sale behvariors of the model, which cannot be revealed by using a naive gradient-based model interpretation approach. The authors show that the proposed HeatFlow framework obeys some nice properties that frequently used in the interpretable AI field, e.g., attribution axioms. The authors compare the proposed method with four baselines including Grad, IG, SG and BlurIG
strength_and_weaknesses: Disclaimer:

 I am not an expert of the model interpretaion and explainable AI fields. My evaulation on this paper is not exhaustive and might be incorrect.

Strength:

1. The paper is based on solid theoretical properties from differential geometry and PDE. It seems that the proposed method is technically sound.

2. Experimental results show that the proposed HeatFlow is likely to interprete the model more convincely compared to other baselines.

Weakness:

1. The proposed method can only address a scalar case, i.e., 1D regression problem or binary classification. However, it can be by-passed by investigating each output of multi-class/multi-dimensional models separately.

2. The paper is a little dense and hard to read. It might be because I am not familar with the math used in this paper.


clarity,_quality,_novelty_and_reproducibility: Clarity:

The paper is a little dense and hard to read. It might be because I am not familar with the math used in this paper.

Quality:

The mathematical background used in this paper seems to be solid.

Novelty:

To me, the use of the heat equation on a Riemannian manifold to interpreting deep learning models is novel. However, I do not know modern literatures on this field well-enough to evaluate correclty.

Reproduciblity:

The paper contains an incomplete set of used model architectures and hyper-parameters. Experimental code is not made publicly available (although it might be not a ciritcal issue considering this paper is theoretical one).

summary_of_the_review: While it is an educational guess, I think this paper contributes both a solid theoretical background and practial method for the relevant field. I would like to vote to accept this paper. However, since I am not an expert in this field, I cannot evaluate this paper exhaustively; thus, my rating might be updated after discussing with the authors and other reviewers.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper under review uses heat diffusion process to understand some multi-scale behavior of the learned model around a test point. Summary scales which characterizes the model on different scales can be drawn. 
strength_and_weaknesses: Strength: This paper is clearly written and well-organized. 
Weakness: The paper introduces many math notations, while no enough emphasis on why heat flow can help us better understand the functions represented by neural networks for instance. There is no really crucial part which help us advance the understanding of DNN for instance. The results showed here are too toy models. 
clarity,_quality,_novelty_and_reproducibility: Clarity: Good. Quality: Fair; Novelty: Fair; It is a theoretical paper. I do not check the reproductivity. But the proof is right but the results are somehow standard in math community. 
summary_of_the_review: I would not suggest to accept this paper. It would be better developed before resubmit to a journal or a conference proceeding.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposes the HeatFlow. 
It calculates the feature attribution using the model's outcome and local average values.
It provides multi-scale explainability results.
strength_and_weaknesses: Pros:
The proposed FeatFlow is proved to be an additive model and satisfies the four axioms of the Shapley value. 
This proof provides solid theory support that FeatFlow can provide fair feature attributions.

Cons:
This paper is a little hard to follow.
Although the authors define each symbol, 
it's still hard to connect these abstract symbols to the explainability task.

For example, in section 2.1, 
I think it can be better if more details about the Laplacian-Beltrami operator are given and provided some sample examples to show the relationship or motivation to apply this method on explainability.

A detailed pseudo-algorithm may be a better way to show the methodology clearly.

Question：
In Figure 4, it seems like IG gets the best result on the MNIST exclusion curve and it's better than the proposed HeatFlow.
However, in the UTKFace exclusion curve, it seems like IG has a much lower MAE than heat and SG.
Could you provide some insight into this phenomenon? I am curious about this.

clarity,_quality,_novelty_and_reproducibility: The idea is interesting to use the diffusion process in explainability. But the writing of this paper is hard to follow and the connection between the math symbol and explainability task should be added.
summary_of_the_review: Above all, I think this paper provides an interesting framework to apply the PDE method to the explainability task. 
HeatFlow can provide a fair attribution assignment by obeying four axioms in Shapley value.
However, the paper is hard to follow, and more details should be included to help the readers to understand.
I am willing to raise my score if the paper flow becomes easier to follow.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new neural network interpretation and attribution framework which looks at heat flow to achieve multi-scale interpretation of a scalar-valued function
strength_and_weaknesses: 
Pros:
- It is an interesting and novel idea to use the diffusion equation for the attribution problem
- The paper is generally well written

Cons:
- Evaluation strategy and analysis (see comment 1 below)
- Related literature (see comment 2 below)

clarity,_quality,_novelty_and_reproducibility: Clarity: the paper is clearly written.

Quality: there are concerns about the evaluation strategy, as as things stand, the evaluation is limited and does not match up with state-of-the-art evaluation practices. 

Originality: the idea builds upon the existing ideas of using heat equation, but it is applied in a different domain of interpretability of machine learning models, and therefore is novel. The efforts must be made to refer to existing machine learning approaches using heat equation (see Question 2).

Comments on quality:
1. Evaluation strategy can be improved in multiple ways. First, there is a doubt about the existing experiments. The first question would be : is MNIST a good dataset for evaluating interpretation? There is not much of a texture, and nearly any peak around the  contours of a symbol could be seen as a successful attribution. The results from the appendix, section A.9, make more sense to me. I think it would be great if the authors could comment on usability of MNIST dataset, maybe the reviewer misses something.  It is also important to show the legends for the heatmaps, see e.g. Figure 2 (which would clarify that say lower values are blue and the higher values are red). For Figure 4, it would be also important to include other datasets, including high-textured ones. Another question to answer is how it performs on the tasks related to the attribution, as it would help understand whether we can highlight specific features of the image (and also make it visually comparable to the other works). Simonyan et al, for example, use high-resolution dataset ILSVRC-2013. Xu et al (2020) use ImageNet and Diabetic Retinopathy data.  Also, do the predictions help identify causes of failures/what would the attribution look like for erroneous predictions?
2. Related literature: the paper discusses explanation methods, but it also needs to cover somehow the background of diffusion models and heat equations which has also recently been popular in the literature (see Kingma et al, 2021). 
3. P5: increasing larger ts -> increasingly larger ts
summary_of_the_review: The experimental evaluation needs to be more extensive, matching existing published models for interpretability and attribution (see Comment 1 above). Therefore, I do not recommend acceptance as things stand at this stage (but I welcome clarifications and improvements)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper interprets neural nets by using heat diffusion process. The reviewers are not convinced by the technical and empirical results in this paper. The writing also lacks clarify, and the experimental results are not comprehensive. Thus a reject is recommended.
justification_for_why_not_higher_score: The reviewers are not convinced by the technical and empirical results in this paper. The writing also lacks clarify, and the experimental results are not comprehensive. Thus a reject is recommended.
justification_for_why_not_lower_score: NA
Decision: Reject