Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript proposes to bridge the gap between post-hoc explanation methods and interpretable-by-design methods.

This is achieved by a pipeline where given a black-box model, an interpretable component iteratively distills parts of the representation from the blackbox with uncovered (undistilled) parts forwarded to a residual component. By iterating on this process, the interpretable components achieves a higher coverage of the blackbox model while still preserving a competitive performance.

Experiments on several datasets show the effectiveness of the proposed method on the image classification task for a variety of datasets. In addition, an alternative application of the method is presented on the task of bias detection.

strength_and_weaknesses: Strengths
+ well-supported idea
+ tested for bias removal
+ Good presentation

Weaknesses
- Relevant aspects not evaluated
- Limited novelty
- Some open questions
clarity,_quality,_novelty_and_reproducibility: Clarity
The manuscript has a good flow and its content is relatively easy to follow. Overall it was a good read.

Quality
The quality of the manuscript is good

Novelty
While the idea of using a surrogate model to enable the explanation of an existing model is not new, the way in which the proposed method implements that idea has several aspects that I find novel.

Reproducibility
The presentation of the method is relatively clear. Unless I overlooked something, I would not expect major challenges while trying to reproduce the reported results.
summary_of_the_review: The proposed idea is sound and most of the design decisions are well motivated. To the best of my knowledge, the idea of having this iterative combination of interpretable components with residual components is novel. 

I also found positive the fact that the proposed method aims at interpreting different levels of the blackbox model and does not limit itself to the last convolutional layer, as is commonly the case. 

The presentation of the manuscript is good, and the structure followed for the presentation of its content is adequate.

My main doubts with the manuscript are the following:

- Positioning with respect to Prototype-based methods. In Section 2, the paper discusses ProtoPNet as a representative of prototype-based methods. As part of this discussion, weaknesses from ProtoPNet are stressed. My concern here, is that ProtoPNet is nothing more than the first of these family of prototype-based methods. In this regard there are more recent variants, e.g. Proto-Tree (Nayuta et al., CVPR21) which have addressed the weaknesses of the original ProtoPNet model. Here I would expect the positioning to be made with respect to more recent prototype-based methods.

- On the selection of the number of experts. When describing the training configuration, the number of experts used for each blackbox-dataset combination was indicated. However, it is not clear to me the basis on which the number of experts is selected. Is there a principle manner to decide on the number of experts to use for the proposed method?

- On the semantic association of the detected concepts. In Section 4.2.2, local explanation produced by the MoIE are discussed. Several of these are described based on concepts with a clearly associated semantic meaning , e.g. IrregularStreaks, IrregularDG and Blue Whithish Veil. H 

- On the explanation capabilities of the proposed method. Independently of its inner-workings, at the end of the day the proposed method is an explanation method. In this regard, it would have strengthen the manuscript if the proposed method was tested under the sanity checks proposed by [Adebayo et al., NeurIPS'18]. This way there could be guarantees on whether the generated heatmaps/attribution maps do constitute valid explanations.
 
- Quantitative Comparison. At the moment there is no quantitative comparison on the outputs of the proposed methods with respect to those from state of the art methods. In this regard, quantitatively speaking, it is hard to assess whether the proposed method is a better explanation method than existing ones.
 
- Minor. There are typos in several places of the manuscript. 
 
-------------- 
References
 
 Meike Nauta, Ron van Bree, Christin Seifert; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14933-14943
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes a method to iteratively extract a mixture of interpretable models from a trained Blackbox
strength_and_weaknesses: + Interesting problem
- Clear missing state of the art cf. below
- Limited experiments - specially in the context of semantic segmentation
clarity,_quality,_novelty_and_reproducibility: + Clear problem
- No novelty as state-of-the-art is missing cf. (1), and specially (2) which is basically capturing AND/OR expression for annotating neurons, and then could used for computing what has been described in the paper.

(1) David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, Antonio Torralba: Network Dissection: Quantifying Interpretability of Deep Visual Representations. CVPR 2017: 3319-3327
(2) Jesse Mu, Jacob Andreas: Compositional Explanations of Neurons. NeurIPS 2020



summary_of_the_review: Overall the paper is well described, but novelty is not high given past works have achieved similar results
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose an approach to explain black box classifiers by iteratively dividing the black-box into an interpretable model and a residual. They use first order logic on concepts for the interpretable models.
A sample input is passed to the system and either routed (by a learnable gating mechanism) to the interpretable model or the residual, until it is interpreted or reaches the last residual.
strength_and_weaknesses: ### strenghts
- the explanations are relatively easy to read thanks to FOL
- experiments are numerous and well described
- the authors demonstrate that biases can be detected

### weaknesses

- a short explanation of the first order logic is missing
- a short explanation of ``neuro-symbolic interpretable model'' is missing
- no code
- complicated approach, training required
- no comparison with similar concept based explanation methods (like CAV)
- no quantitative analysis of the produced explanations


### questions
- You write in section 3.1.1 that the selector routes a sample with a probability... Is this routing really probabilistic, so that I would potentially get different explanations when explaining the same image twice?
- You are using the original black box classifier in the first iteration, if I understand correctly. Why is the performance in Figure 2 then not equal to the black box?
- why do you not show explanations for the same samples in figure 4?
clarity,_quality,_novelty_and_reproducibility: ### novelty
The iterative approach combining several neuro-symbolic interpretable models is novel. 

### clarity
The paper is, for the most part well written. Notation is fine. Some additional descriptions of concepts from other work that the authors use would have been nice.

### Reproducability
Pseudo code is given but no github implementations, so reproducibility may be cumbersome.

### Quality
Given that the approach is quite complicated, the motivation seems a bit poor.
Experiments are of good quality. Formulas are a bit hard to understand.
summary_of_the_review: Overall I'm not quite convinced by the paper. As the procedure is relatively complicated and requires training, the motivation should be a bit clearer. Furthermore, there needs to be more quantitative evaluations and comparisons to state of the art concept based explanations.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper advocates a hybrid approach between inherent interpretability and post-hoc explainability for predictive models trained on image data. The aim is to blur the line between post hoc explanations of a Black-Box and constructing interpretable models, aiming to keep the high performance and flexibility of blackbox models while providing interpretability. To this end, the authors propose an iterative approach where at each iteration, an interpretable model is distilled from the blackbox model. Samples that cannot be routed through the interpretable model(s) at an iteration are routed through a residual network, which is again a blackbox. The method is repeated on the residual network until until the proportion of data explained by the residual network falls below a desired threshold.
strength_and_weaknesses: Strengths: The motivation is clear, the method is described well and the experiments conducted are extensive. The approach has been tested on a variety of datasets and the results have been discussed extensively and from various angles.

Weaknesses: Section 2 on related work seems a bit short, especially the paragraph on concept- based interpretable models could be extended. What methods are used by others and how does the paper at hand differ from those? In line with that, the contributions of the paper in relation to previous work could be highlighted more. Similarly, there are no comparisons done with existing methods (except against the ‘baseline’). If there is a specific reason for that, it was not clear. If not, I would advocate comparing to current state-of-the-art methods for symbolic models for image data. Section 5 (discussion and conclusion) would benefit from a discussion of the strengths and limitations of the presented work. Future research trajectories could be discussed more detailed. 

Here are a few additional remarks and questions:

- The readers would benefit from a brief explanation about neuro-symbolic interpretable models in Section 3.1.

- The optimization problem (2) looks highly non-linear and non-convex. The authors have not mentioned difficulty of the problem, neither have they discussed the cases of obtaining multiple or only local solutions. They only propose one method to solve the problem in the appendix. However, this part is crucial as the method relies on the solution of this problem.

- What is a good value for K to stop the algorithm? Is there a rule-of-thumb when to stop?

- What is the effect of validation threshold (fixed to 0.7 in the paper)?

- How would the method extend to non-image datasets?

- What is a concept extractor in Section 4.1?

- There are few typos that I spotted in the current version:
   * In the beginning of Section 3, all $\phi$ functions should be $\Phi$.
   * When referring to a figure, the authors sometimes use "figure #" instead of "Figure #" 
   * In equation (3), $c_j$ terms should be boldface
clarity,_quality,_novelty_and_reproducibility: Clarity and quality is acceptable. No code is available (yet) — reproducibility not assessable. I recommend to use anonymised git repositories for future submissions.
summary_of_the_review: The authors motivate their case and present extensive results. The aim of the proposed approach is described well but there are important missing details. The performance and results of the approach are discussed extensively. The paper could benefit from a more thorough discussion of related work on symbolic models and limitations of the proposed method. There are several typos in the manuscript.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for making a pre-trained NN model more interpretable by applying an iterative process of distilling an interpretable component from the model through training, hence splitting the model into an interpretable part and a residual black-box, and applying this process to the black box until no further distillation is possible. The reviewers agree that the paper is well written and the performance of the approach is evaluated well through extensive experiments. Saying this, the reviewers agree that the paper is missing baseline comparisons with other relevant work in this area. Even though the authors have argued that their work is uniquely placed between post-hoc interpretability methods and interpretable-by-design approaches, and hence cannot be meaningfully compared to any existing baselines from the respective literature, the reviewers remained unconvinced and still wanted to see better positioning of this work in the context of prior work. 
justification_for_why_not_higher_score: The reviewers remain unconvinced about the value and novelty of this approach in the context of prior work despite a lengthy discussion with the authors
justification_for_why_not_lower_score: NA
Decision: Reject