Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes ProtoPNets, a method to debug models at the concept level.
strength_and_weaknesses: Strengths
- The paper makes good progress on debugging models using concepts. The authors write well: this makes it quite straightforward to see how this paper varies from existing work. The contributions are thus clear.
- The layperson validation of the method is quite convincing and shows the utility of ProtoPDebug.
- Figure 3, which should be Algorithm 1(?), is straightforward to implement

Weaknesses
- Please explain what is meant by "θ' is consistent with F" in Equation 10
- It would be nice to make the the debugging process iterative (instead of one off to build up V or F); e.g., how many examples does it take before no debugging is needed? How much does debugging vary between participants
clarity,_quality,_novelty_and_reproducibility: Clarity: Adequate
Quality: High
Novelty: Adequate
Reproducibility: Adequate.
summary_of_the_review: This is a strong paper that proposes an effective concept-level debugger that the authors validate not only using quantitative experiments but also using human-subject experiments. I foresee this paper garnering some attention for its simplicity and clear validation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a concept-level debugger for ProtoPNets in which a human supervisor, guided by the model’s explanations, supplies feedback in the form of what part-prototypes must be forgotten or kept. 
strength_and_weaknesses: + Interesting idea of iterative explanation
- Missing larger scale end-user evaluation  
clarity,_quality,_novelty_and_reproducibility: + clear paper description
+ incremental novelty but crucial for moving this area forward
summary_of_the_review: Interesting paper presenting an approach to refine explanation through user interaction. I really think this approach has some added value, and is an interesting way of tackling (incrementally) the problem of eplainability.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors proposed a concept-level debugger, called ProtoPDebug, for prototypical part networks (ProtoPNets). ProtoPDebug addresses the issue of confounding prototypes learned by a ProtoPNet, by performing human-in-the-loop model debugging. In each debugging iteration, a human expert can mark a prototype as a "confounder" and add the corresponding highly activated region from the prototype's source image to a set of "forbidden concepts" F. Optionally, a human expert can also mark a prototype as a "high-quality" prototype and add the corresponding highly activated region from the prototype's source image to a set of "valid concepts" V. ProtoPDebug can then fine-tune the network by minimizing a loss that encourages the new parameters to be close to the existing ones, and encourages the new prototypes to not activate on the set of forbidden concepts and to activate on the set of valid concepts. The key benefit of ProtoPDebug (over an instance-level debugger such as the IAIA-BL) is that ProtoPDebug does not require fine-grained annotations on individual input instances at all, and is therefore easy to use.
strength_and_weaknesses: Strengths:

- Novelty: The concept-level debugger, ProtoPDebug, is a very novel idea. It allows us to interactively debug a ProtoPNet, without requiring instance-level fine-grained annotations.
- Technical soundness: ProtoPDebug is in general technically sound. The idea of fine-tuning the network using a loss that discourages new prototypes to activate on forbidden concepts and encourages them to activate on valid concepts is very neat.
- Writing: The paper is clearly written and easy to follow.

Weaknesses:

- Minor issue: I have a minor issue with equation (11). In particular, equation (11) introduces a re-ordering on part prototypes of the updated model. Could you clarify what is meant by "re-ordering the part-prototypes"? Is \pi a permutation of the part prototypes of the updated model? If so, shouldn't the corresponding weight entries in w' be re-ordered as well? Please explain.
clarity,_quality,_novelty_and_reproducibility: This work is strong in terms of clarity, quality, novelty, and reproducibility.
summary_of_the_review: Based on the strengths of this paper, I recommend to accept the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a novel interactive learning "debugger" for ProtoPNets. Contrary to the existing IAIA-BL debugger that uses input level annotations (in the form of masks of irrelevant pixels), ProtoPDebug moves towards concept level interaction by storing forbidden and valid concepts in the embedding space, which permits easier transfer of the feedback between inputs. The benefits of this approach are experimentally shown on two different domains: bird classification and COVID diagnostic from chest radiographies. Part of the study is truly interactive, either using a public platform or internally to the authors' research lab. 
strength_and_weaknesses: This is a well-scoped contribution which is very clearly presented. I find the experimental results particularly convincing of the benefits of ProtoPDebug, especially Figure 4 right which shows a clear gap in terms of generalisation compared to all alternative approaches on the birds dataset. I was able to clearly follow the methodology leading to these results.

I however have some concerns about the main claim that ProtoPDebug is a *"concept-level debugger"*. ProtoPDebug certainly departs from prior work by operating in the embedding space instead of remembering pixel-level masks, but there still seems to be a very strong visual prior throughout the paper:
- I find the user questions (Appendix H) to be heavily biased: *"Your task will be to carefully inspect each image and then to determine whether the highlighted overlay covers some part of the bird or whether, instead, it covers exclusively (or very nearly so) the background."*. This question basically assumes that the confounding factors are already known. To me, this phrasing essentially masks the underlying problem of determining causality, which undermines the claims. It would be better to have a more neutral phrasing, for instance *"Do these patches accurately explain the prediction?"*.
- Section 4 always assume an image structure, such that it is possible to extract a "cut-out", which is then re-embedded into a vector. 

clarity,_quality,_novelty_and_reproducibility: There are some technical choices which I would like to clarify
1. Instead of memorising the re-embedded cut out $x_R$ in Figure 3, why is it not possible to just remember the prototype $p$? If there is high activation, both vectors should be similar.
2. In the constrained distillation loss (11), is it necessary to allow for permutations of the prototypes? I agree that the problem is permutation invariant, but if the weights remain similar, I would expect that the order doesn't change. Here I'm concerned that the enumeration of permutations would create a combinatorial explosion as $k$ increases. 
3. In the losses (12) and (13), have you considered the alternative choice of summing the activations (or their opposite) instead of taking the maximum (rest. minimum)? This would be more in line with prior work which does a sum of squares over pixels, see for instance the proof in Appendix A where you have to upper bound the norm by an infinite norm with the usual $\sqrt{k}$ factor. I'm asking this because it isn't so frequent to see $max$ and $min$ directly differentiated for gradient descent and it seems somewhat inefficient. Maybe it would be possible to at least use soft optima?
4. In the first experiment (5.1) I can't find the information of how the annotation is provided to ProtoPDebug? And which percentage of annotated input does it use?

My knowledge of the related literature is not extensive enough to make a definitive assessment of the originality of the method. Based on the provided references the contribution appears novel.
summary_of_the_review: I find the submission to be a solid and effective contribution. I have some concerns about the generality of the claims but I would still like to recommend acceptance. I'm looking forward to discussion with the authors to clarify some misunderstandings.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a "debugger", which is a method for human experts to provide feedback on model predictions, specifically on what portion of the input is relevant, which is then further used to finetune the model. The reviewers agreed that that this is a creative new contribution in an important area, which is relevant both for work in interpretability and in prototype-based learning.
justification_for_why_not_higher_score: This paper could be considered for an oral.
justification_for_why_not_lower_score: This is a strong paper, with ideas that may be more generally applicable.
Decision: Accept: notable-top-25%