Concept-level Debugging of Part-Prototype Networks
Authors: Andrea Bontempelli
Stefano Teso
Katya Tentori
Fausto Giunchiglia
Andrea Passerini
Venue: ICLR 2023 notable top 25% (2023)
Keywords: explainability
debugging
self-explainable networks
part-prototype networks
concept-based models

Abstract: Part-prototype Networks (ProtoPNets) are concept-based classifiers designed to achieve the same performance as black-box models without compromising transparency. ProtoPNets compute predictions based on similarity to class-specific part-prototypes learned to recognize parts of training examples, making it easy to faithfully determine what examples are responsible for any target prediction and why. However, like other models, they are prone to picking up confounders and shortcuts from the data, thus suffering from compromised prediction accuracy and limited generalization. We propose ProtoPDebug, an effective concept-level debugger for ProtoPNets in which a human supervisor, guided by the model’s explanations, supplies feedback in the form of what part-prototypes must be forgotten or kept, and the model is fine-tuned to align with this supervision. Our experimental evaluation shows that ProtoPDebug outperforms state-of-the-art debuggers for a fraction of the annotation cost. An online experiment with laypeople confirms the simplicity of the feedback requested to the users and the effectiveness of the collected feedback for learning confounder-free part-prototypes. ProtoPDebug is a promising tool for trustworthy interactive learning in critical applications, as suggested by a preliminary evaluation on a medical decision making task.

Published as a conference paper at ICLR 2023
CONCEPT-LEVEL DEBUGGING OF PART-PROTOTYPE NETWORKS
Andrea Bontempelli' Stefano Te Katya Tentori Fausto Giunchiglia Andrea Passerini!
1 DISI 2 CIMeC DIPSCO
University of Trento name. www.ameuuuiin.nn...m.
ABSTRACT
Part-prototype Networks (ProtoPNets) concept-based classifiers designed achieve the same performance as black-box models without compromising trans- parency. ProtoPNets compute predictions based similarity to class-specific part-prototypes learned to recognize parts of training examples, making it easy to faithfully determine what examples are responsible for any target prediction and why. However, like other models, they are prone to picking up confounders and shortcuts from the data, thus suffering from compromised prediction accuracy and limited generalization. We propose ProtoPDebug, an effective concept-level de- bugger for ProtoPNets in which a human supervisor, guided by the model's expla- nations, supplies feedback in the form of what part-prototypes must be forgotten or kept, and the model is fine-tuned to align with this supervision. Our experimen- tal evaluation shows that ProtoPDebug outperforms wate-of-the-ar debuggers for a fraction of the annotation cost. An online experiment with laypeople confirms the simplicity of the feedback requested to the users and the effectiveness of the collected feedback for learning confounder- free part prottpoot..pes. ProtoPDebug is promising tool for trustworthy interactive learning in critical applications, suggested by a preliminary evaluation on medical decision making task.
INTRODUCTION
Part-Prototype Networks (ProtoPNets) are "gray-box" image classifiers that combine the trans- parency of case-based reasoning with the flexibility of black-box neural networks (Chen et al., 2019). They compute predictions by matching the input image with set of learned part-prototypes - i.c. prototypes capturing task-salient elements of the training images, like objects parts thereof - and then making a decision based on the part- part-prototype activations only. What makes ProtoPNets ap- pealing is that, despite performing comparably to more opaque predictors, they explain their own predictions in terms of relevant part-prototypes and of examples that these are sourced from. These by design
explanations
faithful than those extracted by post-hoc approaches (Dom- browski et al., 2019; Teso, 2019; Lakkaraju & Bastani, 2020; Sixt et al., 2020) and can effectively help stakeholders to simulate and anticipate the model's reasoning (Hase & Bansal, 2020).
Despite all these advantages, ProtoPNets are prone - like regular neural networks to picking up confounders from the training data (e.g., class-correlated watermarks), thus suffering from compro- mised generalization and out-of -i-ttri-biiiitiion performance (Lapuschkin et al., 2019; Geirhos et al., 2020). This occurs even with it can impact high-stakes applications like COVID-19 diagnosis (DeGrave et al., 2021) and scientific analysis (Schramowski et al., 2020).
data sets, as we will show, and is especially alarming as
We tackle this issue by introducing ProtoPDebug, simple but effective interactive debugger for ProtoPNets that leverages their case-based nature. ProtoPDebug builds on three key observations: (i) In ProtoPNets, confounders - for instance, textual meta-data in X-ray lung scans (DeGrave al., 2021) and irrelevant patches of background sky or foliage (Xiao et al., 2020) - end up appearing as part-prototypes; (ii) Sufficiently expert and motivated users easily indicate which protooypess are confounded by inspecting the model's 's explanations; (iii) Concept-level feedback of this kind is context-independent, and as such it generalizes across instances.

---

Published as a conference paper at ICLR 2023
Goldfinch
Train Image
Prediction
don't
looks like
Dense
confounded
C. kukler
Auklet
fine tune w/ ProtoPDebug
closs Ø
class
is gond
looks like
Embed
L. Albatross
L. Albatroas
Figure I: Left: architecture of ProtoPNets. Right: schematic illustration of the ProtoPDebug loop. The model has acquired a confounded part-prototype (the blue square " "") that correlates with, but is not truly causal for, the Crested Auklet class, and hence mispredicts both unconfounded images of this class and confounded images of other classes (top row). Upon inspection, an end-user forbids the model learn part-prototypes similar p. achieving improved generalization (bottom row). Relevance of all part-prototypes is omitted for readability but assumed positive.
In short, ProtoPDebug leverages the explanations naturally output by ProtoPNets to acquire concept-level feedback about confounded (and optionally high-quality) part-prototypes, as illus- trated in Fig. Then, it aligns the model using a novel pair of losses that penalize part-prototypes for behaving similarly to confounded concepts, while encouraging the model to remember high- quality concepts, if any. ProtoPDebug is ideally suited for human-in-the-loop explanation-base debugging (Kulesza et al., 2015; Teso & Kersting, 2019), and achieves substantial savings in terms of annotation cost compared to alternatives based on input-level feedback (Barnett et al., 2021). In fact, in contrast the per-pixel relevance masks used by other debugging strategies (Ross et al., 2017: Teso & Kersting, 2019; Plumb et al., 2020; Barnett et al., 2021), concept-level feedback au- tomatically generalizes across instances, thus speeding up convergence and preventing relapse. Our experiments show that ProtoPDebug is effective at correcting existing bugs and at preventing new ones both synthetic and real-world data, and that it needs less corrective supervision do than state-of-the-art alternatives.
Summarizing, we: (1) Highlight limitations of existing debuggers for black-box models and ProtoPNets; (2) Introduce ProtoPDebug, simple but effective strategy for debugging ProtoPNets that drives the model away from using confounded concepts and prevents forgetting well-behaved concepts; (3) Present an extensive empirical evaluation showcasing the potential of ProtoPDebug on both synthetic and real-world data sets.
Contributions.
PART-PROTOTYPE NETWORKS
ProtoPNets (Chen et al., 2019) classify images into one of classes using three-stage process comprising an embedding stage, part-prototype stage, and aggregation stage; see Fig. (left).
Embedding stage: Let x be an image of shape w x h x d, where d is the number of channels. The embedding stage passes x through a sequence of (usually pre-trained) convolutional and pooling layers with parameters é, obtaining a latent representation = h(x) of shape w' x h x d', where d' subtensors of z. Each such subtensor
w' and W' < h. Let Q(z) the set of E ((z) encodes filter in latent space and maps rectangular region of the input image x.
Part-prototype stage: This stage memorizes and uses k part- prototypes P ( (P....., ..). J. Each p, is a tensor shape x explicitly learned as explained below
to capture salient vi- sual concepts appearing in the training images, like heads or wings. The activation of a prottprot p part q E Q(z) is given by cettrrence-offlogrriie function, defined as (Chen et al., 2019):
act(p, q) log(Ip qll log(
ql +
(1) Figure 2: Part-prototype
activation functions.
2

---

Published as a conference paper at ICLR 2023
Here, ||-|| indicates the L2 norm and c > is small constant. Alternatively, employ an exponential activation, defined as (Hase & Bansal, 2020):
act(p, q) = exp(->/lp qll²)
(2)
Both activation functions are bell-shaped and decrease monotonically with the distance between q and p. illustrated in Fig. 2. The image-wide activation of p is obtained via max-pooling, i.e., (ct z) == max (()) act (ct(p. q). Given an input image x, this stage outputs an activation vector that captures how much each part-prototype activates on the image:
a (xx) == (act(p, h(x)) E P) IR*
(3)
The activation vector essentially encodes the image according to the concept vocabulary. Each class y [w] := {1....,e) e. is assigned [ discriminative part-prototypes pV C P learned to strongly activate (only) on training examples of that class, as described below.
Aggregation stage: In this stage, the network computes the score of class y by aggregating the concept activations using a dense layer, i.e., Ej wnaa (x), where w E IR* is the weight vector of class y, and applies a softmax function to obtain conditional probabilities poly x). The set of parameters appearing in the ProtoPNet will be denoted by 9.
Loss and training. The network is fit by minimizing a compound empirical loss over a data set D:
1
{() == D (x.v)ED log x) + AeEela (0) + (0)
(4)
This comprises a cross-entropy loss (first term) and two regularization terms that encourage the part-prototypes to cluster the training set in discriminative manner.
(0) 1 D
Σ
min
|lp pl (5) Eeop (0) = D
q1l2
min PEP/PV
1lp-9li* (6)
PEP (x.9) QEQ()(x))
(x.y) qe (h()))
Specifically, the part-prototypes are driven by the clustering loss &ele to cover all examples of their associated class and by the separation loss Seep not to activate on examples of other classes.
During training, the embedding and part- prototpoe layers are fit jointly, so as to encourage the data be embedded in a way that facilitates clustering. At this time, the aggregation weights are fixed: each weight wy of Pj E 'p' is set to if u = y and to -0.5 otherwise. The aggregation layer is fit second step by solving a logistic regression problem. ProtoPNets guarantee the part-prototypes to map concrete cases by periodically projecting them onto the data. Specifically, each e Py is replaced with the (embedding of the) closest image part from any training example of class y.
The architecture of ProtoPNets makes it straightforward to extract explanations high- lighting. for each part-prototype p: (1) its relevance for the target decision (x, y). given by the score wya, (x); (2) its attribution map attr(p. (p, x), obtained by measuring the activation of p each part of x, and then upscaling the resulting matrix to w x using bilinear filtering:
Explanations.
attr x) == upscale([act(p. ()lpetpppp))))))))) 4))q)
(3) the source example that it projects onto.
3 INPUT-LEVEL DEBUGGING STRATEGIES AND THEIR LIMITATIONS
Explanations excel at exposing confounders picked up by models from data (Lapuschkin et al., 2019; Geirhos al., 2020), hence constraining or supervising them can effectively dissuade the model from acquiring those confounders. This observation lies at the heart of recent approaches for debugging machine learning models, see (Teso et al., 2022) for an overview.
A general recipe. The general strategy is well illustrated by the right for the right reasons (RRR) loss (Ross et al., 2017), which aligns the attribution maps produced by predictor to those supplied + y be differentiable classifier and IG(F, (x, y)) E Rd be
by human annotator. Let the input gradient of ff for a decision (x, y). This attribution algorithm (Bachrens et al., 2010; Simonyan al., 2014) assigns relevance each æy for the given decision based the magnitude
3

---

Published as a conference paper at ICLR 2023
of the gradient w.r.t 2j of the predicted probability of class y. The RRR loss penalizes for associating non-zero relevance to known irrelevant input variables:
(8) Here, m is the ground-truth attribution mask of example (x, i.e., my is O if the j-th pixel of x is irrelevant for predicting the ground-truth label and 1 otherwise. Other recent approaches follow the same recipe (Rieger et al., 2020; Selvaraju al., 2019; Shao et al.. 2021; Viviano et al., 2021); see Lertvittayakumjorn & Toni (2021) and Friedrich et al. (2022) for an in-depth overview.
(0) [D] (xx..)))n m) @ (x,
Debugging ProtoPNets. A natural strategy dehug ProtoPNet with confounded part-prototype is to delete the latter and fine-tune the remaining part-prototypes P := P {p). This solution however very fragile, nothing prevents the updated network from re-acquiring the same con- founder using one of its remaining part part-prototypes. One alternative is to freeze the part-proute purt-prototypes p' and fine-tune the aggregation stage only, but this also problematie: in the presence of strong confound, the model ends up relying chiefly p. with the remaining part-prototypes P being low quality, not discriminative, and associated to low weights. Fine-tuning the aggregation layer on P then typically yields poor prediction performance.
These issues prompted the development of the only existing debugger for ProtoPNets, IAIA- BL (Barnett et al., 2021). It also fits the above template, in that rather than removing confounded part-prototypes - it penalizes those part-prototypes that activate on pixels annotated as irrelevant. The IAIA-BL BL loss is defined as:
liaia (0)
Atala
TDT (xxm,0)) (Epepy |(1 m) C attr(p, x)|| + Epeetpyllattr(p.a x)]]
(9)
where attr is the ProtoPNet's attribution map (see Eq. (7)) and indicates the element wise product. The first inner summation is analogous to the RRR loss, while the second one encourages the part- prototypes of other classes not to activate at all, thus reinforeing the separation loss in Eq. (6).
Limitations. A critical issue with these approaches is that they are restricted to pixel-level supervi- sion, which is inherently local: an attribution map that distinguishes between object and background in given image does not generally carry over to other images. This entails that, in order to see any benefits for more complex neural nets, a substantial number of examples must be annotated indi- vidually, as shown by experiments (see Fig. 3). This is especially troublesome as ground-truth attribution maps are not cheap to acquire, partly explaining why most data sets in the wild do not come with such annotations.
Put together, these issues make debugging with these approaches cumbersome, especially in human- loop applications where, in every debugging round, the annotator has to supply negligible number of pixel-level annotations.
CONCEPT-LEVEL DEBUGGING WITH PROTOPDEBUG
Our key observation is that in ProtoPNets confounders only influence the output if they recog- nized by the part-prototypes, and that therefore they can be corrected for by leveraging concept-level supervision, which brings a number of benefits. We illustrate the basic intuition with example:
Example: Consider the confounded bird classification task in Fig. I (right). Here, all training images of class Crested Auklet have been marked with a blue square "I", luring the network into relying heavily on this confound, as shown by the misclassified Laysian Albatro image. By inspecting the learned part-prototypes- -ttrotyhesttggtheer with source images that they match strongly, for context a sufficiently expert annotator can easily indicate the ones that activate on confounders, thus naturally providing concept-level feedback.
Concept-level supervision sports several advantages over pixel-level annotations:
1. It is cheap to collect from human annotators, using an appropriate UI, by showing part-prototype activations on selected images and acquiring blick-based feedback.
2. It is very informative, as it distinguishes between content and context and thus generalizes across instances. In our example, the blue square is a nuisance regardless of what image it appears in. In this sense, one concept-level annotation is equivalent to several input-level annotations.
3. By generalizing beyond individual images, it speeds up convergence and prevents relapse.
4

---

Published as a conference paper at ICLR 2023
The debugging loop. Building these obser- vations, we develop ProtoPDebug, an effective and annotation efficient debugger for ProtoP-
Algorithm A ProtoPDebug debugging session; is a ProtoPNet trained on data set D.
Nets that directly leverages concept-level su- pervision, which describe next. The pseudo-
1: initialize F - 2, + Ø
2: while True do
code for ProtoPDebug is listed in Algorithm 1. ProtoPDebug takes ProtoPNet f and its train- ing set D and runs for variable number of de- bugging rounds. In each round, iterates over all learned part-prototypes p E P, and for each of them retrieves the a training examples (x, y) that it activates the most on, including its source
3: for p E P do
4: for each (x,y) of the a training exam- ples most activated by p do if p looks confounded then add cut-out X to F else p looks high-quality then
5:
6:
7:
8:
add cut-out x to V if no confounders found then break
example. then asks the user to judge p by
9:
inspecting its attribution map each selected
10:
11: fine-tune S by minimizing E(O) +
example. If the user indicates that a particular
activation of looks confounded, ProtoPDe- bug extracts a "cut-out" XR of the confounder from the source image x, defined as the box (or boxes, in case of disconnected activation areas) containing 95% of the part- activation. It then embeds XR using h and adds to a set of forbidden concepts F. This set organized into class-specific subsets Fy, and the user can specify whether should be added to the forbidden concepts for a specific class y or to all of them (class-agnostic confounder). Conversely, if p looks particularly high-quality to the user, ProtoPDebug embeds its cut-out and adds the result to a set of valid concepts V. This is especially important when confounders are ubiquitous, making it hard for the ProtoPNet to identify (and remember) non-confounded prototypes (see COVID experiments in Section 5.2). Importantly, we store the raw cut-out X rather than its embedding because the latter becomes obsolete upon fine-tuning.
Xe( (0) return
12:
Once all part-prototypes have been inspected, f is updated so as to steer it away from the forbidden concepts in F while alleviating forgetting of the valid concepts in V using the procedure described below. Then, another debugging round begins. The procedure terminates when no confounder is identified by the user.
Fine-tuning the network. During fine-tuning, we search for updated parameters H' = (d', w'} that are as close as possible to 0- thus retaining all useful information that f has extracted from the data while avoiding the bugs indicated by the annotator. This can be formalized as a constrained distillation problem (Gou al., 2021):
argming d(0', 0)2 s.t. 8' activates little or not at all on cut-outs in F
(10)
where d is an appropriate distance function between sets of parameters. Since the order of part- prototypes is irrelevant, we define it as a permutation-invarian Euclidean distance:
d(0,0') = ||ø o' d1112 2 min, Σjeng lIp, 12+
(11)
Here, simply reorders the part- part-prototypes of the buggy and updated models so as to maximize their alignment. Recall that we are interested in correcting the concepts, so we focus on the middle term. Notice that the logarithmic and exponential activation functions in Eqs. (1) and (2) are both inversely proportional to lp p'l2 and achieve their maximum when the distance zero, hence minimizing the distance is analogous to maximizing the activation. This motivates us to introduce two new penalty terms:
for (0)
Σ max (ct(p, PEP"
(12)
frem (0)
1 Σ min act(p, v) pep,
(13)
ME TEF,
DE(N] vEV,
The hard constraint in Eq. (10) complicates optimization, so we replace it with smoother penalty function, obtaining a relaxed formulation argming d(0, 0')² + Eior (0). The forgetting loss toor minimizes how much the part-prototypes of each class y E [v] activate on the most activated concept to be forgotten for that class, written Fy. Conversely, the remembering loss frem maximizes how much the part-prototypes activate on the least activated concept to be remembered for that class, denoted Vy, The overall loss used by ProtoPDebug for fine-tuning the model is then a weighted combination of the ProtoPNet loss in Eq. (4) and the two new losses, namely {(0) Artior (0) +
5

---

Published as a conference paper at ICLR 2023
2.00
1.0
ResurDbbog
W.40.%%
0.73
0.95
0.70
0.8
0.65
0.
0.00
0.75
heofbeeug
140 0L 5%
Preid
IMADL 51
heafiee
1005
Presefle
training spochs
training epochs
training opocto
Figure 3: Comparison between ProtoPDebug (in red), ProtoPNets confounded data (in black), ProtoPNets on unconfounded data (green), and IAIA-BL with varying amount of attribution-map supervision (shades of blue) on the CUB5 data set. Left to right: micro Fi on the training set, cross-entropy loss on the training set, and F1 on the test set. Bars indicate std. error over 17 runs.
Xeerem (0), where Xr and X, are hyper-parameters. We analyze the relationship between Eqs. (12) and (13) and other losses used in the literature in Appendix A.
Benefits and limitations. The key feature of ProtoPDebug that leverages concept-level supervi- sion, which sports improved generalization across instances, cutting annotation costs and facilitating interactive debuggiig. ProtoPDebug naturally accommodates concept-level feedback that applies to specific classes. E.g., snow being useful for some classes (say, winter) but irrelevant for oth- ers (like dog or wo w) (Ribeiro et al., 2016). However, ProtoPDebug makes it easy to penalize part-prototypes of all classes for activating on class-agnostic confounders like, e.g., image artifacts. ProtoPDebug's is two losses bring additional, non-obvious benefits. The remembering loss helps to prevent catastrophic forgetting during sequential debugging sessions, while the forgetting loss pre- vents the model from re-learning the same confounder in the future.
One source of concern is that, if the concepts acquired by the model are not understandable, de- bugging may become challenging. ProtoPNets already address this issue through the projection step. but following Koh et al. (2020); Marconato al. (2022), one could guide the network toward learning a set of desirable concepts by supplying additional concept-level supervision and appro- priate regularization terms. ProtoPDebug focuses confounds that can be visually identified in part-prototypes, as this can facilitate interaction with the user. Confounds that do not affect the part- prototypes might also occur, but they are also less likely to affect the predictions. More generally, like other explanation-baset methods, in sensitive prediction tasks ProtoPDebug poses the risk of leaking private information. Moreover, malicious annotators may supply adversarial supervision, corrupting the model. These issues can be avoided by restricting access to ProtoPDebug to trusted annotators only.
EMPIRICAL ANALYSIS
In this section, we report results showing how concept-level supervision enables ProtoPDebug to debug ProtoPNets better than the state-of-the-ar debugger IAIA-BL and using less and cheaper su- pervision, and that it is effective in both synthetic and real-world debugging tasks. ProtoPDebug and IAIA-BL were implemented on top of ProtoPNets (Chen et al., 2019) using the Adam opti- mizer (Kingma & Ba, 2014).
5.1 CONCEPT-LEVEL VS INSTANCE-LEVEL DEBUGGING
We first compare the effectiveness of concept-level and instance-level supervision in controlled scenario where the role of the confounders is substantial and precisely defined at design stage. To this end, we modified the CUB200 data set (Wah et al., 2011), which has been extensively used for evaluating ProtoPNets (Chen et al., 2019; Hase & Bansal, 2020; Rymarczyk et al., 2021; Hoffmann et al., 2021; Nauta et al., 2021). This data set contains images of 200 bird species in natural environ- ments, with approximately 40 train and 20 test images per class. We selected the first five classes, and artificially injected simple confounders in the training images for three of them. The confounder is colored square of fixed size, and it is placed at random position. The color of the square is different for different classes, but it is absent from the test set, and thus acts as perfect confounder for these three classes. The resulting synthetic dataset is denoted CUB5

---

Published as a conference paper at ICLR 2023
Competitors and implementation details. We compare ProtoPDebug, with supervision on single instance per confounder, against the following competitors: 1) vamilla ProtoPNets; 2) a vanilla Pro- toPNet trained a clean version of the dataset without the confounders, denoted ProtoPNetS.deans IAIA-BBL X, the IAIA-BL. model fit using ground-truth attribution maps on X% of the training ex- amples, for X E (5, 20, 100), and IAIA-BL n = 3. with one attribution mask per confound (same number given to ProtoPDebug). Note that This setting is not interactive, but supervision is made available at the beginning of training. The embedding layers were implemented using a pre-trained VGG-16, allocating two prototypes for each class. The values of X and a were set to 100 and 5, respectively, and Aisia to 0.001, as in the original paper (Barnett et al., 2021); experiments with dif- ferent values of Xiaia gave inconsistent results. In this setting, all supervision is made available to all methods from the beginning, and the corrective feedback fed to the forgetting loss is class-specific.
Results. Fig. 3 reports the experimental results on all methods averaged over 15 runs The left and middle plots show the training set macro F1 and cross entropy, respectively. After 15 epochs, all methods manage to achieve close to perfect Fi and similar cross entropy. The right plot shows macro F1 on the test set. As expected, the impact of the confounders is rather disruptive, as can been seen from the difference between ProtoPNets and ProtoPNettscean- Indeed, ProtoPNets end up learning confounders whenever present. Instance-level debugging seems unable to completely fix the problem. With the same amount of supervision ProtoPDebug, IATA-BL. n not manage to improve over the ProtoPNets baseline. Increasing the amount of supervision does improve its performance, but even with supervision on all examples IAIA-BL still fails to match the confound-fire free baseline ProtoPNetS.eean (IAIA-BL 100 curve). On the other hand. ProtoPDebug succeeds in avoiding be fooled by confounders, reaching the performance of ProtoPNets despite only receiving fraction of the supervision of the IAIA-BL alternatives. Note however that ProtoPDebug does occasionally select natural confounder (the water), that is not listed in the set of forbidden ones. In the following section we show how to deal with unknown confounders via the full interactive process of ProtoPDebug.
does
5.2 PROTOPDEBUG IN THE REAL WORLD
CUBS not data set. Next, we evaluate ProtoPDebug in a real world setting in which confounders occur naturally in the data as, e.g., background patches of sky or sea, and emerge at different stages of training, possibly as a consequence of previous user corrections. To maximize the impact of the natural confounders, we selected the 20 CUB200 classes with largest test F1 difference when training ProtoPNets only on birds (removing background) or on entire images (i.e., bird + background), and testing them on images where the background has been shuffled among classes. Then, we applied ProtoPDebug focusing on the five most confounded classes out of these 20. We call this dataset CUB All architectures and hyperparameters are as before.
Experimental Protocol.
To evaluate the usability and effectiveness of ProtoPDebug with real users, we ran an online experiment, through Prolific Academic pprolific.com). one of the most reliable crowdsourcing platforms for behavioral research. Participants were laypeople, all native speakers of English. To make the experimental stimuli as clear as possible, separate patches for the activation area of the prototype on an image were automatically extracted from the 5 most-activate images (a = 5). See Appendix H for the detailed procedure. For each debugging round, a new group of 10 participants was asked to inspect the images and to determine whether the highlighted patch covered some part of the bird or whether, instead, it covered exclusively (or very nearly so) the background. A patch was flagged as a confounder when 80% of the users indicated it as covering the background. A prototype was accepted when no confounder was identified in its most activated image. Regardless of this, confounders in less activated images of the prototype were still collected as they might serve as feedback for other prototypes (class-agnostic confounders) in possible following iteration(s). The procedure ended when all prototypes were accepted.
Results. The average evaluation time for each image was 6 seconds. Across all images, participants' inter-annotation agreement was 96%, confirming the simplicity of the users' feedback required by ProtoPDebug. Fig. 4 shows the part-prototypes progressively learned by ProtoPDebug. In the first iteration, when no corrective feedback has been provided yet, the model learns confounders for most classes: the branches for the second and the last classes, the sky for the third one. Upon receiving user feedback, ProtoPDebug does manage to avoid learning the same confounders again in most cases, but it needs to face novel confounders that also correlate with the class (i.e., the sea for the
7

---

Published as a conference paper at ICLR 2023
1st ROUND
2ND ROUND
3ND ROUND
ProtoPNet
CLASS
Fj
AP;
AP2
0.48
0.75
0.75
0.38
0.73
0.48
14
0.51
0.79
0.79
0.77
0.86
Avg.
0.56
0.68
ProtoPDebug
CLASS
Fj
A/'
./7
0.54
0.84
0.84
0.64
0.94
0.95
14
0.52
0.95
0.95
Avg.
0.69
0.90
Figure Left: Three rounds of sequential debugging with ProtoPDebug on CUB5 Rows: part-prototypes (two per class) and user feedback (checkmark vs. cross). Note that the prototypes produced before the first correction (left-most column) correspond those learned by plain Pro- toPNets. Right:
Test set performance of ProtoPNets and ProtoPDebug on CUB5 AP; is the attribution precision i-th part-prototype. Bold denotes better performance.
first class). After rounds, all prototypes are accepted, and the procedure ends. Fig. reports the test set performance of ProtoPNet compared to ProtoPDebug in terms of F1 and interpretability metric. The latter has been introduced in (Barnett al., 2021, Eq. 6) quantify the portion of pixels that are activated by a prototype. We slightly modified it to consider also their activation value (see Suppl. for the definition). Results show that ProtoPDebug improves over ProtoPNets in all classes (with an increase in F1 of up to 0.26 for classes 6 and 8), and that it manages to learn substantially more interpretable prototypes (i.c., is right for the right reason (Schramowski et al., 2020)), with 22% improvement in activation precision average.
COVID data set. The potential artificial intelligence and especially deep learning medicine huge (EJ., 2019). On the other hand, the medical domain is known to be badly affected by the pres- ence confounders (Jager et al., 2008; Smith & Nichols, 2018). In order to evaluate the potential of ProtoPDehug help addressing this issue, tested it challenging real- wolld problem from the medical imaging domain. The task to recognize COVID-19 from chest radiographies. As shown DeGrave (2021), classifier trained on this dataset heavily relies on confounders that corre- late with the presence or absence of COVID. These confounders come from the image acquisition procedure annotations the image. We trained and tested ProtoPDebug on the same datasets used in DeGrave et al. (2021) using the data pipeline source code (cod, 2021a). a). To simplify the iden- tification of confounders, we focused on binary classification task, discriminating COVID -oosiive images from images without any pathology. Given the extremely high inter-annotator agreement of the CUB5 nat experiment, this experiment was conducted internally our lab. We leave crowd- sourced evaluation
extended version of this work.
Results. Fig. reports the (non-zero activation) prototypes of ProtoPDebug at different correction rounds. As for Fig, 4 (left), the left-most column corresponds to the prototypes learned by Pro- toPNets. Note that for each prototype, the supervision given to the 10 most-activated images (a = 10). Penalized confounders have been extracted from images which the prototype has non-zero activation, because they influence the classification. However, the patches of the images remember extracted even if the activation is zero, in order force the prototype increase
8

---

Published as a conference paper at ICLR 2023
1sT ROUND
2ND ROUND
3ND ROUND
4TH ROUND
Figure 5: Four rounds of sequential debugging with ProtoPDebug on COVID. Only the prototypes with non-zero activation reported: first prototype refers to COVID- and second COVID+.
the activation them thanks to the remembering loss. Eventually, ProtoPDebug manages to learn non-confounded prototypes, resulting in substantially improved test classification performance. The test F1 goes from 0.26 of ProtoPNets (first column) to 0.54 the end of the debugging process.
RELATED WORK
Input-level debuggers. ProtoPDehug is inspired by approaches for explanatory debugging (Kulesza al., 2015) and explanatory interactive learning (Teso & Kersting, 2019; Schramowski al., 2020), which inject explanations into interactive learning, enabling a human-in-the-logp to identify bugs in the model's reasoning and fix them by supplying corrective feedback. They leverage input attribu- tions (Teso & Kersting, 2019; Selvaraju et al., 2019; Lertvitayyakujjom et al., 2020; Schramowski al., 2020), example attributions (Teso et al., 2021: Zylberajch et al., 2021), and rules (Popor- danoska et al., 2020), but are not designed for concept-based models nor leverage concept-level explanations. IAIA-BL (Barnett et al., 2021) adapts these strategies to PPNets by penalizing part- prototypes that activate on irrelevant regions of the input, but it is restricted to instance-level feed- back, which is neiter as cheap to obtain nor as effective as concept-level feedback, as shown by our experiments. For overview of interactive debugging strategies, Teso et al. (2022).
Concept-level debuggers. Stammer et al. (2021) debug neuro-symbolic models by enforcing log- ical constraints the concepts attended to by network's slot attention module (Locatello al., 2020), but requires the concept vocabulary to be given and fixed. ProtoPDebug has no such re- quirement, and the idea of injecting prior knowledge (orthogonal to contributions) could be fruitfully integrated with it. ProSeNets (Ming et al., 2019) learn (full) prototypes in the embed- ding space given by sequence encoder and enable users to steer the model by adding, removing and manipulating the learned prototypes. The update step however requires input-level supervision and adapts the embedding layers only. iProto-TREs TREX (Schramowski et al., 2021) combines ProtoP- Nets with transformers, and learns part- part-prototypes capturing task-relevant text snippets akin to ratio- nales (Zaidan al., 2007). supports dynamically removing and ooa-swap bad part-prototypes, but it lacks forgetting loss and hence is prone to relapse. Applying ProtoPDebug to (Proto-TREX straightforward and would fix these issues. Bontempelli et al. (2021) proposed a classification of debugging strategies for concept-based models, but presented experimental evaluation. Lage & Doshi- Velez (2020) acquire concepts by eliciting noncept-attribur dependency information with questions like "does the concept depressi can prevent confounding, but restricted to white-box models and interaction with domain ex- perts. FIND (Lertvitayakumjorn et al., 2020) offers similar functionality for deep NLP models, but it relies on disabling concepts only.
depend on feature 1 lorazepam?". This approach
Other concept-based models. There exist several concept-based models (CBMs) besides ProtoP- Nets, including self-explainable neural networks (Alvarez-Melis & Jaakkola, 2018), concept bottle- neck models (Koh et al., 2020; Losch et al., 2019), and concept whitening (Chen al., 2020). Like ProtoPNets, these models stack a simulatable classifier on top of pnon-proooyypcca) interpretable concepts. Closer to our setting, prototype classification networks (Li et al., 2018) and deep embed- ded prototype networks (Davoudi & Komeili, 2021) make predictions based on embedding prototypes of full training examples (rather than parts thereof). Our work analyzes ProtoPNets as they perform comparably other CBMs while sporting improved interpretability (Chen et al., 2019; Hase & Bansal, 2020). Several extensions of ProtoPNets have also been proposed (Hase et al., 2019; Rymarczyk et al., 2021; Nauta et al., 2021; Kraft et al., 2021). ProtoPDebug naturally applies to these variants, and could be extended to other CBMs by adapting the source example selection and update steps. An evaluation these extensions is left future work.

---

Published as a conference paper at ICLR 2023
Ethics statement ProtoPDebug is conceived for leveraging interactive human feedback to improve the trustworthiness and reliability of machine predictions, which is especially important for critical applications like medical decision making. On the other hand, adversarial annotators might supply corrupted supervisions to alter the model and drive its predictions towards malignant goals. For these reasons, access the model and to the explanations should be restricted to authorized annotators only, as standard in safety-critical applications. Concerning the online experiment with laypeople, all participants were least 18 years old and provided informed consent prior to participating the research activities.
Reproducibility statement All experiments were implemented in Python using Pytorch (Paszke et al., 2019) and run machine with two Quadro RTX 5000 GPUs. Each run requires in-between and 30 minutes. We implemented ProtoPDebug and IAIA-BL on top ProtoPNets and we used the source code of the respective authors of the two methods (cod, 2019; 2021b). The full experi- mental setup is published on GitHub at ht tttps:: : //ggiituuui com/ abont e prot opdebug and in the Supplementary Material together with additional implementation details. The repository tains the configuration files reporting the hyper-parameterr' of all experiments, and the procedures to process the data and reproduce the experiments. The implementation details are also available in Appendix C, while Appendix D informs about the data sets, which are all made freely available by their respective authors. For CUB200 (Wah al., 2011) and image masks (Farrell, 2022), the image me-processing reuses the code of the ProtoPNets authors (cod, 2019). The COVID images are processed using (cod, 2021a). The images are downloaded from GitHub-COVID repository (Cohen et al., 2020), ChestX-ray 14 repository (Wang et al., 2017), PadChest (Bustos et al., 2020) and BIMCV-COVIDI9+ (Vayá et al., 2020). Appendix H provides the experimental protocol for the online experiment with real users.
ACKNOWLEDGMENTS
This research of FG has received funding from the European Union's Horizon 2020 FET Proactive project "WeNet The Internet of us", grant agreement No. 823783, The research of AB was sup- ported by the "DELPhi Diisov--iig Life Patterns" project funded by the MIUR Progetti di Ricerca di Rilevante Interesse Nazionale (PRIN) 2017 DD n. 1062 del 31. 05.05.. 2519. The research of ST and AP was partially supported by TAILOR, project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
REFERENCES
Protopnet source code, 2019. URL
htt.:gttttht.......otoeeeeeeeeee http///ccciiennnnk//e/tttttteeee github. com/ wtteeeeelabxxr_cooeeee
Ai for radiographic covid-19 detection selects shortcuts over signal, 2021a. URL https://
laia-bl source code, 2021b. BRLhttps::// httpttttthuuuuh/////////t com/ http.///..iaaaaaaaraetttaaaaaaaa
David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. In NewrIPS, 2018.
David Bachrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus- Robert Müller, How explain individual classification decisions. The Journal of Machine Learn- ing Research, 11:1803 1033-031,,,, 2010.
Alina Jade Barnett, Fides Regina Schwartz, Chaofan Tao, Chaofan Chen, Yinhao Ren, Joseph Y Lo, and Cynthia Rudin. A case-based interpretable deep learning model for classification of mass lesions in digital mammography. Nat. Mach. Intell., 2021.
Andrea Bontempelli, Fausto Giunchiglia, Andrea Passerini, and Stefano Teso. Toward a Unified Framework for Debugging Gray-box Models. In The AAAI-22 Workshop on Interactive Machine Learning. 1.21.
Aurelia Bustos. Antonio Pertusa, Jose-Maria Salinas, and Maria de la Iglesia-Vayy Padch- est: A large chest x-ray image dataset with multi-label annotated reports. Medical Im- age Analysis, 66:101797, 2020. ISSN 1361-8415. doi: htts://doi.org/10. 1016//.medii..
10

---

Published as a conference paper at ICLR 2023
2020.101797. URL http://pwwww......oo..o...././ta $1361841520301614
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32:8930-899 1099930-941,.00
8941, 2019. 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):722-782, 2020.
Joseph Paul Cohen, Paul Morrison, and Lan Dao. Covid-19 image data collec- tion. arXiv 2003.11597. 2020.
URL https://gitaur httpiittuubcooooaeaaoooommooottt com/ 1eee8023/
covid--hesttrray-datassss
Seyed Omid Davoudi and Majid Komeili. Toward faithful case-based reasoning through learning prototypes in a nearest Säighoo-frrendll space. In International Conference on Learning Repre- sentations, 2021.
Alex J DeGrave, Joseph D Janizek, and Su-In Lee. Ai for radiographic covid-19 detection selects shortcuts over signal. Nat. Mach. Intell., 2021.
Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations be manipulated and geometry is to blame. NewrlPS, 2019.
Topol EJ. nigh-peeffrmance medicine: the convergence of human and artificial intelligence. Nar Med. 25(1):44-35 2019.
Farrell. Cub-200-2011
segmentations, apr 2022.
Felix Friedrich, Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting, A typology to explore and guide explanatory interactive machine learning. arXiv preprint arXiv:2203.0366 2022.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2020.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):188) 18199889991191111111111111199981 2021.
Peter Hase and Mohit Bansal. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5540-550 5552,,5522222 2020.
Peter Hase, Chaofan Chen, Oscar Li, and Cynthia Rudin. Interpretable image recognition with hierarchical prototypes. In Proceedings of the AAAI Conference Human Computation and Crowdsourcing, volume 7, pp. 32-40, 2019.
Adrian Hoffmmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks. arXiv preprint arXiv: 1xXiv:2005.0266. 2021.
K.J. Jager, C. Zoccali, A. MacLeod, and F.W. Dekker. Confounding: What it is and how to deal with it. Kidney International, 73(3):256-260, 2008.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2,6990, 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICMI, 2020.
Stefan Kraft, Klaus Broelemann, Andreas Theissler, Gjergji Kasneci, GER Esslingen am Neckar, SCHUFA Holding AG, GER Wiesbaden, and GER Aalen. Sparrow: Semantically coherent pro- totypes for image classification. In The 32nd British Machine Vision Conference, 2021.

---

Published as a conference paper at ICLR 2023
Todd Kulesza, Margaret Burnett, Weng- Keen Wong, and Simone Stumpf. Principles of explanatory debugging to personalize interactive machine learning. In IUI, 2015.
Isaac Lage and Finale Doshi- Velez. Learning interpretable concept- based models with human feed- back. arXiv preprint arXiv:2012.02880 2020.
Himabindu Lakkaraju and Osbert Bastani. "How do fool you?" Manipulating User Trust via Misleading Black Box Explanations. In Proceedings of the AAAI/ACM Conference AI, Ethics, and Society, pp. 79-85, 2020.
Sebastian Lapuschkin, Stephan Waldchen, Alexander Binder, Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Unmasking clever hans predictors and assessing what machines really learn. Nature communications, 2019.
Piyawat Lertvittayakumjom and Francesca Toni. Explanation- human debugging of nlp mod- els: A survey. arXiv preprint arXiv:2104.55135, 2021.
Piyawat Lertvittayakumjonn Lucia Specia, and Francesca Toni. Find: human-in-the-llop debugging deep text classifiers. In EMNLP, 2020.
Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:11525-11538 2020.
Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic Bottleneck Networks. arXiv preprint UrXiv:1907.10882 2019.
Emanuele Marconato, Andrea Passerini, and Stefano Teso Glancenets: Interpretabile, leak-proot concept-based models. In NewrIPS, 2022.
Yao Ming, Panpan Xu, Huamin Qu, and Liu Ren. Interpretable and steerable sequence learning via prototypes. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 903, 2019. 2019
Meike Nauta, Ron Bree, and Christin Seifert. Neural prototype trees for interpretable fine- grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14933-14943, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep lcarning library. Advances in neural information processing systems, 32, 2019.
Gregory Plumb, Maruan Al-Shedivat, Ángel Alexander Cabrera, Adam Perer, Eric Xing, and Ameet Talwalkar. Regularizing black-box models for improved interpretability. In NewrIPS, 2020.
Teodora Popordanoska, Mohit Kumar, and Stefano Teso. Machine Guides, Human Supervises: Interactive Learning with Global Explanations. arXiv preprint arXiv:2009.09723, 200000
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1 144, 2016.
Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In ICMI, 2020.
Andrew Slavin Ross, Michael C Hughes, and Finale Doshi-Velez Right for the right reasons: training differentiable models by constraining their explanations. In IJCAI, 2017.
12

---

Published as a conference paper at ICLR 2023
Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. ProtoPShare: Prototypical Parts Sharing for Similarity Discovery in Interpretable Image Classification. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining, pp. 1430-143,, 2021.
Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaot- ing Shao, Hans-Georg Luigs, Anne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nat. Mach. Intell., 2020.
Patrick Schramowski. Felix Friedrich, Christopher Tauchmann, and Kristian Kersting, Interactively Generating Explanations for Transformer Language Models. arXiv preprint 10.0000,, 2021.
Ramprasaath R Selvaraju, Stefan Lee, Yilin Shen, Hongxia Jin, Shalini Ghosh, Larry Heck, Dhruv Batra, and Devi Parikh. Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded. In ICCV, 2019.
Xiaoting Shao, Arseny Skryagin, P Schramowski, W Stammer, and Kristian Kersting, Right for Better Reasons: Training Differentiable Models by Constraining their Influence Function. In AAAI, 2021.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Vi- sualising image classification models and saliency maps. In International Conference on Learning Representations, 2014.
Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In ICML, 2020.
Stephen M. Smith and Thomas E. Nichols. Statistical challenges in "big data" human neuroimaging. Newron, 97(2):263-268, 2018.
Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting. Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations. In CVPR, 2021.
Stefano Teso Toward faithful explanatory active learning with self-explainable neural nets. In IAL Workshop, 2019.
Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In AIES, 2019.
Stefano Teso, Andrea Bontempelli, Fausto Giunchiglia, and Andrea Passerini. Interactive Label
Cleaning with Example-based Explanations. In NeurlPS, 2021.
Stefano Teso, Öznur Alkan, Wolfang Stammer, and Elizabeth Daly. Leveraging explanations in interactive machine learning: An overview. arXiv preprint arXiv:2207.14526 2022.
Maria de Iglesia Vayá, Jose Manuel Saborit, Joaquim Angel Montell, Antonio Pertusa, Aurelia Bustos, Miguel Cazorla, Joaquin Galant, Xavier Barber, Domingo Orozco-Beltrán Francisco García-García, Marisa Caparrós, Germán González, and Jose María Salinas. Bimcv covid-19+: large annotated dataset of and images from covid-19 19 patients. 2020. doi: 10.48550/ARXIV. 0006,01174, URL https: / arxiv. org/ abs 2006. 01174.
Joseph D Viviano, Becks Simpson, Francis Dutil, Yoshua Bengio, and Joseph Paul Cohen. Saliency is possible red herring when diagnosing poor generalization. In ICLR, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD 200ds-200-2011 Dataset. 2011.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald Summers. Chestx-ray8: -ospital-scale chest x-ray database and benchmarks weakly-supervised classi- fication and localization of common thorax diseases. In 2017 IEEE Conference on Computer Vision and Pattern Recognition(CVPP) pp. 3462-3471, 2017.
13

---

Published as a conference paper at ICLR 2023
Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise signal: The role of image backgrounds object recognition. In International Conference on Learning Rep- resentations, 2020.
Omar Zaidan, Jason Eisner, and Christine Piatko. Using "annotator rationales" to improve machine learning for text categorization. In NAACL, 2007.
Hugo Zylberajch, Piyawat Lerivittaakkujoom and Francesca Toni. HILDIF: Interactive Debug- ging of NLI Models Using Influence Functions. Workshop on Interactive Learning for Natural Language Processing, 2021.
A RELATIONSHIP TO OTHER LOSSES
The forgetting loss in Eq. (12) can be viewed as a lifted version of the separation loss in Eq. (6): the former runs over classes and leverages per-class concept feedback, while the latter runs over examples. The same analogy holds for the remembering and clustering losses, cf. Eqs. (5) and (13). The other difference is that the instance-level losses proportional to the negative squared distance -Ilp all between protoopots and parts and the concept-level ones to the activation instead. However, existing activation functions ing that they only really differ in fall-off speed, as can be seen in Fig. 2. The IAIA-BL loss can also be viewed (lower bound of the) separation loss. Indeed:
monotonically decreasing in the squared distance, mean-
Linia( (0) = Σ
Σ (x.m,)) PEPS
II(1- m) @ attr x||| S V
Σ
max DEPY
[[ - m) O attr(p, x) loc
(x.
=V Σ max pEPS (x.m.y) qEQ(N(x))
((L-Tg) act(pp Vk
Σ
min (1 mg) ||p qll
p.q
(x.m.y)
where mg 2 is the "part-level" attribution mask for part q. obtained by down-scaling the input- level attribution mask of all pixels in the receptive field of q. This formulation highlights similari- ties and differences with our forgetting loss: the latter imposes similar constraints as the IAIA-BL penalty, but (i) it does not require full mask supervision, and (ii) it can easily accommodate concept- level feedback that targets entire sets of examples, as explained above.
B ACTIVATION PRECISION
In order to measure the quality of the explanations produced by the various models, we adapted the activation precision (AP) metric from the original IAIA-BL experiments (Barnett et al., 2021, Eq- 6). Assuming that the attribution mask attr output by ProtoPNets is flattened to a vector with the same shape m, the original formulation of AP be written as:
[m O T- (attr(p, x))];
Σ
(14)
D
Tatr x));
Here, T, is a threshold function that takes an attribution map (i.e., a real-valued matrix), computes the (100- -T) percentile of the attribution values, and sets the elements above this threshold to 1 and the rest to 0. Intuitively, the AP counts how many pixels predicted as relevant by the network are actually relevant according the ground-truth attribution mask
This definition however ignores differences in activation across the most relevant pixels, which be quite substantial, penalizing debugging strategies that manage correctly shift most of the network's s explanations over the truly relevant regions. To fix this, we modified T; not to discard this information, as follows:
EEim @ (atttr(pp,)) O attr(p.))]: E,T. x)) O attr(p, x)];
Σ
(15)
D
(x.D,))
In our experiments, we set 7 = 5%. The same value is used by IAIA-BL BL and ProtoPNets for visualizing the model's explanations and is therefore what the user would see during interaction.
14

---

Published as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
Architectures.
We use the same ProtoPNet architecture for all competitors:
Embedding step: For the CUB experiments, the embedding layers were taken from pre-trained VGG-16, as (Chen et al., 2019), and for COVID from a VGG-19 (Simonyan & Zisserman, 2014).
Adaptation layers: As in (Chen et al., 2019), two 1 x convolutional layers follow the embedding layers, and use ReLu and sigmoid activation functions respectively. The output dimension is 7x7.
Part-prototype stage: All models allocate exactly two part-prototyps per class, as allocating more did not bring any benefits. The prototypes are tensors of shape x x 128.
To stabilize the clustering and separation losses, we implemented two "robust" variants proposed by Barnett et al. (2021), which are obtained by modifying the regular losses in Eqs. and of the main text to consider the average distance the closest 3 part-prototypes. rather than the distance to the closest one only. These two changes improved the performance of all competitors.
Training. We froze the embedding layer prototypes. Similarly to the ProtoPNet implementation, the learning rate of the prototype layer was scaled by 0.15 every epochs. The training batch size is set to 20 for the experiments CUB5boz and COVID data sets, and 128 on CUB5
drive the model toward updating the part-
We noticed that, in confounded tasks, projecting the part-prototypes onto the nearest latent training patch tends the prototypes closer to confounders, encouraging the model to rely on it. This strongly biases the results against the ProtoPNet baseline. To avoid this problem, in our experiments disabled the projection step.
Hyper- pappr-pprammters... The width of the activation function (Eq. in the main paper) used foo was set to e = 10-$ in all experiments.
For CUB5 and CUB5 the weights of the different losses were set to Aete 0.5, Aeep 0.08, Afor 100. In this experiment, the remembering loss was not necessary and therefore it was disabled by setting Arem O. The value of Aiaia 0.001 was selected from {0.001, 0.01, 1.0, 100) so as to optimize the F1 test set performance, averaged over three runs. The number of retrieved training examples shown the user (a = 5).
For the COVID data set, Actss Asep Asor and Aram were set to 0.5, 0.08, 200, and 0 respectively for the first two debugging rounds. In the last round, Acls Asop were decreased to 25% of the above values and Arem increased to 0.01, to boost the effect of Etor and frem- This shifts the focus of training from avoiding very bad part- prototyyos,, to adjusting the partly wrong prototypes and increasing the activation the relevant patches. The value of is set to 10.
D DATA STATISTICS AND PREPARATION
Table reports statistics for all data sets used in our experiments, including the number of training and test examples, training examples used for the visualization step, and classes. The visualization data set contains the original images, i.c., before the augmentation step. and is used to visualize the images on which the prototypes activate most during the debugging loop. All images were resized to 224 x 224 3, as done in ProtoPNet (Chen et al., 2019), IAIA-BL (Barnett et al., 2021) and for the COVID-19 data in (DeGrave et al., 2021).
D.1 CUB DATA SETS
Classes in the CUB200 data set include only 30 training examples each. To improve the performance of ProtoPNets, as done in (Chen et al., 2019), we augmented the training images using random rotations, skewing, and shearing, resulting in 900 augmented examples per class.
CUB5 box The five CUB200 classes that were make up the CUB5 data set are listed Fig. (left). Synthetic confounders (i.e., colored boxes) have been added to training images of the first,
15

---

Published as a conference paper at ICLR 2023
Table 1: Statistics for all data used in experiments.
DATA SET
TRAIN
# TEST
VISUALIZATION
# OF CLASSES
CUB5 box
4500
132
150
5
CUB5 nat
180000
569
600
20 2
COVID
7640
1744
7640
Figure 6: From left to right, the images of confounders on which for relies for computing the activation value: the confounding green box present CUB5boz and background patches in CUBSpat-
second and last class. No confounder was added to the other two classes. Fig. (left) shows one of the three colored boxes. An example image confounded with green box "I" is shown in Fig. 7 (right) along with the corresponding ground-truth attribution mask m used for IAIA-BL.
Figure 7: Left: List of CUB200 classes selected for the CUB5boz data set. Right: Confounded training images from CUB5 and corresponding ground-trun truth attribution mask.
CLASS
LABEL
*CNNOUUDD?
0
001
Black-tooted Albatross
I
002
Layean Albatross
2
003
Sooty Albat ross
Grocve-biilse
005
Crested Auklet
We created a variation of CUB200, namely CUB5 to introduce a confounder that occurs naturally, e.g., sky trees patches in the background, and are learned during the training phase. We followed these steps to select the twenty classes reported Table 2 and used for the experiments:
CUB5
1.
We modified the test sel images of all the 200 classes of CUB200 by placing the bird of class on the background of another Since the outline of the bird removed from the background may not match the pasted one, the uncovered pixels are set to random values. Figure Fig. shows the background swap for one image and the corresponding ground- truth mask. This background shift causes a drop in the performance of the model that relies on the background to classify bird.
2. We trained ProtoPNet background, the images contain only the birds, and in the second, both the bird and the background. We selected the twenty classes with the maximum gap in terms of on the modified test set, out of the 200 classes of CUB200. Thus, the background appears to be confounder for these twenty classes. The right most images in Fig. 6 highlight patches of the background learned by the model.
variations of the training set: in the first, by removing the
3. We repeated the step above but only on the twenty classes, and we selected the five classes provided supervision only these five
with the maximum gap. In the experiments, classes. Since the experiments are run on this subset of twenty classes, the separation loss might force the model to learn different prototypes with respect to the ones learned when
16

---

Published as a conference paper at ICLR 2023
Figure 8: From left to right: original image with the sea in the background, the same bird but with the background of the land and the corresponding ground-truth attribution mask.
trained 200 classes. Hence, the prototypes activate different pixels and the classes with the largest gap are different. This additional ranking of the classes allows us to debug the most problematic For this reason, the results reported in the main paper are computed on only these five classes.
and show the potential performance improvement
them.
Table 2: Twenty classes with largest train-test performance gap used in the our second experiment. The debug column indicates which classes received supervision and are thus debugged.
CLASS
Lanm.
Drncn??
CLASS
LARM
Denen?
001
3lack-taoted batrooss
10
085
Horned Lazk
004
Groove-bi
11
086
Eacitic Loon
Leagt Ackict
Baird Sparzow
026
Bronzed Cowatrd
122
Hacris Sparrow
037
Acedian Flyseticher
14
128
Seacide Sparrow
D40
olivo-sided Flycatcher
137
clifr Swallow
Yellox-bellien Flycatcher
Blus-winged
Pine Grasbeek
Norshern Materthrusn
070
Green Violetear
18
188
Pilcated Nocdpcoker
076
Bark-eyed
200
Common Yellewtareat
D.2 COVID DATA SET
The training set consists of subset of images retrieved from the GitHub-COVID repository (Cohen al.. 2020) and from the ChestX-ray 14 repository (Wang al., 2017). The number of COVID- negative and COVID-positive radiographs is 7390 and 250, respectively. The test set is the union of PadChest (Bustos et al., 2020) and BIMCV-COVIDI9. (Vayá et al., 2020) datasets, totalling 1147 negative and 597 positive images. In (DeGrave et al., 2021), the classifier is trained on 15 classes, ie., COVID and other 14 pathologies. In (DeGrave et al., 2021), the classifier trained on a multi- label data set, in which each is associated with multiple pathologies, and the evaluation is then performed on COVID positive vs. other pathologies. To simplify the identification and thus the supervision given to the machine, generated binary classification problem containing COVID- positive images and images without any pathology. In this setting, supervision is given only on obvious confounders like outside the body profile can provide detailed supervision on which parts of the lungs the machine must look at.
arms and elbows. domain expert
E ADDITIONAL RESULTS
Additional part-prototypes for COVID. Fig. 9 reports all prototypes for the experiment on the COVID data
Confusion matrices. The confusion matrix the left side of Fig. 10 shows that ProtoPNet over- predicts the COVID-positive class, whereas the prediction of ProtoPDebug after three rounds is more balanced since the confounders have been debugged. A domain expert give additional supervision on which parts of the lungs are relevant for the classification task, further improving the performance.
17

---

Published as a conference paper at ICLR 2023
Figure 9: Four rounds of sequential debugging with ProtoPDebug COVID. Top reports the prototypes with non-zero activation: first prototype refers to COVID- and second COVID+. Second row: prototypes with zero activation for each class.
1sT RoUND
2km ROUND
3ND ROUND
4TH1 ROUND
COVID-
COVID-
COVID-
COVID=
COVID-
COVID-
COVID-
COVID-
Figure 10: Confusion Matrices test set where class 0 and 1 is COVID-negative and COVID- positive respectively.
1ST ROUND
4TH ROUND
1000
True label
True label
450
350
568
300
200
Predicted label
Predicted label
F COMPARISON TO REMOVING-AND-FINETTNNI
One alternative strategy for debugging ProtoPNets consists of removing confounded part-prototype (or fixing their weight to zero) and then fine-tuning the aggregation layer. Here, highlight some limitations of this strategy that ProtoPDebug does not suffer from. In all experiments we use the same setup in the experiments reported in the main text.
First, this strategy cannot be used when all prototypes are confounded. This can occur because, in ProtoPNets, there is term encouraging the network lear distinct part-prototypes. In practice, it can happen that all part-prototypes learn to recognize the same confound. This is illustrated by the two left images Fig. 11: here a ProtoPNet trained on the CUB5boz data set has learned the same confounder (the colored box) with both of the part-prolse part-prototypes allocated to the target class. In this situation, the above strategy cannot be applied as there are no unconfounded concepts to rely on for prediction.
What happens if only some part-prit pootoopots are confounded? order to answer this question, we assume that the ProtoPNet loss is augmented with diversification term laiv of the form:
Lai (0)
Σ
Σ min P;EP3" J.PEPt
1l;;- 12
(16)
P
where Aaiv a hyper-parameter. This term encourages the different part-pree prototypott of each class to be different from each other, and is useful prevent them all acquire the same confound.
Now, even only some part part-prototype are confounded, there is no guarantee that the unconfounded ones capture meaningful information. The opposite tends to be true: when strong confounder is present, the network can (and typically does) achieve high accuracy on the training set by learning recognize the confounder and allocating all weight it, and has little incentive to learn other
18

---

Published as a conference paper at ICLR 2023
Figure 11: Left pair: prototype activations of a confounded class in CUB5 box Notice that both prototypes are confounded. Right pair: one prototype activates on the confounder (colored box) whereas the other is neither interpretable nor useful for prediction.
discriminative. high-quality part prototpos...ss. We validated this empirically by training ProtoPNet model on the CUB5 data set using the diversification term above, setting Adiv to 0.1. The results are shown in Fig. 11 (right): the two activation maps show that one part-prototype activates on the confound, while the other captures irrelevant information and that therefore it would perform very poorly when used for classification.
G ABLATING THE REMEMBERING LOSS
We evaluate the contribution of the remembering loss in last debugging rounds of the COVID exper- iment. The remembering loss avoids performance drop over time. We re-run the last two debugging rounds of the experiment in Figure 5 by setting A to In terms of Fi, the performance in the third round goes from 0.55 with the remembering loss to 0.44 without. In the fourth round, performance decrease from 0.54 to 0.4.
H USER EXPERIMENT
We evaluated the usability of ProtoPDebug sequence of debugging round with real users. The participants asked to review patches of prototype activation the training images. The patch extraction procedure starts by selecting the five most activated images for each prototype. For each activation map, we generate a binary image in which pixels are at least 35%-perceetie of all activation (as in (Chen et al., 2019)). zero otherwise. Then, we extract the contours representing the border of the activation patches. We discard the patches that small, difficult to be spotted by the participants and not capturing enough information. We set the threshold patches having less than 200 pixels area. Finally, we overlay the original images with the extracted patches.
if their activation value is
We ran the online experiment through Prolific Academic. For each debugging round experiment, ten participants, laypeople and English native speakers, inspected the images generated as described above sequence of questions like Fig. 12. Each experiment took between 9 and minutes according to the number of patches to review. Participants received an hourly fee for their partici- pation. On top of this amount, we gave an additional bonus based on the agreement with the other participants. We randomly selected five images, and awarded the users whose evaluations agrec with at least 80% of others' evaluations for all five images.
We provided the following instruction the user:
You will now be presented with series of images of birds in various natural settings. Each image has highlighted overlay that goes from dark red (the highlight focus) to light green (the highlight margin).
Your task will be carefully inspect each image and then determine whether the highlighted overlay covers some part of the bird or whether, instead, it covers exclusively (or very nearly so) the background.
Table exposes statistics about user feedback over the debugging rounds. The number patches on which the prototypes activate almost exclusively the backgrounds decreases by giving more
19

---

Published as a conference paper at ICLR 2023
Question in the experiment form. The upper part display activation patch. The Figure 12:
user must choose either that the overlay covers part of the bird or exclusively the background.
Figure
The highlighted overlay covers
some part of the bird
exclusively (or very nearly so) the background
20

---

Published as a conference paper at ICLR 2023
Table 3: Statistics about the patches for all debugging rounds. From left right: number of patches overlaying the background with the inter-annotator agreement of at least 80%, number of patches on which the agreement under 80%, number of patches activating on the bird with agreement of at least 80% and the total number of patches shown to the participants.
ROUND
# BACKGROUND
BIRD
# NO AGREEMENT
TOTAL
FIRST
19
73
5
97
SECOND
6
63
5
74
THIRD
73
2
78
supervision to the model. The inter-annotation agreement is 96% and shows that the ProtoPDebug feedback required from the supervisor simple and effective.
21


Reviews:
[{"review_id": "WNO5fsc9jV", "reviewer": "Reviewer_wHVs", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a novel interactive learning \"debugger\" for ProtoPNets. Contrary to the existing IAIA-BL debugger that uses input level annotations (in the form of masks of irrelevant pixels), ProtoPDebug moves towards concept level interaction by storing forbidden and valid concepts in the embedding space, which permits easier transfer of the feedback between inputs. The benefits of this approach are experimentally shown on two different domains: bird classification and COVID diagnostic from chest radiographies. Part of the study is truly interactive, either using a public platform or internally to the authors' research lab. ", "summary_of_the_review": "I find the submission to be a solid and effective contribution. I have some concerns about the generality of the claims but I would still like to recommend acceptance. I'm looking forward to discussion with the authors to clarify some misunderstandings.", "strength_and_weaknesses": "This is a well-scoped contribution which is very clearly presented. I find the experimental results particularly convincing of the benefits of ProtoPDebug, especially Figure 4 right which shows a clear gap in terms of generalisation compared to all alternative approaches on the birds dataset. I was able to clearly follow the methodology leading to these results.\n\nI however have some concerns about the main claim that ProtoPDebug is a *\"concept-level debugger\"*. ProtoPDebug certainly departs from prior work by operating in the embedding space instead of remembering pixel-level masks, but there still seems to be a very strong visual prior throughout the paper:\n- I find the user questions (Appendix H) to be heavily biased: *\"Your task will be to carefully inspect each image and then to determine whether the highlighted overlay covers some part of the bird or whether, instead, it covers exclusively (or very nearly so) the background.\"*. This question basically assumes that the confounding factors are already known. To me, this phrasing essentially masks the underlying problem of determining causality, which undermines the claims. It would be better to have a more neutral phrasing, for instance *\"Do these patches accurately explain the prediction?\"*.\n- Section 4 always assume an image structure, such that it is possible to extract a \"cut-out\", which is then re-embedded into a vector. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RqUXZKt_w1H", "reviewer": "Reviewer_FdCM", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a concept-level debugger for ProtoPNets in which a human supervisor, guided by the model’s explanations, supplies feedback in the form of what part-prototypes must be forgotten or kept. ", "summary_of_the_review": "Interesting paper presenting an approach to refine explanation through user interaction. I really think this approach has some added value, and is an interesting way of tackling (incrementally) the problem of eplainability.", "strength_and_weaknesses": "+ Interesting idea of iterative explanation\n- Missing larger scale end-user evaluation  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NiEtwazdY0", "reviewer": "Reviewer_oazP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a concept-level debugger, called ProtoPDebug, for prototypical part networks (ProtoPNets). ProtoPDebug addresses the issue of confounding prototypes learned by a ProtoPNet, by performing human-in-the-loop model debugging. In each debugging iteration, a human expert can mark a prototype as a \"confounder\" and add the corresponding highly activated region from the prototype's source image to a set of \"forbidden concepts\" F. Optionally, a human expert can also mark a prototype as a \"high-quality\" prototype and add the corresponding highly activated region from the prototype's source image to a set of \"valid concepts\" V. ProtoPDebug can then fine-tune the network by minimizing a loss that encourages the new parameters to be close to the existing ones, and encourages the new prototypes to not activate on the set of forbidden concepts and to activate on the set of valid concepts. The key benefit of ProtoPDebug (over an instance-level debugger such as the IAIA-BL) is that ProtoPDebug does not require fine-grained annotations on individual input instances at all, and is therefore easy to use.", "summary_of_the_review": "Based on the strengths of this paper, I recommend to accept the paper.", "strength_and_weaknesses": "Strengths:\n\n- Novelty: The concept-level debugger, ProtoPDebug, is a very novel idea. It allows us to interactively debug a ProtoPNet, without requiring instance-level fine-grained annotations.\n- Technical soundness: ProtoPDebug is in general technically sound. The idea of fine-tuning the network using a loss that discourages new prototypes to activate on forbidden concepts and encourages them to activate on valid concepts is very neat.\n- Writing: The paper is clearly written and easy to follow.\n\nWeaknesses:\n\n- Minor issue: I have a minor issue with equation (11). In particular, equation (11) introduces a re-ordering on part prototypes of the updated model. Could you clarify what is meant by \"re-ordering the part-prototypes\"? Is \\pi a permutation of the part prototypes of the updated model? If so, shouldn't the corresponding weight entries in w' be re-ordered as well? Please explain.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "ENZdTPAwjj5", "reviewer": "Reviewer_tqKr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ProtoPNets, a method to debug models at the concept level.", "summary_of_the_review": "This is a strong paper that proposes an effective concept-level debugger that the authors validate not only using quantitative experiments but also using human-subject experiments. I foresee this paper garnering some attention for its simplicity and clear validation.", "strength_and_weaknesses": "Strengths\n- The paper makes good progress on debugging models using concepts. The authors write well: this makes it quite straightforward to see how this paper varies from existing work. The contributions are thus clear.\n- The layperson validation of the method is quite convincing and shows the utility of ProtoPDebug.\n- Figure 3, which should be Algorithm 1(?), is straightforward to implement\n\nWeaknesses\n- Please explain what is meant by \"θ' is consistent with F\" in Equation 10\n- It would be nice to make the the debugging process iterative (instead of one off to build up V or F); e.g., how many examples does it take before no debugging is needed? How much does debugging vary between participants", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%