Route, Interpret, Repeat: Blurring the Line Between Posthoc Explainability and Interpretable Models 
Authors: Shantanu Ghosh
Ke Yu
Forough Arabshahi
kayhan Batmanghelich
Venue: Submitted to ICLR 2023 (2023)
Keywords: Explainable AI
Posthoc explanation
Computer Vision

Abstract: The current approach to ML model design is either to choose a flexible Blackbox model and explain it post hoc or to start with an interpretable model. Blackbox models are flexible but difficult to explain, whereas interpretable models are designed to be explainable. However, developing interpretable models necessitates extensive ML knowledge, and the resulting models tend to be less flexible, offering potentially subpar performance compared to their Blackbox equivalents. This paper aims to blur the distinction between a post hoc explanation of a BlackBox and constructing interpretable models. We propose beginning with a flexible BlackBox model and gradually carving out a mixture of interpretable models and a residual network. Our design identifies a subset of samples and routes them through the interpretable models. The remaining samples are routed through a flexible residual network. We adopt First Order Logic (FOL) as the interpretable model's backbone, which provides basic reasoning on the concept retrieved from the BlackBox model. On the residual network, we repeat the method until the proportion of data explained by the residual network falls below a desired threshold. 
Our approach offers several advantages. First, the mixture of interpretable and flexible residual networks results in almost no compromise in performance. Second, the rout, interpret, and repeat approach yields a highly flexible interpretable model. Our extensive experiment demonstrates the performance of the model on various datasets. We show that by editing the FOL model, we can fix the shortcut learned by the original BlackBox model. Finally, our method provides a framework for a hybrid symbolic-connectionist network that is simple to train and adaptable to many applications.

Under review as a conference paper at ICLR 2023
ROUTE, INTERPRET, REPEAT: BLURRING THE LINE BETWEEN POST HOC EXPLAIN- ABILITY AND INTERPRETABLE MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
The current approach to ML model design is either to choose flexible Blackbox model and explain it post hoc or to start with an interpretable model. Blackbox models are flexible but difficult to explain, whereas interpretable models are de- signed to be explainable. However, developing interpretable models necessitates extensive ML knowledge, and the resulting models tend to be less flexible, of- fering potentially subpar performance compared to their Blackbox equivalents. This paper aims blur the distinction between post hoc explanation of Black- Box and constructing interpretable models. We propose beginning with a flexible BlackBox model and gradually carving out mixture of interpretable models and a residual network. Our design identifies a subset of samples and routes them through the interpretable models. The remaining samples are routed through a flexible residual network. We adopt First Order Logic (FOL) as the interpretable model's backbone, which provides basic reasoning on concepts retrieved from the BlackBox model. On the residual network, we repeat the method until the propor- tion of data explained by the residual network falls below a desired threshold. Our approach offers several advantages. First, the mixture of interpretable and flexible residual networks results almost no compromise in performance. Second, the route, interpret, and repeat approach yields highly flexible interpretable model. Our extensive experiment demonstrates the performance of the model on various datasets. We show that by editing the FOL model, we can fix the shorteut learned by the original BlackBox model. Finally, our method provides a framework for hybrid cumbolic-connectionn network that is simple to train and adaptable many applications.
INTRODUCTION
Model explainability is essential in high-stakes applications of AI, such as healthcare. While BlackBox models (e.g.. Deep Learning) offer flexibility and modular design, post hoc explana- tion is prone to confirmation bias Wan al. 2022), lack of fidelity to the original model Adebayo şt al. (018), and insufficient mechanistic explanation of the decision-making process Rudin (201)) becepretable-by-deiseee
models do not suffer from those issues but tend to be less flexible than Blackbox models and demand substantial expertise to design and fine-tune. Using post hoc expla- nation or adopting an interpretable model is a mutually exclusive decision phase of Al model design. This paper aims to blur the line on that dichotomous model design.
made at the initial
The literature on post hoc explainable Al is extensive, The methods such as model attribution (e.g., Saliency Map Simonyan et al. (2013); Selvaraju et al. (2017)). counterfactual approach Abid et al. 2001 Singla et al. (2019), and distillation methods Alharbi et al. (202T); Cheng et al. (2020) are examples of post hoc explainability approaches. Those methods either identify important features of input that contribute the most to the network's output Shrikumar et al. (2016), generate pertur- bation to the input that flips the network's 's output [Samek et al. (2016), Montavon et al. (2018). or estimate simpler functions that locally approximate the network output. The advantage of the post hoc explainability methods is that they do not compromise the flexibility and performance of the BlackBox. However, the post hoc explainability method suffers from several undesirable signifi- cant drawbacks, such as a lack of fidelity and mechanistic explanation of the network output Rudin

---

Under review as a conference paper at ICLR 2023
Without mechanistic explanation, recourse model's undesirable behavior is unclear. Interpretable models are alternative designs to the BlackBox model that do not suffer from many of those drawbacks.
2019).
Interpretable models also have a long history in statistics and machine learning Letham et al. (013);;
Breiman al. (1984). Several families of interpretable models exist, such as the rule-based approach and generalized additive models Hastie & Tibshirani (987p.. Many methods focus on tabular or categorical data and less on high-dimensional structured data such images. Interpretable models for structured data rely mostly on projecting to a lower dimensional concept or symbolic space that is understandable for humans Koh al. (220)) Aside from a few exceptions Ciravegna et al. (2021); ;;; Barbiero et al. (2022), the current State- Mate-OOf-hhe-A (SOTA) design does not model the interaction between concepts and symbols, hence offering limited reasoning capabilities and less robustness. Furthermore, current designs are not as flexible as the Blackbox model, which may compromise the performance of such models.
We aim to achieve the best of both worlds: the flexibility of the BlackBox and the mechanistic ex- plainability of the interpretable models. The general idea is that a single interpretable model may not be sufficiently powerful to explain all samples, and several interpretable models might be hidden inside the Blackbox model. We construct a hybrid neuro-symbolic model by progressively carving out mixture of interpretable model and residual network. Our design identifies a subset of sam- ples and routes them through the interpretable models. The remaining samples are routed through a flexible residual network. We adopt First Order Logic (FOL) the interpretable model's backbone, which provides basic reasoning on concepts retrieved from the BlackBox model. FOL is a logical function that accepts predicates (concept presence/absent) as input and returns a True/False output being logical expression of the predicates. The logical expression, which is a set of AND, OR, Negative, and parenthesis, can be written in the so-called Disjunctive Normal Form (DNF). DNF is a FOL logical formula composed of a disjunction (OR) of conjunctions (AND), known as the "sum of products." On the residual network, we repeat the method until the proportion of data explained by the residual network falls below a desired threshold. The experimental results across various computer vision and medical imaging datasets reveal that our method accounts for the diversity of the explanation space and has minimal impact the Blackbox's 's performance. Additionally, apply our method's explanations to detect shortcuts in computer vision and successfully eliminate the bias from the Blackbox's 's representation.
2 RELATED WORK
Post hoc explanations Simonyan et al. (013); Selvaraju et al. (2017): SSmikov et al. (017) dis- cuss the post hoc based explanation method using saliency maps explain convolution neural network. This method aims to highlight the pixels in the input images that contributed to the net- work's prediction. Adebayo et al. (0188): Kindermans al. (2019) demonstrates that the saliency maps highlight the correct regions in the image even though the backbone's 's representation was arbi- trarily perturbed. Additionally, in LIME Ribeiro et al. (016), given superpixel, a surrogate linear function attempts to learn the prediction of the Blackbox surrounding that superpixel. SHAP Lund- berg & Lee (017)) utilizes prominent game-theoretic strategy called SHAPLY values estimate the Blackbox's 's prediction by considering all the permutations of adding and removing specific feature to determine its importance in the final prediction. Having the explainations in terms of pixel intensities, they do not correspond to the high-level interpretable attributes (concept), understood by humans. In this paper, we aim to provide the post hoc explanation of the Blackbox in terms of the interpretable concepts, rather than the pixel intensities.
Interpretable models In this class, the researchers try to design inherently interpretable models to eliminate the requirement for post hoc explanations. In the literature, we find interpretable mod- els in Generalized Additive Models (GAM) Hastie & Tibshirani 1987), or on logic formulas, as in Decision Trees Breiman al. (1984) or Bayesian Rule Lists (BRL) Letham et al. (2015). However, most of these methods work well in categorical datasets rather than continuous data such as images. Additionally, Chen et al. (2019; Nauta al. 2021) introduce a "case-basee reasoning" technique where the authors first dissect an image in Prototypical parts and then classifies by combining evi- dence from the pre-defined prototypes. This method is highly sensitive to the choice of prototypes
2

---

Under review as a conference paper at ICLR 2023
and the distance metric used to compare with the prototype. So instead of using any prototypes, proposed method uses human- unmaa---ddesttadaalle concepts to build a mixture of interpretable models.
Concept-based interpretable models Recently, Concept bottleneck models (CBM) Koh et al. (2020) broaden the initial aim Lampert et al. (2099] Kumar et al. (2009) to leverage the weak annotations of visual attributes as high-level human- compreecmmpe concepts Kim et al. (2017) in a fine-grained image classification dataset to predict the concepts from input images. Subse-
quently, the downstream classifier predicts the class labels from the discovered concepts. [Chen
2020) aims to align the axes of the latent space with the concepts of interest. Margeloiu et al.
al.
2021; Mahinpei et al. 202Tg demonstrated how the information is encoded in the concept representation. However, they do not explain how the final classifier composes those concepts for prediction. Ciraveena et al. T02T utilizes constraint-based approach from Ciravegna et al. 2020a) Ciravegna et al. 20206 to address this gap by providing explanations in terms of first-order logic (FOL) using the concepts. However, a strong pruning strategy and LI regularization hinder the learning capability of LENs. E-Lens Barbiero et al. (022) solves this problem by introducing an attention-driven entropy layer that focuses on the relevant concept for downstream classification.
Later, Yuksekgonul et al. 2022) introduces the Post hoc-concept-baan based model (PCBM) to learn the
concepts from the backbone of a trained Blackbox and utilize an interpretable classifier to learn the
class labels from the concepts. In addition, they propose the Hybrid-Posi hoc-concept-basee model (PCBM-h) to replicate the performance of the original Blackbox by fitting the unexplained portion of the Blackbox. However, all these methods either employ single interpretable by-design model or a surrogate model to explain a Blackbox, failing to apprehend different nuances of explanations per sample. Instead of a single interpretable model for all samples, this paper introduces a mixture of interpretable models that utilize the Blackbox's 's flexibility and offers local explanation.
3 METHOD
Notation Assume we have dataset (X, V. C), where X, V. and C. the input images, class labels, and human interpretable attributes, respectively. 1 x²= - V/ is a pre-trained deep neural network (our initial Blackbox model), trained only with supervision from the labels V. We assume that f is composition λº t. where + X - R is the image embeddings and ht" R' - y is a transformation from the embeddings, 4, to the class labels. We denote the learnable function t t R' + C. projecting the image embeddings to the concept space. The concept space is the space spanned by the attributes C. Per this definition, function / outputs a scalar value representing concept for each input image.
Method Overview Figure shows an overview of our approach. We aim to iteratively carve out an interpretable model from the given Blackbox model. Each iteration results in an interpretable model (the downward grey paths in Figure D and residual (the straightforward black paths in Figure Dp. We start this process with f the initial Blackbox given to us. At iteration k; we distill the Blackbox of the previous iteration pk-1 with a neuro-symbolic interpretable model, g* CC+ V. We generate explanations in terms of First Order Logic (FOL) from g as suggested by Barbiero et al. 2022). We coin the term residual y* to refer to the portion of pk-1 that 9² cannot explain, We then approximate p with / (x(((()) f will be the Blackbox for the subsequent iterations and explained by the respective interpretable models. learnable gating mechanism, denoted by T² - {0, 1) (shown the selector in Figure I is responsible for routing an input image towards either g² or x² The thickness of the lines in Figure represents the samples covered by the interpretable models (grey line) and the residuals (black line). With every iteration, the cumulative coverage of the interpretable models increases, but the residual decreases. We name our method route, interpret and repeat.
specifically n =ppt- ² p
3.1 NEURO-SYMBOLIC KNOWLEDGE DISTILLATION
Neuro-symbolic AI is an area of study that encompasses deep neural networks with symbolic ap- proaches to computing and AI to complement the strengths and weaknesses of each, resulting in robust Al capable of reasoning and cognitive modeling Belle (2020). Neuro-symbolic systems are hybrid models that leverage the robustness of connectionist methods and the soundness of symbolic
3

---

Under review as a conference paper at ICLR 2023
reasoning to effectively integrate learning and reasoning Garcez al. 2015:: Besold et al. 2017
Neuro-Syme Seuro-Symbolic knowledge distillation in our method involves three parts. (1) a series of train- able soft attention scores that routes each sample through the interpretable models and the residual networks, (2) A sequence of learnable neuro-symbolic interpretable models, each providing FOL. explanations to interpret the Blackbox, and (3) repeating with Residual networks for the samples that cannot be explained with their interpretable counterparts. We explain each component in detail in what follows.
3.1.1 THE SELECTOR FUNCTION
The first step our method to rout an input sample through the interpretable model g and residuals y using a series of soft attention scores x² for k: E [0, K], where K the number of iterations. We denote this as the selector in Figure [ Specifically, at each iteration k, the specific selector rontes a sample {.., Mj Cy} towards g" and r with probability (e,) and 1- (T/))
x (cy). the 1 m
respectively. We define the empirical coverage of the 4thh iteration as, ((=**)
m
empirical mean of the samples selected by the selector for the corresponding interpretable model g", with m being the total number of samples in the training set. Thus the total selective risk is,
Lh (g +y) cj)
R* (TT gg*)
((TT))
where Lk at is the optimization loss used learn g and nK together, discussed in section 3.1.2 For given coverage of 7* E (0, 1]. we solve the following optimization problem
02.0% min 00.00
s.t. ((T"(:: 2
For 4tth iteration, 0- i are the optimal parameters for the selector n and the interpretable model ge respectively. In this work, " are neural networks with sigmoid activation. At inference time, the concept vector Cj of sample v, fed to g" if and only π (c,) Z 0.5 for k E K].
3.1.2
INTERPRETABLE MODELS
In this stage for the iteration k, we design interpretable model A to interpret the Blackbox fk-1 from the previous iteration -1 1 by optimizing the following loss function,
k-1
( (.), cj) ( (m,), ( (c))))
(1-x(())
(3)
trainable component For current Steration k
fixed component trained previous iterations
The term multiplied by the loss function l in equation 3 is the probability of sample æj reaching the interpretable model g+, which is the probability of going through the residual for iterations through (i.e. I1==1 (1 (c;))) times the probability going through the interpretable model at iteration k (i.e. n (c;)). Refer to Figure for an illustration. Note that X .................. 7 k-1 are learned in the previous iterations and are not trainable iteration k. each interpretable model g" specializes in explaining a specific subset of samples (denoted by coverage 7), we refer to it as an expert moving forward. We adopted the optimization strategy, discussed Geifman & El- Yaniv (2019) to optimize equation 2 We discuss the loss function in equation iin detail in Appendix A.3 After training, we refer to the interpretable experts of all the iterations as "Mixture of Interpretable Experts" (MolE) cumulatively.
3.1.3 THE RESIDUALS
The last step is to repeal with the residual n* (xjj Cj) (ag) -g((e)) generate f*. effectively making Blackbox for the next iteration k+ Specifically, train f h² (T(.)
4

---

Under review as a conference paper at ICLR 2023
Fix Ø
@ h
h
h
h
Residual (r")
Residual (r*)
Residual (r2)
Residual (r*)
Blackbox Model
Symbolic Model
Goodad
Selector
Figure 1: Schematic view of route, interpret and repeat. At step k. the selector routes each sample either towards the interpretable model 9 (to interpret) with probability x" or the residual (to repear in the further iterations) r = fk 9 with probability previous iteration, k: 1. If the sample goes through n², we obtain the explanations for that sample terms of First Order Logic (FOL.). Otherwise, the sample goes through the next step until either goes through subsequent interpretable model reaches the last residual. Note that components in black and grey indicate the fixed and trainable modules in our model.
R². Here, fk-1 is the Blackbox of the
match the residual r² This step is crucial to make f specialize over those samples not covered by g². For the th iteration, we optimize the following loss function to obtain fh:
C) eg) (a),
II - (cz))
(4)
trainable component for itcration k nom tonittranablee component iteration
Notice that the embedding ((.) is fixed for all the iterations. Due to computational reasons, only update the last few layers of the Blackbox (h*) in order to train f*. At the final iteraion K, method yeilds MolE, explaining the interpretable component of the initial Blackbox f and
the final residual for the uninterpretable component of fo Algorithm | in Appendix [A.4 explains the training procedure of our model and extraction of FOL explain samples locally. Figure X in
Appendix A.4 shows the architecture of our model at inference.
4 EXPERIMENTS
To validate our method, we utilize four datasets 1) bird species classification using CUB-200 Wah al. (201T) dataset, 4) animals species classification using Animals with Attributes2 (Awa2) dataset Xian et al. 2018),3 3) skin lesion classification using HAM10000 dataset Tschandl et al. 200 8), 4) Effusion and Cardiomegaly classification from radiology images using MIMIC-CXR dataset John- son et al. First, we demonstrate that MolE does not hinder the performance of the Blackbox. Sec- ond, we conduct qualitative analysis to illustrate the effectiveness of local explanations. Third, we validate that the cumulative coverage of the residuals decreases over successive iterations. Finally, we leverage the Waterbirds dataset Sagawa et al. (0019 to eliminate shortcut bias in vision datasets.
Training configurations As Blackbox. we use ResNet-101 He al. (016) and Vision Trans- former (VIT) Wang et al. (2021) for the CUB-200 and Awa2 datasets, For HAM 10000 and MIMIC- CXR, we utilize Inception Szegedy et al. 2015)) and Densenet- 121 Huang al. 2017) as the same. For CUB-200, we employ six experts each for both VIT and ResNet-based Blackboxes. For Awa2, we utilize and experts for ResNet-101 and VIT, respectively. For HAM0 and MIMIC-CXR (for both Cardiomegaly and Effusion), we use 5 and 3 experts, respectively. Refer to Appendix A.2 and and[A. A.7 for details about the dataset and hyperparameters. Furthermore, we only include concepts as input to g if their validation accuracy auroc exceeds certain threshold (in all of our experiments, we fix 0.7 or 70% as the threshold of validation auroc or accuracy). For 1000,000, we use the identical "concept bank" proposed in Yuksekgonul et al. (0222)
5

---

Under review as a conference paper at ICLR 2023
Accuracy scores datasets
1.00
Medical imaging datanatt
Accurcyy
20
0.00
Coe D.80
Cam Re
Aeid Coe-Ba
HdM Car
MusiC CAR tefoiio
0.06
MAM Cor 0.86
uBaI CAR Eitusian Cor
Cerdioree
Detaset
Datagct
Figure 2: MolE does not hurt the performance of the Blackbox. From left to right, compare the (a) accuracy and (b) auroc metrics respectively. For the MolE (ours), we specify the empirical coverage as cov in the figure w.r.t each datasets.
4.1 BASELINE
Our baseline is based on the interpretable by design principle, consisting of two parts: 1) a concept extractor & X + C, a mapping from the image to the concept space; and 2) g C - V, mapping from the concepts to the prediction. We train & and g sequentially from scratch with the supervision of the human attributes C and class labels y respectively. The concept extractor for Resnet, Inception, and Densenet 121 includes all layers until the last convolution block. The VIT- based T consists of the transformer encoder block (excluding the classification head). We use the identical configurations for g as described in Appendix A.7 for each dataset.
4.2
RESULTS
4.2.1
QUANTITATIVE ANALYSIS OF MOIE WITH THE BLACKROX AND BASELINE
MALIGNANT
BENIGN
Exjert
B Expent
II Expern
Expert2
Experti
7ppioll
TypkaliggrenNNeieent
AarStaraaa
AyptalPggentiheterr
RegularStreaks
Stress
Imageler 00
RepptrDG
RegularDG
BWV
BWV
Sea
12]
(6)
Figure 3: From left to right, for each expert, bar plot showing the importance of concept for predicting a skin lesion as Malignant and Benign respectively. For example, adypccaliimmnnneeorrk concept plays crucial role for predicting "Malignant" skin lesion by expert5 as is present in all the local explanations for expert5.
Figure 2 displays the performance of method using heldout test "MolE" and "MolE Residual" denote the mixture of interpretable experts (g) excluding and including the final residual, respectively. Coverage (Cov in the figure) represents the proportion of samples the test set cov- ered by all the experts throughout iterations. For CUB-200, Awa2, and HAM10000, MolE achieves similar performance to the Blackbox. For CUB-200 and HAM10000, our method achieves a sig- nificant performance gain compared to the baseline. For Awa2, the baseline performs at par with our model. (ResNet-based baseline exceeds ours). Note that all the datasets include noisy concepts except Awa2, making it appropriate for the interpretable by design models. In general, VIT-derived experts performed better than their ResNet-based counterparts. The MIMIC-CXR dataset is highly imbalanced, with the positive samples being too few compared to the negative ones. As result, the baselines for them attain similar accuracy scores as our MolE model (Figure 2b). Therefore, precision and recall are better metrics for assessing the models' performance these datasets. To

---

Under review as a conference paper at ICLR 2023
Expertó
[Interpretable by design)
Experté
(Interpretable by design)
seelly
Payoed
Betwook
Experts
Experta
-igrrosionn Stevotura
Zeregulor
bookirett
vongurch coadly bendin
Stresks
(b)
Figure 4: Illustration of flexibility of FOL explanations by MolE. We report the FOL for different experts and the baseline to identify (a) Malignant lesion and (b) Horse. For example, in Figure (a), the FOL by the baseline composes BWV, RegressionStructure and IrregularStreaks concepts to distinguish all the samples of the skin lesion "Malignant". However in the same Figure (a), expertó identifies the instances of "Malignant" in the first two rows with
and RegularDG as the identifying concept. Similarly expert4 classifies the samples of "Malignant" in the third row of Figure (a) using other BWV, IrregularStreaks, RegularDG and IrregularDG.
demonstrate this, we plot the auroc in Figure 2b. As it can be seen, MolE significantly outperforms the baseline in terms of auroc.
4.2.2 LOCAL EXPLANATIONS BY MolE
As shown in E-Lens Barbiero et al. 2022), each expert constructs local FOL per sample, composing the concepts based on their attention weights. Due to space constraints, we report the analysis for the
FOL explanation of HAM10000 and CUB-200 datasets here; Appendices A.9.T and [ include the results of Awa2 and MIMIC-CXR datasets, respectively. Figures 3 - exhibit quantitative analysis of the relative frequency of the concepts used by various experts to correctly classify the "Malignant" and "Benign" skin lesions, respectively. As per Figure Bh. local FOLs' of all samples analyzed by experts 2, 3, and include the concept AtypicalPignen Weeeneeerggen/,eeeerrr. proving it to be an essential concept for predicting malignancy. Also, expert4 (the red bars in the Figure Ba) classifies "Malignancy." " using multiple concepts, such as IrregularStreaks, IrregularDG, and Blue Whitish Veil (BWV), whereas expert2 (the blue bar in the Figure 31) just utilizes ArppialPigmentNerworr
Expert1
Symbol
Concapt
0g
back patter stripped
Cg
leg_rolor__re grey
0
shope_perchinge perching, like
Bay breasted warbler
116
.44
96
wing color white
nope color grey
Expert2
46
primary color black
100
primary pollr_oooereeee grey
C0
46
pattem
pattememmmahhclooeee mullicolooe
4p
hape_solor Blakk
-10
bill_shape_ailinn pillpurpoue
.11
nape joler__bleee
Expert4
418
forchead golor, Mack
418
undertail calk
.14
thorat_poior solor_ Mack
primary_svlor_ searrsssorr_rbbee wlohe
616
pallyymettem_mm Jolid
spperpurt__soov. Black
(a)
(b)
Figure 5: Illustration of class-level or global explanations. We report the global explanation of "Bay breasted warbler" using the VIT-based blackbox by combining (using DNF) the local explanations. (a) Explanations by expertl (top row), expert2 (middle row) and expert4 (bottom row). Shared concepts across all the experts are highlighted in blue. We highlight the same for experts and 2 in orange. (b) Symbols used in Figure 3a and their corresponding concept labels.
7

---

Under review as a conference paper at ICLR 2023
Cevenagm experts tym
Ceverapes sprien experts pi77
AwAz
Figure 6: Coverage and performance of each VIT-derived expert and residual for (a) CUB-200 (b) Awa2
Figure compares the FOL explanations by different experts with the baseline. As the baseline includes single interpretable model g. all the FOLs for each sample per class incorporate identical concepts. According to Figure Rh. expert4 relies on Blue Whitish Veils (BWV), Irregular Streaks, Regular and Irregular Dots and Globules (DG) to identify "Malignancy", whereas expertó utilizes Rregular Dots and Globules and ATypical Pigment Net. Appendices A.9. and A.8.7 display more qualitative FOLs for Awa2 and HAM10000 datasets respectively. MolE demonstrates similar flexi- bility in creating the FOL explanation for Awa2. For example, in 4 b. the baseline utilizes longneck, smelly, longleg and buckteeth to identify all the samples of "Horse" MolE employs the concepts on per-sample basis, so different experts adopt different concepts to classify the samples of a particu- lar class. In Figure the expert4 uses smelly, but the expert5 uses fields and longneck to distinguish "Horse".
We perform the same analysis for CUB-200 to formulate a global explanation of a class by combin- ing the different local FOLs per sample using conjunctive normal form (CNF) as in E-Lens. Figure 5 depicts such global explanations for the species "Bay-breasted warbler" by three distinct experts. Here we capture the heterogeneity in class-level explanations since each FOL corresponding to an expert has an amalgamation of distinct and shared concepts. For instance leg_coll leg_color__e _l__ooooooreyyyeeeee is unique to expert4, but belly_pattern_slle and pal_paattemmmllilllrrre are unique to experts and 2, respectively.
4.2.3 PERFORMANCE OF EACH EXPERT AND THE RESIDUAL
Figure displays the proportional accuracy of MolE and the residual for the CUB-200 and Awa2 datasets. Proportional accuracy of each model (experts and/or residuals) is defined as the accuracy of that model times its coverage. Recall that the model's coverage is the empirical mean of the samples selected by the selector. The dashed line in each of the plots indicates the performance of the blackbox, f0, denoting the upper bound of our MolE models. As shown in Figure Ch. the VIT- derived expert and residual cumulatively achieve an accuracy of approximately 91% for the CUB- 200 dataset in iteration 1, where the residual's contribution (black bar) is greater than the expert1 's (blue bar). Later iterations reveal that the MolE's performances cumulatively increase, and the residual worsens. In the final iteration, the MolE contributes most to performance as they carve out all the interpretable segments from the Blackbox. We observe that the final uninterpretable samples that are routed towards the final residual are "harder" samples for the model to predict, resulting in a lower residual accuracy. Tracing these samples back to fo, we observe that the original blackbox has similar low performance for predicting these samples. We have shown some of these "hard" images side by side with the ones explained by the experts for CUB-200 in Appendix A.8.8 Also, Table verfies this claim, Per our investigation and as it can be in the images, "hard" examples tend to be blurry or distorted images or ones with birds not fully centered or hard to tell from the background.
As shown in the coverage plot, this experiment reinforces Figure where the flow through the experts gradually becomes thicker compared to the narrower flow of the residual with every iteration. We observe similar trends for the other architectures and datasets. For medical imaging (HAM and MIMIC-CXR) datasets and ResNet-derived experts of vision datasets, refer to figures 12 and in the Appendix and A.8.2 respectively.
8

---

Under review as a conference paper at ICLR 2023
Auroe scores of the beckground concepts
Blased Blackbow
ocean
1911E
51
Greendirat Winettet
Nell
Preekcties Laxiddd
Eqpinetin Leaflted
beckground concupts
Rotest Blackhoo
Oves lica's Waoelle
Pawkk tox. Waoelled
Baylsiike Manellel one Bell wal.hlatt de "illl wnteeee do all wering Cal
DLEI LIFe Dardee
Tensi
X
16
Figure 7: Applying MolE to eliminate shortcuts. (a) Performance of the biased Blackbox. (b) Performance of final MolE, trained using the supervision of the robust Blackbox after removing the shortcuts using MDN. (c) Examples of samples and their explanations by the biased (top-row) and robust Blackhoxes row). (d) auroc and (e) accuracy metrics of the biased Blackbox (when the shortcuts are present) and the robust Blackbox (when the shortcuts are removed using MDN).
4.3 APPLICATION IN REMOVAL OF SHORTCUTS
To conduct this experiment, we create the Waterbirds dataset as in Sagawa et al. (2019) by spuriously correlating birds' images from the Caltech-UCSD Birds-200-201 dataset to the images of land and water background from the Places dataset. Specifically, we use forest and bamboo as the spurious land concepts for landbirds; ocean and lakes as the spurious water concepts for waterbirds. Thus, have 112 visual concepts (108 concepts from CUB-200 and four new concepts for the background). We aim identify each bird as a Waterbird the Blackbox f The Blackbox quickly latches on the spurious background to classity the birds. As result, the black box's accuracy varies for land- based subsets versus aquatic subsets of the bird species, as shown in Figure 7.. The Waterbird on water is more accurate than on land ((96% vs. 67 % in the orange bar in the Figure (a). We train to retrieve the concepts from the biased Blackbox. Next, we train our MolE to explain the bird species using FOL. As per Figure 7 (middle row), the FOL of a waterbird misclassified as a landbird captures the spurious concept forest. Assuming the background concepts metadata, we aim to reduce the background bias from the representation of the Blackbox using metadata normalization (MDN) Lu et al. (202Ip. Therefore, we leverage the MDN layers between two successive layers of the convolutional backbone to fine-tune the biased Blackbox. Next, we train f, using the embedding & of the robust Blackbox. Figures I compare the auroc and accuracy scores of the spurious background concepts from the embedding of the biased and robust Blackboxes, respectively. The validation auroc (accuracy) of all the spurious concepts retrieved from the robust Blackbox fell well short of the predefined threshold 0.7 (70 %) compared to the biased Blackbox. Finally, we re-train the MolE distilling from the new robust Blackbox. Figure D illustrates that the accuracies of Waterbird on water vs. Waterbird on land become pretty similar (91% % -88 %). The FOL from the robust Blackbox does not include any background concepts (Figure c. bottom row). Refer to the Figure in Appendix A.6 for the flow diagram of this experiment.
Landbird. We use ResNet50 (He et al. (2166) as
5 DISCUSSION & CONCLUSIONS
This paper proposes a novel method to iteratively extract a mixture of interpretable models from trained Blackbox by utilizing the flexibility of the Blackbox bridge the gap between post hoc explanation and the construction of interpretable models. The comprehensive experiments on a var- ious datasets demonstrate that 1) our model maintains a similar performance compared to the Black box while making it explainable, 2) the local explanations by our model account for the diversity of explanations of the various samples a dataset, 3) the coverage of the residuals gradually declines over the iterations. Finally, we employ our method to eliminate the shorteuts in images effectively. We identify the concepts from the representation of the blackbox based on correlation, not causation. So, it is still unclear if the blackbox uses those concepts for prediction. A possible future direction can be the discovering only those concepts causally, used by the blackbox for prediction. In the future we also aim apply method to other modalities such as text or video.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abubakar Abid, Mert Yuksekgonul, and James Zou. Meaningfully explaining model mistakes using conceptual counterfactuals. arXiv preprint arXiv:2106. 12723, 2021.
Julius Adebayo, Justin Gilmer, Michael Muelly, lan Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
Raed Alharbi, Minh N Vu, and My Thai. Learning interpretation with explainable knowledge distillation. In 2021 IEEE International Conference Big Data (Big Data), pp. 705-714. IEEE, 2021.
Forough Arabshahi, Sameer Singh, and Animashree Anandkumar. Combining symbolic expressions and black-box function evaluations in neural programs. 2018.
Forough Arabshahi, Jennifer Lee, Antoine Bosselut, Yejin Choi, and Tom Mitchell. Conversa- tional multi-hop reasoning with neural commonsense knowledge and symbolic logic rules. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 7404-7418, Online and Punta Cana, Dominican Republic, November 2021a. Association Computational Linguistics. doi: 10. 10.18653/11/2001.mmmll// 1etp002L-.mmpmmmn..n.
Forough Arabshahi, Jennifer Lee, Mikayla Gawarecki, Kathryn Mazaitis, Amos Azaria, and Tom Mitchell. Conversational neuro-symbolic commonsense reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021b.
Samy Badreddine, Artur d' Avila Garcez, Luciano Serafini, and Michael Spranger. Logic tensor networks. Artificial Intelligence, 303:103649, 2022.
Pictro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pictro Lió, Marco Gori, and Stefano Melacci. Entropy-basee logic explanations of neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 6046-6054, 2022.
Vaishak Belle. Symbolic logic meets machine learning: A brief survey in infinite domains. In International Conference on Scalable Uncertainty Management, pp. 3-16. Springer, 2020.
Tarek R Besold, Artur d' Avila Garcez, Sebastian Bader, Howard Bowman, Pedro Domingos, Pas- Hitzler, Kai-Uwe Kühnberger, Luis C Lamb, Daniel Lowd, Priscila Machado Vieira Lima, et al. Neural-symbolic learning and reasoning: A survey and interpretation. arXiv preprint arXiv:1711.03902. 2017.
L Breiman, J Friedman, CJ Stone, and RA Olshen. Classification and regression trees (crc, boca raton, fl). 1984.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 12925-12935. 2020.
Gabriele Ciravegna, Francesco Giannini, Marco Gori, Marco Maggini, and Stefano Melacci. Human-driven fol explanations of deep learning. In IJCAI, pp. 2234-22400 2020a.
Gabriele Ciravegna, Francesco Giannini, Stefano Melacci, Marco Maggini, and Marco Gori. A constraint- based approach to learning and explanation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3658-3665, 2020b.
Gabriele Ciravegna, Pictro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, and Stefano Melacci. Logic explained networks. arXiv preprint arXiv:2 /49, 2021.
10

---

Under review as a conference paper at ICLR 2023
Roxana Daneshjou, Kailas Vodrahalli, Weixin Liang, Roberto A Novoa, Melissa Jenkins, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai: Assessments using diverse clinical images. arXiv preprint arXi::2111.0000 2021.
Luc De Raedt, Sebastijan Dumančić, Robin Manhaeve, and Giuseppe Marra. From statistical rela- tional to neuro-symbolic artificial intelligence. arXiv preprini urXiv:2003.083//6 2020.
Artur d' Avila Garcez, Tarek R Besold, Luc De Raedt, Peter Földiak, Pascal Hitzler, Thomas Icard, Kai-Uwe Kühnberger, Luis C Lamb, Risto Mikkkulainen, and Daniel L Silver. Neural-symbolic learning and reasoning: contributions and challenges. In 2015 AAAI Spring Symposium Series, 2015.
Yonatan Geifman and Ran EI- Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International conference machine learning, pp. 2151-2159, PMLR, 2019.
Trevor Hastie and Robert Tibshirani. Generalized additive models: some applications. Journal of the American Statistical Association, 82(398):371-388 1987.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv: AXXiv:1503.02331 2(7), 2015.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7700-478,, 2017.
Qiuyuan Huang, Paul Smolensky, Xiaodong He, Deng, and Dapeng Wu. A neural-symbolic approach natural language tasks. 2018
Saahil Jain, Ashwin Agrawal, Adriel Saporta. Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang. Matthew P Lungren, Andrew Y Ng, et al. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2 14463, 2021.
Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimie-exr-jpg-chest radiographs with structured labels.
Jeremy Kawahara, Sara Daneshvar, Giuseppe Argenziano, and Ghassan Hamarneh. Seven-point checklist and skin lesion classification using multitask multimodal neural nets. IEEE journal of biomedical and health informatics, 23(2):538 -465388--66200 2018.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (201)) arXiv preprint arXiv: /71 1279, 2017.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmano, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference Machine Learning, pp- 5338-5344 5348. PMLR, 2020.
Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pp. 365-372. JEEE, 2009.
Luis Lamb, Artur Gareez, Marco Gori, Marcelo Prates, Pedro Avelar, and Moshe Vardi. Graph neural networks meet neural-symbolic computing: survey and perspective. arXiv preprint arXiv:2003.0m 000300 2020.
11

---

Under review as a conference paper at ICLR 2023
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition, pp. 951-958. IEEE, 2009.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412. 2019.
Benjamin Letham, Cynthia Rudin, Tyler H McCormick, and David Madigan. Interpretable classi- fiers using rules and bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics, 93):1350-1371, 2015.
Mandy Lu, Qingyu Zhao, Jiequan Zhang, Kilian M Pohl, Li Fei-Fei, Juan Carlos Niebles, and Ehsan Adeli. Metadata normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10917-10927, 2021.
Zhichu Lu, Forough Arabshahi, Igor Labutov, and Tom Mitchell. Look-up and adapt: A one-shot semantic parser. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1129-1121111 1139, 2019.
Adriano Lucieri, Muhammad Naseer Bajwa, Stephan Alexander Braun, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. On interpretability of deep learning based skin lesion classi- fiers using concept activation vectors. In 2020 international joint conference on neural networks (IJCNN, pp. 1-10. IEEE, 2020.
Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceed- ings of the 31st international conference on neural information processing systems, pp. 4768- 4777, 2017.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Deshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314, 2021.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. 2019.
arXiv preprint
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn intended? arXiv preprint arXiv:2105.04288 042889 2021.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital signal processing, 73:1-15, 2018.
Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- grained image recognition. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 14933-14943, 2021.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. why should i trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. arXiv preprint arXiv: /91 I. 1.08731. 2019.
Wojciech Samek, Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, and Müller. Evaluating the visualization of what a deep neural network has learned. IEEE transactions neural networks and learning systems, 28(1 1 1:2260026673 2016.
12

---

Under review as a conference paper at ICLR 2023
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Avanti Shrikumar, Peyton Greenside, Anna Shcherbina, and Anshul Kundaje. Not just black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713. 2016.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: networs Vi- sualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
Sumedha Singla, Brian Pollack, Junxiang Chen, and Kayhan Batmanghelich. Explanation by pro-
gressive exaggeration. arXiv preprint arXiv: 1911v:911.000480 2019.
Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg. Smoothgrad: 2017.
removing noise by adding noise. arXiv preprint
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham10000 dataset, large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNIP-IJCNIP)) pp. 5784-5789, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v URL /tttss///acllamhhoooo.oo////////
1585
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Charles Wan, Rodrigo Belo, and Leid Zejnilovic. Explainability's 's gain is optimality's 's loss? how explanations bias docision-mmainne In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pp. 778-787, 2022.
Jun Wang, Xiaohan Yu, and Yongsheng Gao. Feature fusion vision transformer for fine-grained visual categorization. arXiv preprint arXiv:2107.02341 2021.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning a comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 1-2265,, 2018.
Ke Yu, Shantanu Ghosh, Zhexiong Liu, Christopher Deible, and Kayhan Batmanghelich. Anatomy-guided arXiv:2206. 2022.
abnormality localization in chest x-rays. arXiv preprint
Mert Yuksekgonul, Maggic Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205. 15480, 2022.
13


Reviews:
[{"review_id": "i7G6qC8b83", "reviewer": "Reviewer_1Nhn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an approach to explain black box classifiers by iteratively dividing the black-box into an interpretable model and a residual. They use first order logic on concepts for the interpretable models.\nA sample input is passed to the system and either routed (by a learnable gating mechanism) to the interpretable model or the residual, until it is interpreted or reaches the last residual.", "summary_of_the_review": "Overall I'm not quite convinced by the paper. As the procedure is relatively complicated and requires training, the motivation should be a bit clearer. Furthermore, there needs to be more quantitative evaluations and comparisons to state of the art concept based explanations.", "strength_and_weaknesses": "### strenghts\n- the explanations are relatively easy to read thanks to FOL\n- experiments are numerous and well described\n- the authors demonstrate that biases can be detected\n\n### weaknesses\n\n- a short explanation of the first order logic is missing\n- a short explanation of ``neuro-symbolic interpretable model'' is missing\n- no code\n- complicated approach, training required\n- no comparison with similar concept based explanation methods (like CAV)\n- no quantitative analysis of the produced explanations\n\n\n### questions\n- You write in section 3.1.1 that the selector routes a sample with a probability... Is this routing really probabilistic, so that I would potentially get different explanations when explaining the same image twice?\n- You are using the original black box classifier in the first iteration, if I understand correctly. Why is the performance in Figure 2 then not equal to the black box?\n- why do you not show explanations for the same samples in figure 4?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dXZ8NZDgy-C", "reviewer": "Reviewer_sRwr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes to bridge the gap between post-hoc explanation methods and interpretable-by-design methods.\n\nThis is achieved by a pipeline where given a black-box model, an interpretable component iteratively distills parts of the representation from the blackbox with uncovered (undistilled) parts forwarded to a residual component. By iterating on this process, the interpretable components achieves a higher coverage of the blackbox model while still preserving a competitive performance.\n\nExperiments on several datasets show the effectiveness of the proposed method on the image classification task for a variety of datasets. In addition, an alternative application of the method is presented on the task of bias detection.\n", "summary_of_the_review": "The proposed idea is sound and most of the design decisions are well motivated. To the best of my knowledge, the idea of having this iterative combination of interpretable components with residual components is novel. \n\nI also found positive the fact that the proposed method aims at interpreting different levels of the blackbox model and does not limit itself to the last convolutional layer, as is commonly the case. \n\nThe presentation of the manuscript is good, and the structure followed for the presentation of its content is adequate.\n\nMy main doubts with the manuscript are the following:\n\n- Positioning with respect to Prototype-based methods. In Section 2, the paper discusses ProtoPNet as a representative of prototype-based methods. As part of this discussion, weaknesses from ProtoPNet are stressed. My concern here, is that ProtoPNet is nothing more than the first of these family of prototype-based methods. In this regard there are more recent variants, e.g. Proto-Tree (Nayuta et al., CVPR21) which have addressed the weaknesses of the original ProtoPNet model. Here I would expect the positioning to be made with respect to more recent prototype-based methods.\n\n- On the selection of the number of experts. When describing the training configuration, the number of experts used for each blackbox-dataset combination was indicated. However, it is not clear to me the basis on which the number of experts is selected. Is there a principle manner to decide on the number of experts to use for the proposed method?\n\n- On the semantic association of the detected concepts. In Section 4.2.2, local explanation produced by the MoIE are discussed. Several of these are described based on concepts with a clearly associated semantic meaning , e.g. IrregularStreaks, IrregularDG and Blue Whithish Veil. H \n\n- On the explanation capabilities of the proposed method. Independently of its inner-workings, at the end of the day the proposed method is an explanation method. In this regard, it would have strengthen the manuscript if the proposed method was tested under the sanity checks proposed by [Adebayo et al., NeurIPS'18]. This way there could be guarantees on whether the generated heatmaps/attribution maps do constitute valid explanations.\n \n- Quantitative Comparison. At the moment there is no quantitative comparison on the outputs of the proposed methods with respect to those from state of the art methods. In this regard, quantitatively speaking, it is hard to assess whether the proposed method is a better explanation method than existing ones.\n \n- Minor. There are typos in several places of the manuscript. \n \n-------------- \nReferences\n \n Meike Nauta, Ron van Bree, Christin Seifert; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14933-14943\n\n", "strength_and_weaknesses": "Strengths\n+ well-supported idea\n+ tested for bias removal\n+ Good presentation\n\nWeaknesses\n- Relevant aspects not evaluated\n- Limited novelty\n- Some open questions", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MeEkitLWTn8", "reviewer": "Reviewer_7Y2K", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to iteratively extract a mixture of interpretable models from a trained Blackbox", "summary_of_the_review": "Overall the paper is well described, but novelty is not high given past works have achieved similar results", "strength_and_weaknesses": "+ Interesting problem\n- Clear missing state of the art cf. below\n- Limited experiments - specially in the context of semantic segmentation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Esa8Iv1ycIn", "reviewer": "Reviewer_gZAS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper advocates a hybrid approach between inherent interpretability and post-hoc explainability for predictive models trained on image data. The aim is to blur the line between post hoc explanations of a Black-Box and constructing interpretable models, aiming to keep the high performance and flexibility of blackbox models while providing interpretability. To this end, the authors propose an iterative approach where at each iteration, an interpretable model is distilled from the blackbox model. Samples that cannot be routed through the interpretable model(s) at an iteration are routed through a residual network, which is again a blackbox. The method is repeated on the residual network until until the proportion of data explained by the residual network falls below a desired threshold.", "summary_of_the_review": "The authors motivate their case and present extensive results. The aim of the proposed approach is described well but there are important missing details. The performance and results of the approach are discussed extensively. The paper could benefit from a more thorough discussion of related work on symbolic models and limitations of the proposed method. There are several typos in the manuscript.", "strength_and_weaknesses": "Strengths: The motivation is clear, the method is described well and the experiments conducted are extensive. The approach has been tested on a variety of datasets and the results have been discussed extensively and from various angles.\n\nWeaknesses: Section 2 on related work seems a bit short, especially the paragraph on concept- based interpretable models could be extended. What methods are used by others and how does the paper at hand differ from those? In line with that, the contributions of the paper in relation to previous work could be highlighted more. Similarly, there are no comparisons done with existing methods (except against the ‘baseline’). If there is a specific reason for that, it was not clear. If not, I would advocate comparing to current state-of-the-art methods for symbolic models for image data. Section 5 (discussion and conclusion) would benefit from a discussion of the strengths and limitations of the presented work. Future research trajectories could be discussed more detailed. \n\nHere are a few additional remarks and questions:\n\n- The readers would benefit from a brief explanation about neuro-symbolic interpretable models in Section 3.1.\n\n- The optimization problem (2) looks highly non-linear and non-convex. The authors have not mentioned difficulty of the problem, neither have they discussed the cases of obtaining multiple or only local solutions. They only propose one method to solve the problem in the appendix. However, this part is crucial as the method relies on the solution of this problem.\n\n- What is a good value for K to stop the algorithm? Is there a rule-of-thumb when to stop?\n\n- What is the effect of validation threshold (fixed to 0.7 in the paper)?\n\n- How would the method extend to non-image datasets?\n\n- What is a concept extractor in Section 4.1?\n\n- There are few typos that I spotted in the current version:\n   * In the beginning of Section 3, all $\\phi$ functions should be $\\Phi$.\n   * When referring to a figure, the authors sometimes use \"figure #\" instead of \"Figure #\" \n   * In equation (3), $c_j$ terms should be boldface", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject