CHiQPM: Calibrated Hierarchical Interpretable Image Classification
Authors: Thomas Norrenbrock
Timo Kaiser
Sovan Biswas
Neslihan Kose
Ramesh Manuvinakurike
Bodo Rosenhahn
Venue: NeurIPS 2025 poster ()
Keywords: XAI
Global Interpretability
Local Interpretability
Conformal Prediction
Image Classification
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: Interpretable Model with Global Interpretabiltiy and Novel Local Hierarchical Explanations, that can Provide Built-in Interpretable Coherent Set Prediction with Conformal Prediction

Abstract: Globally interpretable models are a promising approach for trustworthy AI in safety-critical domains. Alongside global explanations, detailed local explanations are a crucial complement to effectively support human experts during inference. This work proposes the Calibrated Hierarchical QPM (CHiQPM) which offers uniquely comprehensive global and local interpretability, paving the way for human-AI complementarity. CHiQPM achieves superior global interpretability by contrastively explaining the majority of classes and offers novel hierarchical explanations that are more similar to how humans reason and can be traversed to offer a built-in interpretable Conformal prediction (CP) method. Our comprehensive evaluation shows that CHiQPM achieves state-of-the-art accuracy as a point predictor, maintaining 99% accuracy of non-interpretable models. This demonstrates a substantial improvement, where interpretability is incorporated without sacrificing overall accuracy. Furthermore, its calibrated set prediction is competitively efficient to other CP methods, while providing interpretable predictions of coherent sets along its hierarchical explanation.

CHiQPM: Calibrated Hierarchical Interpretable Image Classification
Thomas Norrenbrock, Timo Kaiser & Bodo Rosenhahn Institute for Information Processing (tnt) L3S Leibniz Universität Hannover, Germany
(norrenbr, kai ser, rosenhahn } @tnt uni amiihannover.dde
Sovan Biswas & Neslihan Kose
Intel Labs, Germany
weslihan.n... kose.kooe...ooo.. cihangirr Minteeng.rrriitelnc.mo
Ramesh Manuvinakurike
Intel Labs, USA
ramesh, www.mmmanviiiuua. keê mnte..c..
Abstract
Globally interpretable models are promising approach for trustworthy in safety- critical domains. Alongside global explanations, detailed local explanations are a crucial complement to effectively support human experts during inference. This work proposes the Calibrated Hierarchical QPM (CHiQPM) which offers uniquely comprehensive global and local interpretability, paving the way human-Al com- plementarity. CHiQPM achieves superior global interpretability by contrastively explaining the majority classes and offers novel hierarchical explanations that are more similar to how humans reason and can be traversed to offer built-in interpretable Conformal prediction (CP) method. Our comprehensive evaluation shows that CHiQPM achieves state-of-the-art accuracy as point predictor, main- taining 99% accuracy non- -oniirterpeetabee models. This demonstrates a substantial improvement, where interpretability is incorporated without sacrificing overall accuracy. Furthermore, its calibrated set prediction is competitively efficient to other CP methods, while providing interpretable predictions of coherent sets along its hierarchical explanation.
Introduction
Deep Learning has made remarkable advances and is being used more widely, including in high- stakes domains like medicine ① autonomous driving (27]. Using more transparent models, e.g. those that are interpretable by-design, is promising approach to facilitate safety, robustness, and trust (53) and even required by law for some applications [61]. For domains like autonomous driving, with no expert present during inference, models with built-in global interpretability. that can generally explain their behavior, are valuable as their reasoning can robustly tested and verified before deployment. QPM and Q Q-SENN 440 41 follow that goal by enforcing very compact class representations, that are made up of general, diverse and contrastive features, which are properties of human- human-friendly explanations [35]. The generality of these features is in contrast prototypice networks like ProtoTree [37, which seem inherently interpretable, as they use the similarity between image patches crucial element of their computation. However, the space in which the similarity is
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Bronzed Cowbird Features
Images
Feature
Bronzed Cowbird
Shiny Cowbird
Shiny Cowbird Features
Figure 1: Contrastive global Explanation, comparing the class representations of Shiny and Bronzed Cowbirds for CHiQPM that represents every class with 3 of 30 features. The cowbirds differenti- ated based on the red eye.
Bronzed Cowbird Peatures
Taxt Image
Feature
Feature
Actwerry
Fand
Coubird
Figure 3: Coverage rela- tive to the set size on CUB- 2011 63 for various set pre- diction methods applied to CHiQPM with out of to- tal of 50 assigned features per class. The stars denote dif- ferent calibration or hierarchy Figure Exemplary local explanation provided by our CHiQPM, levels and are linearly inter- with the global explanation in Figure [ for a difficult test image of polated. Traversing the hi- Bronzed Cowbird with pale red eye that is not clearly visible. erarchical explanations (Fig- This leads to negligible activation of the red-eye detecting Feature ure 2 the built-in confor- The calibrated CHiQPM provides a hierarchical explanation that mal prediction method pre- communicates clear evidence for the predicted coherent set of black dicts coherent sets with com- birds (marked in bold with green edges, including the correct label), petitive efficiency CP meth- ods THR (50 or APS [46].
Auidet, +7
Groove billd
Warbler
Black capped Viren, +1
Blackbird, +3
Shiny Cowbird Bronzed Cowbird
but no sufficient evidence to differentiate between them.
computed is freely learnt which leads to class-specific prototypes 41 and similarities humans can frequently not predict (20 24). Due to these shortcomings, this work follows the goal of models like
QPM [41] to represent classes using general features.
Considering the cognitive limitations of average humans 34 QPM represents every class with the
binary assignment of very few broadly shared features, usually < 5 features. A consequence is the emergence of highly contrastive class representations. Similar to Figure [ the difference between two classes in the model's 's representation space can be concretely pointed out. like differentiating two birds via their eye, just like humans (63). While the contrastive representations are very helpful for QPM, they are also fairly rare, e.g. on average just 0.13% pairs per class CUB-2011 1. Other domains, e.g. medicine or science, can profit off of additional interpretability. When a human expert is present, they should be supported rather than replaced, notion known as human-Al AI com-

---

plementarity, While global interpretability is still beneficial in this scenario, the value of explaining the decision for a single sample rises, known as local explanation. These typically have the form of saliency maps, such as GradCAM [54), that visualize where the explainee saw support for its decision. They can also be meaningfully computed for individual features if the globally interpretable model can be decomposed into detecting general human understandable concepts. This crucial property is a key feature of our proposed CHiQPM, as demonstrated in Figures [ and 2 However, those heatmaps generally do not transport notion of certainty. Therefore, predicting a set of classes with configurable guarantees on the accuracy using Conformal Prediction (CP) [62] has emerged as a promising direction for supporting experts [56 57.. Intuitively, more classes are predicted for uncertain or less conform samples, whereas a point-predicton always predicts just one class. However, these sets typically contain larger variety of classes, that resemble the misalignment between human and machine representations.
This work introduces the Calibrated Hierarchical QPM (CHiQPM). It improves the global inter- pretability of QPM while maintaining or improving the state-of-the-art accuracy by enforcing more pairs of classes with highly contrastive class representations and adapting the training pipeline to ensure class representations made of interpretable features via the proposed Feature Grounding Loss Lieas combined with ReLU activation. CHiQPM is the first model with built-in interpretable set prediction that can be calibrated via CP, inheriting all its robust guarantees. Intuitively, CHiQPM predicts sets of classes by predicting all classes that share the dominant n features with the most likely class, e.g. predicting all the black birds in the hierarchical explanation in Figure 2 beloo the blue feature the tree level n predictor compared to CP methods, while CHiQPM can be calibrated using CP to dynamically select the appropriate level for a concrete sample. This results in a novel way of providing hierarchical local explanations and traversing them to dynamically and understandably construct coherent prediction sets similar to how human would reason. Considering the graph in F the CHiQPM found the green and blue feature, that identify black birds, but no sufficient evidence to differentiate between them. Therefore, it predicts the coherent set of various black birds, including the correct class. Holistically, the novel hierarchical local explanations answer an unprecedented range of questions simultaneuosly: 1. What meaningful features of which classes are found in this image? 2. How does each feature narrow down the set of potential predictions into increasingly similar classes? 3. Which set shall be predicted to guarantee a configurable average accuracy? 4. Which features would have needed activate stronger in order to predict smaller set with sufficient certainty?
F demonstrates the already competitive efficiency of this set
Our main contributions] are:
We present the Calibrated Hierarchical QPM (CHiQPM). It is based on heavily constrained discrete quadratic problem (QP), that selects features from black-box model and assigns them to classes. The features of CHiQPM then adapt to the optimal solution, resulting in a globally and locally interpretable model.
CHiQPM offers novel hierarchical local explanations and can be calibrated to reach target coverage with competitive efficiency while ascending through its dynamically constructed interpretable class hierarchy and selecting the appropriate level. Thus, CHiQPM can be considered an interpretable conformal predictor.
We present the Feature Grounding Loss Lisat, which, alongside an additional ReLU, leads to learning more grounded and sparser features that facilitate compact hierarchical explanations along more human concepts.
The state-of-the-art performance of CHiQPM point- and built-in interpretable calibrated coherent eet-predittor is evaluated across multiple architectures and datasets, including ImageNet-1K K (48), where the gap to the black-box baseline is more than halved.
2 Background
2.1
Interpretable Machine Learning
The field of Interpretable Machine learning can be split into models with interpretability by design and methods that aim to explain models post-hoe 4 13 21 42]. This paper introduces a model that offers interpretability by design, which is why we focus on that part. Two directions
'The code is httpaa/ttttts.//.....oooooo/o/an
ht : htwp://wwww......ooooaaaaaaaaaaa
3

---

Solve QP with Hierarchy Constraint for Feature with Cteat and ReLU
Compute Constants R.R.b
Finetune Features
Train Dense
Model with Ldiv
Calibrate
CHiQPM
and similar classes K for QP
Selection and Assignment
Figure 4: Overview of our proposed pipeline to obtain a CHiQPM
of interpretabilityy can be defined: Explaining the decision for single sample is called local, whereas a global explanation is concerned with the general behavior of a model [36]. Measuring interpretability is an unsolved task its own, but several desired criteria have been determined. A human-frite srnan-friendly explanation should be diverse, general, compact and contrastive [28 35 45 The
SENN (2) framework, which CHiQPM can be considered as, further adds grounding criteria, which refers to the alignability of learned representation with human concepts. One issue with measuring grounding is the prevalence of polysemantic neurons [11 52 60), which refers features that detect multiple human concepts simultaneously. Nevertheless, multiple metrics have been proposed to quantify the desirable aspects [40 [1), which evaluate in this work.
Globally interpretable models typically make their classification using simple interpretable model applied to more interpretable features. These features are either freely learned with priors losses to induce interpretability (38.4], supervised with human concepts in the family of Concept Bottleneck Models (CBM) 10
51 or restricted to closely resemble parts of the training data, so-called 49). Recently, the simple interpretable model is typically a very compact linear layer [8 but another example is decision tree. its most interpretable form, Prototree learns static deep decision tree to classify based on similarities to learned prototypes. Like all prototype models, these similarities are typically not predictable for a human 20 24] and the interpretability of very deep decision tree is debatable. Apart from the static tree in Prototree, CHiQPM with its sample-specific low-depth hierarchical explanation is most comparable to SI.DD-Model, Q-SENN and QPM [1). They follow pipeline of training a black-box model, making it compact, i.e. selecting features from the black box and sparsely connecting them to classes, and then fine-tuning the compact model to achieve a superior securacy-compactnes tradeoff. SLDD-Model and Q-SENN used gim-saga (65] for feature selection and their class assignment, which includes normalizing the features and is very local optimization. QPM improves upon that by formulating the selection and assignment as quadratic problem and solving it globally optimal, improving upon accuracy and interpretability via the binary comparable class representations with a fixed number of features. The incorporated constraints on compactness enable a quantitative measurement of a metric that relates to interpretabilityy While prior work [15] 16 47 primarily optimized compactness for efficiency, this and other work in interpretable machine learning (30
23
32
33
prototypes 6
31
37
38
focus on for interpretability.
2.2 QPM
QPM [41] follows a pipeline similar to Figure 4 with an auxiliary Feature Diversity Loss Laiv 39 to ensure that the initial n, feature maps activate on distinct locations of the image.
The first step is training a black-box dense model
QP Afterwards, a quadratic problem formulated to jointly find an optimal selection of ny features E {0. 13 and assignment between features and n. classes W E {0, 11]"..n;, with W () denoting W for the selected features. These variables are then optimized so that the selected features are assigned to classes they are maximally correlated to, described in the class-feature similarity matrix A E x Additionally, the problem formulation steers the feature selection towards distinct and local features, encoded the feature-featur similarity matrix Ra respectively. Additionally, the quadratic problem is
R R linear bias term constrained to result in solution with the desired level of compactness:
Sa = n
WeddSa Nowe Wc E n.)
The even sparsity can cause duplicates to arise. QPM prohibits these with an iterative optimization, that also includes an efficient incorporation of Equation ( For more details, refer the reader

---

to A1.
Finally, the features are fine-tuned with fixed W", so that they adapt to their assigned classes and detect shared general concepts, detailed in Appendix
2.3 Conformal Prediction
We calibrate our model using the Conformal Prediction (CP) 62] framework. For a more thorough introduction, refer the reader to [7] which we summarize shortly. Conformal Prediction is mathematically robust framework with minimal assumptions that can guarantee a desired error rate is not exceeded. The most popular framework and the prediction 43]. It is based on splitting off calibration data Deal to calibrate any model and guarantees an error rate of up a on data (Ftest, yhest) that is exchangeable with D
use is called split conformal
1 S P(yeest E (X(est))
(2)
Here, (ztest) describes the predicted classes for the test sample. It is constructed using a noncon- formity score s(z, c), that should relate how well the predicted label e fits to the input æ. This
calibration score is computed on the entire calibration set Deal, resulting in a set of scores S and its values are used to construct the test prediction sets:
(3) Simply put, if (1 (1-ol a) of the true labels on the calibration data were less nonconform, i.e. had a lower nonconformity score, than candidate class during prediction, will not be included in Y.
(Iteat) = fc C: :::( (Zitest, c) S Quantile(1 - -0, a,S)}
The nonconformity score s(Etest: c) can be fairly arbitrarily defined and just needs to capture some notion of conformity and have sufficiently many distinct values. that the quantile has the desired resolution. CP is typically evaluated under two aspects: a) Unconditional coverage that guarantees a minimum average accuracy on equally distributed test set. b) Conditional coverage that aims for the desired expected accuracy for all test samples, even difficult ones. The simple method THR (50) is fairly ideal for unconditional coverage. Therefore, research has focused on conditional coverage, e.g- APS [46). also in difficult settings [9] or conformal training (58). which involves directly optimizing for the size of predicted sets during training. Cortes-Gomez al. (8) even steer the prediction sets towards more coherence. To the best of our knowledge, this is the first work that proposes the notion of interpretable CP. The predicted sets are constructed by traversing hierarchical explanations, leading to coherent sets of similar classes by design, while efficiently ensuring unconditional coverage.
3 Method
The proposed CHiQPM is designed classify an input image, denoted I 33xwxb into more of n. classes, represented as c E C {ci, C22 ..., }. CHiQPM consists of a deep feature extractor, 4, which processes I to generate a low-dimensional feature vector f E RR via feature maps M E Rn7 xwM xha of dimensions иM x l.M. and a sparse interpretable final assigmment W E {0. 1) Mnjj of these features to classes. Thus, the prediction of CHiQPM can be expressed as y = W' with é = argmax(y)) being the predicted class. Compared to QPM, we apply an additional ReLU to the features = RcLU((f, so that CHiQPM only reasons positively, and negligible activations number wew of features per class, which leads to easily comparable class representations. It is easiest, when two classes share exactly newe 1 features, the differentiating factor can then be coneretely pointed out, as shown in Figure D We denote the set of these pairs with
suppressed. Notably, all classes in CHiQPM represented with the same
P= ((i,)) < < Re and w CHiQPM follows a similar pipeline to QPM, shown in Figure 4 CHiQPM improves upon QPM (Section2 2.2) with easier interpretable class representations for more of the classes via an increased [IP), directly enforced in the QP (Section 3.J). alongside a built-in interpretable conformal prediction method traversing the novel local hierarchical explanations (Section (32)) and improved fine-tuning with new Feature Grounding Loss Lieat that improves grounding and compactness of CHiQPM's explanations (Section 3.3
Twwe 1}
3.1 Hierarchical Constraint
In order to ensure a higher cardinality [TP] resulting in more easily comparable class representations, an additional constraint is added to the quadratic problem. A set IK of pairs of classes that are highly
5

---

similar in our dense model is determined, and we ensure that most of these are part of the resulting P via a two-step process. Specifically, we calculate the class-class similarity K R" based on the similarities computed for the QP A and set KK as its p. The most similar class pairs:
K = AAT Inc
K = ((.,)): Si<j S and Ki, 2 0}
(5)
Here, = sort K) p.n. describes the (2-p- )-t highest value in K. In. is the identity matrix and p is hyperparameter that controls how many classes each class should be very similar in the representations of the resulting CHiQPM, with = 0.5 enforcing on average one very similar class each. A higher p increases the number of classes that can be contrastively globally explained as in Figure []] thus improving global interpretability.. Forcing similar classes to be represented very similarly by CHiQPM further induces an efficient use of the nij features, shared concepts need only be detected by shared features. This improves the grounding of the hierarchical explanations and thus the efficiency of our built-in set prediction. However, with increasing p. the risk of adding classes to IP that do not share we 1 general concepts rises. Those pairs would cause the shared features to be less robust and thus harm performance. Thus, there can be a tradeoff between point and set prediction. Therefore, P should be using calibration data to efficiently achieve target coverage while balancing point prediction, which is ablated in Section 4.3 and Appendix [D] After obtaining IK, the similarity of each included pair is directly added constraints to the discrete optimization:
(6) We relax this constraint after finding the initial global solution to find the most optimal way for the CHiQPM to have sufficient, i.e. |TP| 2 [K] highly similar classes in its representations. This is detailed in Appendix G
(w, o w Twwe v(c,d') E KK
3.2
Set Prediction
This section describes how the proposed CHiQPM can be used predict interpretable sets Y along its local hierarchical explanation. The construction of these explanations and how they can be used to predict sets at fixed hierarchy level is first formalized. Then, we show how CHiQPM is calibrated using Conformal Prediction to predict coherent sets Y that contain the target label with an error rate (0, 1) while still traversing the explaining hierarchy.
3.2.1 Hierarchical Explanation
Our CHiQPM enables the construction of hicrarchical local explanations for a concrete test sample, as shown in Figure 2 It contains nodes for all nonzero feature activations and indicates the presence of all classes that are assigned to at least one of the shown feature-nodes. For every class e with its shown in the order of their activation, which can be interpreted as reasoning from the more clearly visible feature like the neck in Figure to the less certain features, such as the pale red eye. The class node then attached to its last activating feature and the class would be predicted if CHiQPM's calibration or fixed level determines that this feature and all its descendants should be predicted. To formalize predicting at a fixed level, we define the order of activations For of the assigned features for every class c, where io is ordered so that: V i.j E (L,..., where <
assigned features F {1....,,7 n2 ]""e, the features
f Additionally, we introduce the indicator function of at depth n E {I,..., which indicates the class e shares the same top features with the predicted class é:
(7)
8 {t)
I. if = 189 Vi e (I....,n)
(8)
otherwise
Thus, predicting at a fixed depth of n features shared with the predicted class è is:
y f C: =1
For example in the graph in Figure 2 Yl contains all shown class nodes, Y² the five classes that share the green and blue feature and finally 3 T {c), as only the predicted class è shares all features with itself. Note that the stars for the fixed depth line in Figure Jindicate predicting with a fixed n E {1..... ]owe). For more compact explanations, we restrict the graphs in the main paper to include only those classes which share the most activating feature with the predicted class. However, even for large datasets, full graphs are still informative and included in the appendix, along further visualization details in Appendix C

---

3.2.2 Nonconformity Score
As introduced in Section 2.3 the nonconformity score is crucial aspect of any Conformal Prediction method. In order construct the prediction sets for CHiQPM by going up the class hierarchy, we compute the nonconformity score based on how similar the class is in the hierarchy to the initially predicted class é, as we are ascending the hierarchy that led to Note that the introduced method offers all the guarantees of CP, as only different nonconformity score is used. For every class e, we propose to use the activations of the features in the shared path down the tree as nonconformity score:
Row
Sup (c)
69533
(10)
Note that of ensures that all i features are shared and the dependency on the input sample of all variables is omitted for brevity. We call this simple nonconformity score up as goes strictly up the tree.
Subtree Selection The conformal predictor can predict with more granularity and achieve its guarantees when the prediction can also go down towards only some subtrees below the feature node determined by (cp That also allows CHiQPM predict those descendants preferably that have some support beyond the shared path, e.g. choosing to predict only the cowbirds in Figure 2 Therefore, we extend the nonconformity score to also account for the activation of the feature the point of diversion:
div
Spel = Sup(c)= fint
(11)
aF=)
Limited Level Finally, the maximum number of levels the set is constructed from is limited to ensure efficient sets. Towards that goal, the minimum reachable error rate a on the calibration data for each fixed level n is calculated according to Equation (9). The conformal prediction then limited to the highest level n limit that still reaches the target coverage defined by To ensure the limitation, we limit Sup to n limit, and multiply the score with the indicator function o indexed at -iima This ensures all classes that were not correctly predicted under n limit
get the most nonconform score of O:
Tewe
(c))
8 Alimni
8950
fai
(12)
f=
Limitation
Limited sup
Subtree Selection
3.3 Feature Grounding Loss
This section presents our novel Feature Grounding Loss Lreat- Its motivation is shown in the example in Figure 5 In this example, two similar classes share 2 of their 3 features, hence they are in P. This is relevant, as most classes in each CHiQPM have a similar class due to the constraints described in Section 3.1 For concrete training example, where one of these is the ground truth, Cross-Entropy loss only causes a significant gradient on the differing Ground Truth Exclusive (GTE) feature. Therefore, the distinguishing feature GTE is pushed also activate on other concepts of the ground truth class, instead of just activating on the general concept that would differentiate them. To alleviate that issue, we propose the Feature Grounding Loss Lieat.
Eier
Si FI
-EEF
f IF]
Lieat
(13)
max (f*)
where F and F E (I,...,n?]"" -nwe indicate the indices of the features of the ground truth class or
nwe those not assigned to respectively. Lieat describes the difference in average activation between F and F, scaled by the maximum activation to prevent increasing activations. As shown in Figure 5 has the same gradient for all positively assigned features and also has a nonzero gradient for all features F. Therefore, it encourages all the features of the ground truth class to detect more general concepts while also inducing sparsity in the feature activations, enabling the visualization of the entire hierarchical explanation. We add Lieat to our overall training loss during fine-tuning with weighting Aeatt
7

---

Table I: Comparison on Accuracy, Compactness, Contrastiveness and Structural Grounding. Compact describes the number of features nj and features per class Re.. It is binned into very compact +
(Nwe and n 50), medium and the baseline, denoted (nwe 2048 and nj 2048), exact
figures in Table 18 Figure shows the increasing gap when further raising compactness. Among
more compact (o above) models, the best result marked in bold, second best underlined.
Accuracy 1
Com-
Contrastiveness
SG
Method
CUB
CAR
IN
pact
CUB
CAR
IN
CUB
Dense Resnet50
86.6
92.1
76.1
74.4
75.1
71.6
34.0
glm-sagas
78.0
86.8
58.0
74.0
74.5
71.7
2.5
PIP-Net
82.0
86.5
99.5
99.5
6.7
ProtoPool
79.4
87.5
76.7
78.9
13.9
L.DD-Model
84.5
91.1
72.7
+
87.2
89.7
93.4
29.2
Q-SENN
84.7
91.5
74.3
+
93.0
94.2
92.6
23.4
QPM
85.1
91.8
74.2
+
96.0
97.7
89.3
47.9
CHiQPM (Ours)
85.3
91.9
75.3
+
99.9
100
99.9
75.0
Experiments
Following QPM (Section 2.2 we
evaluate method CUB-2011, Stanford Cars 26 and ImageNet- 1K. CUB-2011 and Stanford Cars are the most commonly used datasets for interpretability, while ImageNet- IK is suitable to demonstrate how the method scales larger prob- lems with more real-world applica- tions. CUB-2011 I includes human an- notations of relevant concepts for ev- ery image. which makes it suitable for evaluating the alignment between human representations and the ones learned. As the proposed method can applied to any backbone, we show results Resnet50 (18),
W
Classes Sim Misc
CE: GTE
0.6
CE: GTS
GTE
CE: GT
DE: GTS
GTS
GTS
1
o
0.2
Typical Activation
Features
GT
1
1
-0.2
GT
1
0
10
Activation
Figure 5: Gradient on features f for a train sample labeled GT for toy example with 3 classes and features, with W shown left. At the average activation on the CUB dataset, the Ground Truth Exclusive (GTE) feature has roughly 4000 times higher gradient than the other assigned features, which are shared with Sim, Ground Truth Shared (GTS).
Resnet34, Inception-v3 [59] and Swin- Siii-TTraffomerr Smallllll---mlll [29 to allow
an easy comparison with previous work. The main paper focuses re- sults for Resnet50, always reporting the across 5 random seeds, with 3 for ImageNet- 1K More results are included in Tables to and 18 to 25 demonstrating the robustness of our method across random seeds and architectures. For implementation details, we generally followed QPM [41]. but report details and slight deviations in Appendix A Specifically, all models that follow pipeline similar to ours, MLDDD-odel, Q-SENN and QPM, use the same parameters for the dense training. As usual in literature, wewe and n 50 are set if reported otherwise. Further, we generally the density parameter for class hierarchy p 0.5, as it is sufficient to demonstrate the improvements in built-in set prediction without sacrificing accuracy as point- print-predictor. Finally, Ateat = 3 is set as higher values cause reduced accuracy.
4.1 Metrics
CHiQPM is designed improve upon QPM (Section 2.2 primarily via more easily interpretable class representations, being able to produce meaningful hierarchical explanations and by offering the built-in interpretable calibrated set prediction. Therefore, evaluate CHiQPM across all QPM

---

Table 2: Average Set Size Y of CHiQPM calibrated to reach various coverages 1 comparing
different conformal prediction methods. All methods are very close or reach the desired coverage.
E4 Inter- pretable
CUB
CARS
INET
Method
n=0.12
e=0.1
e=0..755
o=0.05
n=0.075
e=0.05
n=0.00225
c=0.22
o=0.2
o=0..15
o=0.
Ours
1.22
1.73
2.94
9.05
1.05
1.25
8.25
1.10
1.42
3,25
4.58
8
Fael
4.62
6.15
9.53
29.4
3.62
5.95
28.4
8.14
11.3
17.9
30.5
fap
V.
3.03
3.91
8.87
18.7
2.32
3.27
17.9
4.36
6.23
11.8
31.4
THR
X
1.16
1.32
1.67
2.41
1.02
1.15
2,09
1.05
1.16
1,40
1.87
APS
x
6.30
7.20
8.54
11.3
5.64
6.83
9.61
16.7
18.9
22.1
26.8
metrics in addition to the accuracy as point predictor in relation to its compactness. For evaluating the performance as set predictor, we report the size of the predicted sets, when calibrated to reach a specific accuracy or coverage. Finally, the annotations in CUB-2011 are used to compute the Set Coherence of the predicted sets and also measure the grounding of the features via the Alignment [40] metric, alongside QPM's Structural Grounding. Following QPM, we compute the ground truth class- class similarity matrix ot AA" using the annotated average class attributes A E [O, with columns A. [0, 1| where Aej indicates the fraction of images with label c, in which attribute j is annotated to be present. g enables quantifying the similarities of classes that are predicted together, the novel Set Coherence SC:
x312 x
((Y(etest)) = (wage |., Y(Iust) c'
SC
mean
U K(Y(Iteat)) (14)
Ziest Euuc
defined as the mean over all the class similarities of classes that the predictor predicts jointly. Thus, a higher value indicates sets with more similar classes in reality. For brevity, we focus on a subset of more strongly affected metries in the main paper but report results on all of them in the appendix alongside every metrics formulation in B
4.2 Results
This section discusses the main quantitative results of our proposed method. Further qualitative examples are included in the appendix. Figures 14 to 20 showcase global explanations, Figures 21 to 30 include the novel local hierarchical explanations and Figures 31 demonstrate how the features of CHiQPM are general concept detectors. The accuracy as point predictor along the generally preferable qualities of Compactness, Contrastiveness and Structural Grounding is shown in Table CHiQPM shows tfate-of-the-art accuracy for compact point predictors. Further, it scores nearly perfectly on Contrastiveness. CHiQPM learns features that can be more clearly separated between active and inactive than even the class detectors of PIP-Ner (8), indicating gap between the Rel.U-induced minimum of and the activations where a relevant concept is found. The clear distinction between active and inactive enables saliency maps, like in Figures [and] 2 to also transport activation rather than just location without a reference test image and therefore enables extensive local explanations in practice. The details are explained in Appendix C.2 Finally, Structural Grounding quantifies that the additionally added pairs via Equation (31) are also similar in reality and thus lead to more grounded class representations. The state-of of-te-of-thearrr accuracy point predictor paves the way for accurate set prediction along the hierarchical explanation, as the sets are conditioned on the predicted class.
For calibrating Conformal Prediction methods, the first 10 test examples per class are split off into the calibration data Deat- That way, we can use the same models for evaluating point and prediction. Notably, applying Split Conformal Predictions requires exchangeability between calibration and test data. Our experimental setup is designed to ensure this exchangeability, as detailed in Appendix L As comparable CP methods, THR [50] and APS [46] are used, as they are applicable without hyperparameters and broadly used [7]. Table Sableeeomparce our built-in CP method with these and also with the two simpler nonconformity scores Ssel and Sup- Evidently, our proposed nonconformity score that restricts the sets to be constructed by going up the hierarchical local explanations shows competitive efficiency to THR for higher eгror rate a and approaches APS for lower values. The reason can be seen when comparing our approach with the simpler 8sel that does not restrict the tree level to n limit. With lower a, the gap decreases, as n limit has to be set more loosely, allowing larger and therefore inefficient sets, which can be gauged from Figure Figure visualizes the Set Coherence (Equation [4)) of multiple models and CP methods and also indicates the set size for

---

Table 3: Impact Leest on CUB-2011. effectively increases the Feature Alignment while reducing the fraction of nonzero features on the test data.
Feature Alignment
Feature + Sparsity
Method
Acc t
CHiQPM (Ours)
85.3
3.8
22.3
w/o Lisat
85.3
3,0
39.9
and LeI-Regularization
85.3
3.4
31.3
and LI for
85.1
3.6
26.4
desired a. As expected, CHiQPM predicts the most coherent sets, as it moves up the hierarchical local explanation constructed from class representations with Structural Grounding. CHiQPM further clearly surpasses QPM in efficiency and Set Coherence, making it the only model with efficient, calibrated, built-in, and interpretable coherent set prediction.
4.3 Ablation Studies
This section investigates the impact of p. Lieat and summarizes the abla- tions shown in the appendix. Figure 6 demonstrates how a higher p leads to an increased [IP), more Set Coher- ence and efficient sets. For example, = 0.075 can be reached with 6.3 classes per prediction with == or 2.9 with م 0.5, as n can be higher. Note that p can be opti- mized calibration data, steering the CHiQPM towards efficiently reach- ing desired a, demonstrated in Ap- pendix [D] Further visualizations that explicitly relate p to the efficiency included in the appendix, Figures 7 8 The impact of Lisat is shown in LableBandd also contrasted with a ci-regularization on the features f" as another form to induce sparse features. The degree of regularization was cho- sen so that a further increase greatly reduces the accuracy. Without sacri-
CHiQPM (o (0=0.1)-
90
CHiQPM (o 0.3) [P] 91
CHiQPM 005) [P] 120
85
Set Coherence
CHiQPM (0=
145
80
QPM (p 0)- 266
Dense Baseline THR
75
Dense Baseline APS
70
65-
0
10
15
20
25
30
Set Size
Figure 6: Set Coherence CUB-2011 for all classes pre- dicted together. Across the set sizes, the built-in set pre- diction produces the most coherent sets. The stars denote different a E f0.12, 0.1, 0.075, 0.05} used for calibration. Note that increased set size corresponds lower a. Therefore, the competitive efficiency is also visible.
ficing any accuracy, the proposed CHiQPM reaches the highest Feature Alignment and best sparsity and thus validates Lleat as suitable formulation to induce compact grounded hierarchical local explanations, with Appendix H visualizing the impact of Areat. As further validation for our method, Appendix ssoows the benefit using a sample-specific hierachy instead dynamic one. Addition- ally. Appendix K quantifies that CHiQPM's features quantifiably localize more on relevant regions in the image. Finally, further ablations show a larger gap between CHiQPM and QPM even higher compactness (Figure 9 and the strong positive impact of ReLU (Appendix I].
5 Conclusion
This work introduces the Calibrated Hierarchical QPM (CHiQPM). Faithfully following its grounded globally interpretable class representations, CHiQPM provides hierarchical local explanations. CHiQPM is calibrated as a form of built-in interpretable Conformal Prediction to traverse the hierarchy at test time and predict of coherent classes, similar to how human reasons, which can be a step towards human-Al complementarity. Finally, CHiQPM's improved global and additional novel form of local interpretability come with state-of-the-ard accuracy as compact point predictor and efficiency on with non non-coherent set predictors even on ImageNet- IK, ensuring broad applicability.
10

---

Acknowledgments
This work was supported by the Federal Ministry of Education and Research (BMBF), Germany, under the Al service center KISSKI (grant no. 01IS22093C), the Deutsche Forschungsgemeinsccattt (DFG) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC2122), the MWK of Lower Sachsony within Hybrint (VWZN4219), the European Union under grant agreement no. 101136006 - XTREME. The work has been done in collaboration and partially funded by the Intel Corporation. This work was partially supported by the German Federal Ministry of the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (GreenAutoMLAFAS project no. 67K132007A).
References
[1] Ahsan, M. M., Luna, A., and Siddique, Z. Machine-learning based disease diagnosis: comprehensive review. In Healthcare, volume 10, pp. 541. MDPI, 2022.
[2] Alvarez Melis, D. and Jaakkola, T. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018.
[3] Ayonrinde, Position: Interpretability bidirectional communication problem. In ICLR 2025 Workshop Bidirectional Human-Al Alignment.
[4] Bau, D., Zhou, B.. Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6541-6549, 2017.
[5] Böhle, M., Fritz, M., and Schiele, B. B-cos networks: Alignment is all need for interpretabil- ity. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp.
2022.
[6] Chen, C.. Li, O., Tao, D., Barnett. A., Rudin, C., and Su, J. K. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
[7] Correia, A., Massoli, F. V. Louizos, C., and Behboodi, A. An information theoretic perspective on conformal prediction. In The Thirty-eighth Annual Conference on Neural Information Pro- cessing Systems, 2024. URL / ntt/ htt.://orrmmmigggggggee d= 3m9
[8] Cortes-Gomez, S., Patiño, C. M., Byun, Y., Wu, S., Horvitz, E., and Wilder, B. Utility-directed conformal prediction: A decision-awarr framework for actionable uncertainty quantification. In The Thirteenth International Conference Learning Representations, 2025. URL https
//openreview.nog net formm? ho-
[9] Ding, T., Angelopoulos, A., Bates, S., Jordan, M., and Tibshirani, R. J. Class-conditional conformal prediction with many classes. Advances in Neural Information Processing Systems, 36, 2024.
[10] Dominici, G., Barbiero, P., Giannini, F., Gjoreski, M., Marra, G., and Langheinrich, M. Coun- terfactual concept bottleneck models. In The Thirteenth International Conference on Learning Representations, 2025.
net feceet/ffoummmfd www.orowwwwwwwwiiiiiioeoo.
[11] Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z. Lasenby, R., Drain, D., Chen, C., et al. Toy models of superposition. arXiv preprint arXiv:2209. 10652, 2022.
[12] Farrell, R. Cub-200-2011 segmentations, Apr 2022.
[13] Fel, T., Picard, A., Béthune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2711-2721, June 2023.
11

---

[14] Gildenblat, J. and contributors. Pytorch library for cam methods. https://github.com 2021.
[15] Glandorf, P. and Rosenhahn, B. Pruning by block benefit: Exploring the properties of vision transformer blocks during domain adaptation. In International Conference on Computer Vision Workshop, 2025.
[16] Glandorf. P., Kaiser, T., and Rosenhahn, B. Hypersparse neural networks: Shifting exploration to exploitation through adaptive regularization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1234-122 1243, 2023.
[17] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https://ww: www gurobi comm
[18] He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
[19] Hewitt, J., Geirhos, R., and Kim, B. We can't an't understand ai using our existing vocabulary. arXiv preprint arXiv:2502.07586, 2025.
[20] Hoffmaan,, A., Fancomi, C., Rade, R., and Kohler, This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks, 2021.
[21] Kaiser, T., Norrenbrock, T., and Rosenhahn, B. UncertainSAM: Fast and efficient uncertainty quantification of the segment anything model. In Forty-second International Conference on Ma- chine Learning. 2025. URL https:ttp...... : http///ppeeeeeeeeeeeeeeeeeeeeeee net/ forum w id= J q
[22] Kalibhat, N.. Bhardwaj. S.. Bruss, C. B., Firooz, H., Sanjabi, M., and Feizi, S. Identify- ing interpretable subspaces image representations. In Krause. A., Brunskill, E., Cho. K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Con- ference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15623-15660 PMLR, 23-29 Jul 2023. URL htttps://proceee/// / v202 (et22/aalbbat2222 html
mlr press/
[23] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pp. 2668-2677 PMLR, 2018.
[24] Kim, S. S., Meister, N., Ramaswamy, V. V., Fong. R., and Russakovsky, O. Hive: evaluating the human interpretability of visual explanations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pp. 280-298. Springer, 2022.
[25] Koh, W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, Concept bottleneck models. In International Conference Machine Learning, pp. 5338-5348. PMLR, 2020.
[26] Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
[27] Kuutti, S., Bowden, R., Jin, Y., Barber, P., and Fallah, S. survey of deep learning applications to autonomous vehicle control. IEEE Transactions on Intelligent Transportation Systems, 22(2): 712-733, 2020.
[28] Lipton, P. Contrastive explanation. Royal Institute of Philosophy Supplements, 27:247-266. 1990.
[29] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVE international conference computer vision, pp. 10012-10022, 2021.
12

---

[30] Lou, Y., Caruana, R., Gehrke, J., and Hooker, G. Accurate intelligible models with pairwise interactions. Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 621-631,22 2013.
[31] Ma, C., Zhao, B.. Chen, C., and Rudin, C. This looks like those: Illminating prototypical concepts using multiple visualizations. Advances in Neural Information Processing Systems, 36:39212-39235, 2023.
[32] Marconato, E., Passerini, A.. and Teso, S. Glancenets: Interpretable, leak-proof concept-based models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: htt.tpss//ppeneview..co www.:eevwwww.ee/oormm..... forummt//ooormoooo
J72Y9175GoG
[33] Margeloiu, A., Ashman, M., Bhatt, U.. Chen, Y., Jamnik, M., and Weller, A. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 2021.
[34] Miller, G. A. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2)811 1956.
[35] Miller, T. Explanation artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019.
[36] Molnar, C. Interpretable machine learning. Lulu. com, 2020.
[37] Nauta, M., van Bree, R., and Seifert, C. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14933- 14943, 2021.
[38] Nauta, M., Schlötterer, J., van Keulen, M., and Seifert, C. Pip-net: Patch-based intuitive prototypes for interpretable image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[39] Norrenbrock, T., Rudolph, M., and Rosenhahn, B. Take 5: Interpretable image classification with handful of features. In Progress and Challenges Building Trustworthy Embodied AI, 2022.
[40] Norrenbrock, T., Rudolph, M., and Rosenhahn, Q-senn: Quantized self-explaining neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21482-21491, 2024.
[41] Norrenbrock. T., Kaiser, T., Biswas, S., Manuvinakurike, R., and Rosenhahn, B. QPM: Discrete optimization for globally interpretable image classification. In The Thirteenth International Con- ference on Learning Representations, 2025. htt.ss/ts:///// httpt::::///peeevviiie ew net. forum?
id=GlAeLO T8LX
[42] Oikarinen, T. and Weng, T.W.. CLIP-dissect: Automatic description of neuron representations in deep vision networks. In The Eleventh International Conference on Learning Representations, 2023.
net./ / rumm? d=1PW 1 wWHc1V
[43] Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, Inductive confidence machines for regression. In Machine learning: ECML 2002: 13th European conference machine learning Helsinki, Finland, August 19-23, 2002 proceedings 13, pp. 345-356 Springer, 2002.
[44] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T. Lin. Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z.. Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chin- tala, S. Pytorch: An imperative style, high-performance deep learning library. In Wallach, H., Larochelle, H., Beygelzimer, A., d' AAlché-Buc, F., Fox, E., and Garnett, R. (eds.), Advances in Neurul Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL.
neurips www.rrisscccppeepe cc/paper/ htt.ppprrr..rrrippssccppp/eerrr 9013-pytorch-an-Bie otg.::/eeee.i.tttteeeoeeeeeeeeer
pdf
13

---

[45] Read, S. J. and Marcus-Newhant A. Explanatory coherence in social explanations: A parallel distributed processing account. Journal of Personality and Social Psychology. 65(3):429, 1993.
[46] Romano, Y., Sesia, M., and Candes, E. Classification with valid and adaptive in Neural Information Processing Systems, 33:3581-359) 2020.
adaptive coverage, Advances
[47] Rosenhahn, B. Optimization of sparsity-constrained neural networks mixed integer linear program: Nn2milp. Journal of Optimization Theory and Applications, (99(3)931-9544 2023.
[48] Russakovsky, O., Deng, J., Su, H., Krause, J.. Satheesh, S., Ma, S., Huang, Z.. Karpathy, A., Khosla, A.. Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211-252,
2015. doi:
[49] Rymarezyk, D., Struski, t.., Tabor, J., and Zieliñskii, B. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420-1430, 2021.
[50] Sadinle, M., Lei. J., and Wasserman, L. Least ambiguous vet-velued classifiers with bounded error levels. Journal of the American Statistical Association, 1 1146555)22332255 2019.
[51] Sawada, Y. and Nakamura, K. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758-411 -1775,, 2022.
[52] Scherlis, A., Sachan, K., Jermyn, A. S., Benton, J., and Shlegeris, B. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022.
[53] Schier, M., Schubert, F., and Rosenhahn, B. Explainable reinforcement learning via dynamic mixture policies. In 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025.
[54] Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
[55] Selvaraju, R. R.,, Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: visual explanations from deep networks via gradient-based localization. International journal of computer vision, 128:336-33 -5599, 2020.
[56) Straitouri, E. and Rodriguez, M. G. Designing decision support systems using counterfactual prediction sets. In Proceedings of the 41st International Conference on Machine Learning. ICML'24. JMLR.org, 2025.
[57] Straitouri, E., Wang, L., Okati, N., and Rodriguez, M. G. Improving expert predictions with conformal prediction. In International Conference on Machine Learning, pp. 32633-32653. PMLR, 2023.
[58] Stutz, D., Dvijotham, K. D., Cemgil, A. T., and Doucet, A. Learning optimal conformal classifiers. In International Conference on Learning Representations, 2022. URL | //openreview.co net/ fttp://ffrmmmmddt000 4LKFVX
[59] Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-281888 2226, 2016.
[60] Templeton, A. Scaling monosemanticity: Extracting interpretable features from claude sonnet. Anthropic, 2024.
[61] Veale, M. and Zuiderveen Borgesius, Demystifying the draft eu artificial intelligence act acalanalysing the good, the bad, and the unclear elements of the proposed approach. Computer Law Review International, 22(4))97 112, 2021.
[62] Vovk, V., Gammerman, A., and Shafer, G. Algorithmic learning random world, volume 29. Springer, 2005.
14

---

[63] Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech- uesd dataset. 2011.
[64] Wei, H. and Huang, Torchep: A library for conformal prediction based on pytorch, 2024.
[65] Wong, E., Santurkar, S., and Madry, A. Leveraging sparse linear layers for debuggable deep networks. In International Conference Machine Learning, pp. 11205-11216. PMLR, 2021.
[66] Zhang, J., Bargal, S. A., Lin, Z., Brandt, J., Shen, X., and Sclaroff, S. Top-down neural attention by excitation backprop. International Journal of Computer Vision, 126( (0011000044-1000 2018.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The Built-In Hierarchical Explanations, e.g. Figure 2 are indeed novel. Figure 12] showcases improved set coherence and improved global interpretability via more class pairs, denoted in the legend. Table [ shows improvements in interpetability metrics and performance as point predictor and Table the competitive performance as interpretable conformal predictor. Table demonstrates the positive impact of the novel Feature Grounding Loss.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
That is discussed in Appendix L
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There is theoretical result. Conformal Prediction only assumes exchange- ability which we mention in Section 2.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper extends QPM, which offers published code (ttpp:: //g thub
com/ThoomssNor QPM). The methodical extensions of Equation 6 are straightforward integrate into QPM and all parameters
clearly described. Similarly, the proposed nonconformity score of Equation (12) can be directly used with torchop for use in conformal prediction. The code will be published after acceptance and all used parameters clearly described this work.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Unfortuntately, we are unable to submit the code alognside the paper, but will release it after acceptance. We want emphasize that the experiments are clearly described modifications to published code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
Dublic/quidee /ete//Coeeuubttme ssisns sionPolicyy olicy) for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
wttp:ggiddeecceeeeeeeeeee Codee cerionnPolione for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main paper includes the novel parameters and Xfeat and why their values are chosen in Section [4 and describes that all other parameters mirror the previous work QPM, with the only differences being outlined in Appendix A The datasets all have default split which is used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
18

---

Justification:
Due to space limitations, we do not report standard deviations in the main paper, but include standard deviations for the tables with main results the appendix. Appendix F details what variability they capture and how they are computed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is discussed in Appendix O
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpt:://eetrrr wwtppccccp/ceeeeeoccccccc csGuide! nes? ies?
Answer: [Yes]
Justification:
We adhered to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
19

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: That is discussed in Appendix N
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model does not pose such risks, but its interpretability enables thorough inspection before release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We follow standard practices and cite every paper whose datasets we use, as well as the code packages we used. We were unable to find the correct license for any the
used datasets.
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw sww. thcode. com/ / data asets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
21

---

Justification:
No research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [NA]
Justification: Only used for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy settss:/neeuriss.cccoooaa LLM) for what should or should not be described.
2025/
22

---

A Implementation Details
For implementing CP, we utilized the torchep [64] package and implemented the model using Pytorch [44]. Note that all details will also be clear in the published code.
The QP is solved described, with the additional constraints from Section but with two relaxations based on observations: First, the MiP--ap of the discrete optimization in Gurobi [17 can be relaxed without significant effect the resulting metries, hence set it to 1%. Second, due to the high number of classes, n2 = 1000, the resulting assignment for ImageNet- 1K contains more pairs IP| than desired anyway. Therefore, ImageNet- After solving the QP, we further fine-tune the model for 70 epochs using the learning rate schedule from Q-SENN, fine-tuning as opposed to QPM, as quantified in Appendix [F.]] The longer training with higher learning rate facilitates that the features can be reorganized according to the imposed hicrarchical structure. We set Aieat = 3. but skip this loss during the final 10 epochs, as its desired effects are induced without reducing top-1 Accuracy. All other steps of the pipeline equal QPM 4T. By mistake, the subtrahend in Equation (13) was scaled by a factor of 2 in all experiments. We repeated the experiments in Appendix HH with the correct implementation and observed the expected results with negligible impact of the mistake. Therefore, we did not rerun every experiment.
do not add any pairwise constraints to the QP for
observed significantly larger changes in the features during the
Note that for accuracy point-predictor and for é for conformal prediction, the argmax of y is considered the predicted class, leading to the lower index in case of a tie.
In order to apply our novel CP QPM, we subtract the minimum of all features of the features, ensuring that reasoning is restricted down the hierarchical explanation and unaffected by slightly varying μ ,
B
Metrics
This section defines the metrics that are not introduced in this work for the first time.
Following QPM [1]] we measure the general qualities of learning diverse, general and contrastive features via SID@ @5
Md
M
Ed Sij
(15)
ias h.
Σ
M
Em,
mmn
ha Lisl
Wi 1=
max
($1 b.j
$2
25
SID@5
i.j"
(16)
5
Class-Indepenn Class-nndeenndence T.
n
レ== 19 min a.)
max
(17)
71
j=1 (Fz.d min f(a))
and Contrastiveness:
n
Contrastiveness
Overlap(A],
(18)
d=1
SID@5 describes how diversely the top weighted features activate for an average test sample. Class-Independence measures which fraction of each features' activation does not activate on its most related class and low Inaeeenndeeedeecc indicates that features are class-detectors. Finally, Contrastiveness measures the overlap between two Gaussian distributions, parametrized by ut. of ug, o2, into the feature distribution f.d- A high Contrastiveness indicates features that can be split into activating and not activating.
Using the computed c, Structural Grounding classes reality, gr AAT are represented in the class representations Model WWT:
41 quantifies how similar the top 25 most similar
EcdeCsm M/-Model
StrueturalGrommding
(19)
Eed' 1t.tt
23

---

The Feature Alignment r [40] between the learned features and these attributes is also measured to estimate if the features F
ER Ro x are more human-like:
15t
Flrain
ptrain 1.
(20)
0,5
Pat
Pa
iEPa
iEPa-
n²T
max A50
(21)
n
ptrain 13
miny Eirain
Visualization Details
C.1 Graph Visualization
This section discusses the details of how the hierarchical explanations in the form of graphs are visualized, e.g. for Figure 2 and also the global hierarchical explanations in Figures 23 and [4] Generally, the content of the graphs is explained in Section 3.2.1 To indicate the activation of each feature i, the radius of each feature node scales with its fraction of the maximum activation fr (uxx(f")sccalee where Tscale constant.
Using colors, we add additional information to the graphs. The nodes are colored according to the saliency maps, that are used in the global explanation and also as part of the local explanation. All nodes, that correspond to features not visualized saliency map, represented in gray. Further, use color to indicate the actual prediction done. Only those classes are predicted by CHiQPM, for which colored path coming from the root of the graph connects to the class nodes.
Finally, the class labels at each node are summarised if multiple classes are at the same node, i.e. sharing the same top-n features, but not having any activation on their other assigned features. To summarise the class names, the text description "ClassName, + xx used. Here ClassName is chosen from the classes at that node, usually the name with the shortest words to reduce horizontal overlap, and x refers the number classes not called ClassName at this node.
C.2 Visualizing saliency maps
This section explains how the saliency maps for the individual features as part of global and local explanation are visualized.
Typical saliency map methods and available implementations [1] like GradCAM S4
only focus
on showing where support for decision is found. As they are used showing why decision was made, they do not have to transport activation. Therefore, they overlay the computed saliency map
S = on the image scaled to the full range of values irrespective of activation to get the Saliency explanation I
I3 3x
I NoActivation = ColorMap( max(S)
)8 (1 (1- B
(22)
NoActivation
I
(23)
NnActivation
max
Here, ColorMap converts the spatial information into RGB values and / defines a weighting between image and saliency map. While typically jet is used, we use individual colors in this work, scaled using gamma correction to match the impression of jet, as decompose the decision process into detecting individual features or concepts. Notably, different saliency methods differ in how they compute S.
Similar to GradCAM, we the feature maps M R") xwa xhu the basis for our visualization, as they directly cause the activations our model uses and are also evaluated and steered to be diverse. However, we visualize the map dd E R x has for every feature individually rather than a weighted sum because every feature can and should be interpreted on its own.
Different to prior work, CHiQPM has near perfect contrastive features, as measured by Contrastive- ness. That means that for concrete sample, the feature d can be clearly sorted into the inactive
24

---

distribution Nf or the active distribution Nog We make use of that distinction and especially the mean activation of active samples μ to produce saliency maps that indeed already transport whether the feature activated or not. Specifically, the Saliency explanation for single feature is computed following:
M
d
Activation
ColorMap(
minl
1))8+ 1
(24)
max (M*)
P2
Activation
ra
(25)
max I
Activation
1) scales the saliency map based on how active the feature is. Due to the perfect Contrastiveness, the threshold nd is a good indicator how objectively active the feature is. Notably, this form of transporting confidence can be applied to a single test image, as the actíve threshold 2 is computed on training data. This contrast to how QPM [1] scaled activations for the same features across images.
Here, the term min( f2
12
In our formulas, we omit how all feature maps are linearly scaled from their feature map size the input image size.
D Impact of Tree Density p
91
933
Accuracy
92
CHIQPM 0.1) PP] 51
CHiQPM (p 0.3)
90-
CHIQPM (==0.5)- - 120
CHiQPM (p 0.7) [P] 145
80
QPM (p 0)
Set
Size
Figure 7: Accuracy over set size using the built-in conformal prediction for varying p. The stars denote different (0.12, 0.1, 0.075, 0.05) used for calibration. Better fixed level performance, shown in Figure leads to more efficient calibrated sets due to being able to limit the depth via limit.
This section discusses the impact of and provides the detailed relation between and the efficiency of the hierarchical and calibrated predictions. As noted in Section 4.3 Figure demonstrates how higher leads to an increased [PP,, more Set Coherence and efficient sets. Figures] 7 and explicitly relate p to the efficiency of the resulting predictor and visualize the relationship between fixed level efficiency in Figure [ and calibrated efficiency in Figure 7 As shown in Figure in order to reach reach 0.075, CHiQPM can typically be limited to n limit 2. However, for p 0.I. it does
92.5% accuracy with fixed n 2 and therefore has to be calibrated with n limit 1. This causes the significant increase in Set Size and therefore inefficiency for 0.075, from 2.9 with the default configuration of 0.5 to 6.3 with p 0.1, as clearly visualized in Figure 7 Therefore, optimizing p using calibration data for desired a an effective option. However, note that the point- pinnttrreiiting performance starts to deteriorate slightly when further increasing beyond the start of the graphs in Figure g This likely due to classes being forced to represented very similarly, even though they do not share Towr general concepts. The [P] is further evidence for that, as the number of unenforced pairs, [PP - p * ne. declines, with just 5 for = 0.7. Thus, the optimal value for p also depends on the similarities of the classes in the dataset.
0.5, as indicated by
25

---

96-
Accuracy
92
CHiQPM (p= 0.1)
CHIQPM (3=
86
CHiQPM 0.5)
CHiQPM 0.7)
QPM (o 0) 20
6
15
20
Set Size
Figure 8: Set Accuracy of CHiQPM with Twe on CUB, when predicting with fixed level in the sample specific hierarchy. Each mark represents one level.
85.0
Accuracy [%]
84.0
ChiQPM Ane
ChiQPM Die
QPM with Dac
83.0
QPM with Dae nj
20
30
40
60
Number of Features ny
Figure 9: Accuracy on CUB in relation to Compactness: CHiQPM uses the allowed low compactnesss more efficiently, both when scaling Thee with n' and when not. When n becomes higher, restricting
Tww 10
leads to a less efficient use of some features.
E
Detailed Results
Table 4: Impact of Relu and Normalization on CUB-2011 with Resnet50
Method
Acc
C-I
Contrast.
Feature Alignment
CHiQPM (Ours)
85.3
94.1
99.9
3.8
w/o Creat and ReLU
84.8
96.7
97.1
1.9
w/o Norm
83.9
95.1
96.2
3.2
This section contains more results to demonstrate the quantifiable high performance of our CHiQPM.
First, Figure II visualizes how QPM performs with different CP methods. When comparing it to Figure CHiQPM's drastic improvement is evident, making interpretable coherent set prediction competitively efficient. Further, Figure 12 demonstrates how the built-in interpretable coherent set prediction predicts more coherent sets than conventional CP methods, even for the same CHiQPM. However, notable, that THR predicts significantly more coherent sets in that scenario, as applied to our CHiQPM with Structural Grounding.
26

---

Finally, we include the results on Resnet50, Resnet34, Swin- Swin-Transfomee Transformer-small [29] and Inception-v3 in Tables 18to to 25 with their standard deviations, computed as np.std(xx), where x are the individual results across the different seeds. The CP efficiency across architectures with standard deviations is shown Tables tooless5too These tables demonstrate how our state-of-the-ee performance architecture independent. Note that recreated the Q-SENN results on CUB-2011 and Stanford Cars for these tables in order to get standard deviations and results on more architectures but included the reported results in Norrenbrock al. [41] in the main paper. Notably SID @ has slightly decreased due to the introduction
Lieat. Usually, SID@ is high due to the Ldiv. With the addition of Lreat. especially on ImageNet-1K, its relative weighting is slightly lower. Therefore, increase the weighting for it for ImageNet- IK. For easier comparison and compute limitations, we did not optimize the weighting of Ldiv at all and left it at the weight used by SLDD-Mode [39], Q-SENN 40) and QPM 41). However, initial experiments show that one can increase without significant impact accuracy, shown in Table TO, The strong performance of CHiQPM across all the criteria is also summarized the radar plot in Figure [TO which follows QPM and thus only includes its metrics. Even ignoring the excellent Feature Alignment, Sparsity and novel hierarchical explanations with calibrated coherent set predictions, CHiQPM clearly sets the state of the art.
might want to
ProtoPool 0.93
SID@5
Baseline Resnet50 - 1.08
PIPNet 1.26
Accuracy
glm; - 1.47
SLDD-Model 2.21
1
Q-SENN 2..7
n
QPM 2.34
f
CHiQPM((urs) 2.61
Class-Independence
0.2
0.4 0.6
1
Nwc
Structural Grounding
Contrastiveness
Figure 10: Radar plot across the QPM metrics for CUB-2011 on Resnet50. Every value transformed to fraction of the maximum, following QPM [1]
Table 5: Average Set Size Y of CHiQPM with Resnet50 calibrated to reach various coverages 1 comparing different conformal prediction methods. All methods are very close or reach the desired coverage.
CUB
CARS
Method
0=2.11
a=0.1
a=0.075
a=00.55
a=00.755
a=0.05
a=0.0025
a=0.001
Ours
1.22 = 0.04
1.73 + 0.08
2.94 +: 0.17
9.05 + 0.20
1.05 + 0.02
1.25 0.14
8.25 0.38
74.4 1.83
THR
1.16 +: 0.04
1.32 + 0.05
1.67 4: 0.08
2.41 0.122
1.02 4: 0.01
1.15 0.02
2.09 = 0.16
6.93 1.49
APS
6.30: 0.41
7.20
0.65
8.54 +: 0.50
11.3 4 0.96
5.64 + 0.30
6.83 0.55
9.61
0.55
18.6 1.94
27

---

94
Accuracy
Fixed Level
86-
Built-in
THR
86
APS
6
10
15
20
Set Size
Results for QPM CUB-2011 for different CP methods applied to it, comparable to Figure CHiQPM significantly boots the efficiency of the built-in CP.
Figure 11:
Table 6: Average Set Size Y of CHiQPM with Resnet50 on ImageNet-I calibrated reach various coverages 1 - comparing different conformal prediction methods. All methods are very close or reach the desired coverage.
Y
IMG
Method
a=0.22
a=0.2
a=0.175
a=0.15
Ours
1.10 + 0.01
1.42 + 0.02
3.25 + 0.07
4.58 + 0.11
THR
1.05 + 0.00
1.16 + 0.01
1.40 + 0.01
1.87 + 0.03
APS
16.7 0.33
18.9 + 0.28
22.1 + 0.49
26.8 0.504
F Fine-tuning
Fine-tuning After solving the quadratic problem, the feature selection and binary assignment are set as fixed parameters of our model. Following Q-SENN, SLDD-Model and QPM, we continue training model, so that the features adapt to the concept(s) that or are shared by assigned classes and also follow their normalization. Specifically, the mean μ² E Ra and standard deviation " E Ra7 are computed on the training set:
fj wMrn
mdd.kk
(26)
1=
E/= Ji
(27)
(frd
n7y pa
)2
σ "
(28)
1r
where m²,.& refers to the value of the feature maps at the spatial position 1, k: for feature and sample j. This normalization for every feature d is maintained for the fine-tuning mne-tuning:
hu k=
F 27
fa WMhM
MaLkk
(29)
T
While this was initially done due to the reliance of Q-SENN and SLDD-Modis on glm-saga for sparsification, has also proven effective for QPM [T). While it is beneficial for the accuracy, as we show in Table this reduces the global interpretability, as their absence, sum of 0 across the entire feature map M, has an effect due to varying means 4 Hence, additionally apply ReLU
28

---

Table 7: Average Set Size Y of CHiQPM with Resnet34 calibrated to reach various coverages 1 comparing different conformal prediction methods. All methods are very close or reach the desired coverage.
1214
CARS
Method
6=0.12
a=0.1
a=0.075
a=0.05
a=0.075
a=0.05
e=0.0025
a=0.001
Ours
1.45 0.19
2.22 0.37
6.65 + 1.96
26.8 2:
1,05 + 0.01
1.47 0.18
12.6 1.19
96.0 4.36
THR APS
1.25 = 0.04
1.44
0.02
1.88 + 0.05
2.9 0.22
1.03 + 0.02
1.22 0.06
2.57
0.09
9.50 2.15
6.97 0.45
7.88 + 0.39
9.80 +: 0.62
12.9
6.13 + 0.69
8.20 = 1.00
12.0 = 0.86
21.5 3.18
Table 8: Average Set Size | of CHiQPM with Inception-v3 calibrated to reach various coverages 1 - & comparing different conformal prediction methods. All methods are very close or reach the
desired coverage.
1
CUB
CARS
Mothod
6=0.12
a=0.1
a=0.075
a=0.05
a=0.075
a=0.05
a=0.0025
a=0.001
Ours
1.52 0.05
2.34
7.63 + 0.15
61.5 4.25
1,02 : 0.01
1.47 = 0.22
47.0 22.7
81.8 3.04
THR
1.24 + 0.04
1.47 + 0.07
2,09 + 0.10
4.07 0.38
1,00 + 0.01
1,17 0.04
3.17 = 0.39
13.5 2.68
APS
7.08 = 0.44
7.94 + 0.65
10.5 + 0.96
13.8 + 1.35
5.14 + 0.30
6.77 = 0.76
10.8 = 0.77
21.1 # 1.82
the features after scaling, so that CHiQPM only reasons positively, and negligible activations are suppressed:
= ReLU(f)
(30)
Note that this already makes CHiQPM a set-valued predictor, as e.g. when only the fww features shared by 2 classes in I' activate, both would be predicted. However, on the test datasets used this happens very seldom and, as explained in Appendix A we use argmax to break ties for evaluating point-predictive performance.
Built-in CP THR
90-
85
AP5
Cohtrence
75
70
65
10
Set Size
Average Similarity compared to other conformal prediction methods for the same CHiQPM. The stars denote different a E (0.12, 0.1, 0.075, 0.05) used for calibration.
Figure 12:
F.1 Change during Fine-tuning
This section presents results on how much features change during the fine-tuning of CHiQPM. Table IT quantifies the Pearson Correlation between the same features before and after the fine-tuning in relation to p. Evidently, higher p causes higher change in the features. As consequence CHiQPM is less reliant on the perfect choice of initial features, as they undergo more changes regardless. For reference, QPM, has average correlation of 0.855.

---

Table 9: Average Set Size | of CHiQPM with Swin-Traasformer-SSee calibrated to reach various coverages a comparing different conformal prediction methods. All methods are very close or reach the desired coverage.
1244
CARS
Mothod
6=0.12
a=0.1
a=0.075
a=0.05
a=0.075
a=0.05
e=0.0025
a=0.001
Ours
1.14 0.05
1.44 0.21
2.44 0.32
8.09 0.77
1.28 + 0.24
3.3
20.2 = 25.6
75.0 3.14
THR
1.09 0.01
1.22
0.03
1.49 +: 0.04
0.14
1.15 0.13
1.42 0.21
2,44
0.62
7.62 2.36
APS
1.73 = 0.05
1.95 + 0.08
2.49 + 0.16
2.02 + 0.32
2.52 0.43
4.64 = 0.94
9.38 2.05
Table 10: Accuracy and SID@5 CHiQPM when increasing the weighting the Feature Diversity Loss Laiv | beyond the default value D. used for Q-SENN, the SLDD-Model and QPM. Without reducing accuracy, SID@5 can further increased for CHiQPM.
CUB
IMG
Metric
B = D
B = V2D
B 2D
== D
B
4D
8 = 12D
Accuracy SID@5
85.3
85.3
85.1
75.3
75.3
75.1
88.1
92.7
96.5
42.9
70.3
77.0
G
Relaxation of Hierarchical Constraint
This section presents the iterative relaxation of the hierarchical constraint introduced in Section 3.1 We relax the constraint of Equation (6) after finding the initial global solution for W and s, W init and
8mit Specifically, we set 8iuit to simplify the problem and allow the addition of m {0, 1) additional variable to optimize to the binary problem without introducing cubic constraints. directly optimizes which pair classes shall share wwe - features with the lowest reduction on the objective for the desired density p. Here T describes the set of pairs of classes that ever were represented sharing we 1 features during the iterative optimization process. Because the shared concepts be represented by the shared features exclusively, other pairs in IP not included in K emerge as other classes share this concept too. Extending how the QPM (Section (..2p ensures no duplicates, T is continuously updated, and the optimum found when all variables, including T, have not changed for an iteration:
hierarchical constraint
(we, we Sinit
(Nww 1) mm V(c,c)) ET
(31)
m 2
K
(32)
H Impact of Atent
This section is concerned with the weighting factor Aicat of the Feature Grounding Loss Lisat As described, it is set to Afest = 3, as the ablation in Figure [3 demonstrates that a further increase harms accuracy, without significantly further improving sparsity and feature alignment. This ablation also highlights that features become more aligned with the attributes in CUB-2011, as the sparsity increases. This is likely due to the shared concept between the assigned classes being less abstract. For reference, QPM reaches a Feature Aligmment of 1.9, half of CHiQPM, and thus learns significantly less grounded features than CHiQPM.
I Impact of ReLU
The necessity of the normalization and the improvements through our method are summarized in Table 4 First, is clear that normalization required to achieve state-of-the-ar performance and ReLU further improves accuracy. However, having features with a shared fixed minimum, either without normalization or due ReLU evidently increases Feature Alignment. Additionally, the ReLU significantly boosts Contrastiveness, all the non-activating features are exactly 0, leading to
30

---

Table 11: Impact of p on Average Pearson Correlation between features before and after fine-tume fine-tuning.
p
Average Pearson Correlation
0.0
0.825
0.1
0,823
0.3
0.805
0.5
0.776
0.7
0.743
lower distribution N (Equation (18) with very low standard deviation and therefore an almost perfect distinction between active and inactive features, However, the Class-Independee reduced with a shared minimum, as there are no minimal inactive activations that otherwise count towards Plass-Independenee
is slightly
Regardless, Class-Inaeppa bass-Innependence of around 95% is indistinguishable from a perfect score, as a perfect feature, that equally activates on all images its assigned classes and never on other images would have Class- baaas--ndeeendeece
1 20 90%. This calculation is based on the default configuration (Nowe = 5 and n = 50) for CUB-201 1, where every feature is on average assigned to 20 out of the 200 classes. Hence, the feature activation would be focussed on these 10% of the class samples. A further increased value can be attributed to robust features that are only sometimes part of a classes appearance. One example are birds that change their color during their lifetime, causing these color features to activate even without being assigned to these classes.
200
J Impact of Dynamic Hierarchy
Table 12: Comparison of dynamic and static feature ordering across different levels in the hierarchy for CHiQPM with Resnet50 on CUB-2011.
Method
Metric
Level 1
Level 2
Level
Level 4
Level 5
Dynamic Hierarchy (CHiQPM)
Set Accuracy
96.5
93.8
91.6
89.5
85.3
Set Size
20.17
4.43
2.12
1.45
1.00
Static Hierarchy (Avg. Order)
Set Accuracy
89.7
87.4
86.2
85.6
85.2
Set Size
18.52
2.85
1.30
1.09
1.00
This section evaluates the importance of using our sample specific hierarchy instead a static one. Our used hierarchy based order of the features of the predicted class has significant advantages compared to other choices. Its sample specific nature lends itself to a local explanation, as one can consider the order of the features as how dominant they appear in the image. Additionally, that enables traversing up the hierarchy in a meaningful way, since the least certain features get omitted first, causing accurate and efficient sets each step. We believe that the main alternative is class specific fixed order of features, as that would enable global hierarchical explanation for each class. The comparison between a fixed order of features for each class, computed based on the average order on training data and our prediction up the hierarchy is shown in Table 12 Evidently, predicting with such a fixed order causes less accurate inefficient sets compared to our sample specific dynamic hierarchy. For example, our dynamic hierarchy reaches 91.6% accuracy with an average set size of 2.1, while the fixed hierarchy never reaches that accuracy. We believe that this is likely due the unique feature, e.g. the red eye of the Bronzed Cowbird, being average quite important to that class and thus causing prediction sets that are too specific the top-1 prediction.
K Localization Quality of Features
This section evaluates the localization quality of the feature maps. We evaluate a form of the Pointing Game, similar the initial form in (66]. using segmentation masks provided for CUB-2011 [12]. Specifically, we calculate the fraction of the activation of the GradCAM [55] saliency map that is
focused on the segmented bird S {0, 1] mx
GradCAM
δ = Σ
Σ GradCAM OSS
(33)
31

---

85.3
85.2
Features Per Class Owe
6
855.2
85.2
85.1
85.2
85.0
85.1
85.1
85.0
85.3
85.1
85.2
85.2
85.3
85.2
84.5
84.9
85.2
85.1
85.1
84.9
85.1
85.1
84.8
85.2
84.7
0.031
0.1
0.31
1.0
3.0
10.0
30.0
84.6
84.5
Accuracy
5,0
N Features Per Class nue
6
4.5
5
4.0
4.6
4.6
4.4
4.5
5.0
5.3
5.3
5:3
33.5
0.031
0.31
1.0
Ateat
(b) Feature Alignment
40
N Features Per Class Dwc
23.5
35
26.8
312 32.6
26.4
22.3
30
34.8 33.4
29.2 23.7
19.8
18.5
20.3
30.8
29.2
26.0 20.7
17.1 15.8
17.3
-20
Asat
(c) Sparsity
Figure 13: Accuracy Feature Alignment and Sparsity relation to Ageat and the number of features per class Tw CUB-2011
As the bird can be considered the region of the image responsible for the classification, higher overlap with the segmentation indicates that the saliency maps localize more faithfully. The results are shown in Table Importantly, one would not necessarily expect an overlap of 100%, as the edge region relevant to describing shape, causing activations both on and off the segmentation. Across the entire test dataset, CHiQPM's 's GradCAM focusses to 83.4% on the bird, whereas the Dense baseline only does so with 75.8%. Hence, CHiQPM has activation maps with improved faithfulness on CUB-201. validating their use for saliency maps of individual interpretable features.
32

---

Table 13: Accuracy and faithfulness comparison on CUB-2011 with Resnet50. The final column measures the average GradCAM overlap with ground truth segmentations.
Model
Accuracy
Avg. GradCAM Overlap
Dense
86.6%
75.8%
CHiQPM
85.3%
83.4%
L Limitations
This section discusses the limitations of the proposed CHiQPM. CHiQPM is model that learns few general features which are used in a very interpretable way with binary sparse assignment. The learnt features already enable improved interpretability, as they tend to localize consistently on the same concepts enabling an intuitive understanding of the features and predictable local behvaiour based on global explanations. For example, given the global explanation in Figure [ a user would be able to predict that the image in Figure 2 would show little activation the eye is different. Notably, this is different to most prototypical models, which learn class-specific features [41 that detect red
eye of Bronzed Cowbird instead of red eye and thus lose human interpretability (20 24. However, for perfect human- saaaannnndtttaaablee.. global interpretability all of the individual features have to be understandable for humans. Dependent the dataset, there are 2 to 3 remaining obstacles:
1. As touched upon in Section 1
features can learn to detect multiple concepts, a phenomenon knows as polysemanticity. While the visualizations of features in Figures 3T to indicate a consistent localization of the same feature across many classes the same concept. proper metrics are missing to even measure that. However, we believe that a model like CHiQPM is very well suited to investitage this phenomen further, superposition [[] can likely be ruled out in its final features. The issue of polysemanticity exacerbated on ImageNet-IK, where every feature is assigned to z 100 classes. This connects another limitation, where the number of selected features n can not be set arbitrarily high, as the time takes to solve the QP grows exponentially with it, as shown in
2. While the activation maps of CHiQPM on CUB-2011 and Stanford Cars seem to localize very accurately, e.g. highlighting the red eye and not activating if is not visible, the activation maps on ImageNet- K seem to not always faithfully highlight the image region they respond to, e.g. Feature 23 in Figures 19 29 and concsissenty distributes a large portion of its activation on the same image patch. More elaborate saliency map methods or built-in methods like B-cos Networks [5] might be required for such activation maps. Finally, with the seemingly lacking faithfulness of the activation maps for ImageNet- IK, the use of SID@5 5 for this dataset can also be questioned.
3. CHiQPM learns general features that are well suited to classify the dataset given the training data. However, we do not restrict the features to be based on concepts that humans have noticed named before. Therefore, there may exist conceptual gap between the concepts learnt by CHiQPM and the ones known to humans. This Bi-directional Communication Problem B can however be reduced by examining the learnt general features of CHiQPM with its faithful global interpretability, as one can e.g. say that there is a distinctive pattern the necks of the black birds, that can be used to differentiatee them. Teaching humans the general shared features as ncologisms [19] seems promising to bridge this gap. Again, the learnt features of CHiQPM are uniquely suited as they individual neurons that already detect concepts. Finally, further work should aim at distinguishing polysemantic features from features that capture one concept unknown to humans.
The proposed method is evaluated quite thoroughly multiple datasets with various architectures. Thus, CHiQPM can clearly be applied to other general vision datasets. However, these datasets need to have many classes that have shared concepts, as the method relies on learning shared concepts between the classes.
For assumptions, it is to note that the guaranteed coverage of Conformal Prediction only holds under the exchangeability assumption of test and calibration data. We designed our experiments to ensure this property. Specifically, splitting calibration off from the test data ensures ii.d., which is stronger
33

---

property than exchangeability. Additionally, we observe that all CP methods closely match the target coverage for which we calibrate on test data.
Finally, the built-in interpretable Conformal Prediction method of this work guarantees unconditional coverage under the given assumptions, i.e. achieving the desired accuracy across the entire test dataset, instead of for all test samples. Achieving unconditional instead of conditional coverage can relate to fairness, as some groups might be undercovered. Limiting the number of levels ascend during the conformal prediction can theoretically further negatively affect the conditional coverage, as some classes are further away from other classes. However, conditional coverage challenging to even measure, as one needs fair assessment of difficulty. For the metrics we checked, using n limit. did not negatively affect conditional coverage. However, as the set construction interpretable for CHiQPM, the user can understand high uncertainty when all classes below the maximum number limit are predicted, which may reduce the need for built-in conditional coverage.
M Future Work
This section discuses the future directions of this work. One direction is a formal analysis of the used QP. Understanding the properties more detail might enable faster solving or scaling the model to even larger datasets that might require even more features.
Another very valuable direction validating the effectiveness of the introduced explanations via human studies, which have been out of scope for this work.
Finally, we hypothesize that CHiQPM is well suited to handle class-imbalanced data, as the constraints on the QP ensure that every class can be predicted if their general features are present.
N Broader Impact
This paper presents work whose goal is to advance the field of Machine Learning towards more interpretability. Globally interpretable models enable deeper understanding of the decision-making before and during deployment, and can therefore lead to increased safety, robustness, trustworthiness and potentially even scientific discovery. However, there also potential negative consequences. Most notable, a bad actor can use a model with faithful global interpretability to deliberately and systematically ensure decision-makin based spurious discriminating factors. Nevertheless, only such globally interpretable models offer the transparency to robustly ensure decision-making based the correct concepts.
Compute Ressources and Runtime
This section discusses the compute ressources used for the experiments in this paper. As GPU ressource, this work made of internal cluster composed of several NVIDIA RTX 2080 Ti. Every experiment fit on one GPU. As CPU ressource for solving the QP, this paper used an internal CPU cluster composed of primarily AMD EPYC 72F3 and up 250GB of ram. This reference hardware with sufficient memory is assumed for all estimates.
Table 14: Rough time in minutes needed to obtain a CHiQPM, nww = and nj = 50, with Resnet50 on the three datasets used in this work.
Dataset
Dense Training (GPU)
QP (CPU)
Fine-Tuning (GPU)
Total Time (Hours)
CUB-2011
176
200
82
458 (7.6)
Stanford Cars
234
200
110
544 (9.1)
ImageNet-1K
660
6300
6960 (116)
All main experiments were done on four architectures. The total time it takes to obtain CHiQPM dependes on the time spent training on the GPU, scaling with model and dataset size, and time spent solving the QP on the CPU, which scales with the dimensions of the variables and constraints. With Twe = and n = 50, the CPU time depends primarily on the number of classes in the dataset n. and the number of features of the block-box dense model ny- As Stanford Cars has a very similar number of classes to CUB-2011 (196 to 200), we caleulate with the same number of classes and
34

---

Table 15: Rough time for the seeds in minutes (hours) spent on training each of the architectures for the main experiments on CUB-2011 I and Stanford Cars (taking the time of CUB-2011 1 on GPU).
Architecture
GPU
CPU
Resnet50
258 5 .
200 :5 -2
Resnet34
143 5
58 -5-2
Inception-v3
76 5
200 -5
Swin-Transformes Smalll
478 5.
200 :5 -2
Σ
11141 (185)
6580 (110)
Table 16: Rough time in hours spent training the additional models for the Ablations
Ablation
Number of Models
GPU
CPU
Table
15
21
0
Table
14
Figure
70
75
230
Table
12 (6 on IMG)
352
Figure
(QMM)
5
17
Figure
50
127
487
297
times the number of training samples (8144 to 5994). The estimated time spent per dataset is shown in Table How that varies between architectures is shown in Table[[5] While there is some variance in cpu time due to the ablations, assume even cpu runtime for our estimates in Table I6 which effectively results in an upper bound on the cpu time spent.
Generating the visualizations or explanations took negligible compute ressources as everything required computed in one forward pass. Similarly, evaluating the models also very fast in comparison.
Finally, most of the experiments with Resnet50 before starting training for the fixed seeds and across multiple architectures. Similarly, further experiments of roughly the same number were run ultimately converge to the presented method. Also, starting from 61GB for solving the QP, very few seeds needed more than that, which we started after the inital ones crashed. For the three solved QPs ImageNet-IK, we always allocated the available memory of 250G. Additionally, we had to retrain Q-SENN to include it in our tables. Each of these runs takes roughly half an hour more on the GPU than CHiQPM. Note that, Q-SENN did always converge with Swin- Tein---ranfoorr-Siiiiee needs to tweak hyperparameters to ensure convergence. Similarly, Figure TT required training QPM. We want to emphasize that we reused the same dense model or even QP solution where possible, e.g. for CHiQPM, QPM and Q-SENN or many of the ablations. This puts the total amount of GPU hours to roughly 1382 and CPU hours 1040, shown in Table 17
3 5 random seeds with CUB-2011
which is why no result is reported there as one likely
For storage, we temporarily save every model and their low-dimensional feature vectors to speed up metric calculations. While our internal clusters offers significantly more storage, the experiments of this paper needed less than 100 GB.
P Further Qualitative Examples
This section comments the extensive qualitative examples. For all examples, we always present explanations for the same model, i.e. all explanations for CHiQPM on CUB-2011 with fww and n 50 explain the same model. Therefore, this section discusses results of models, 3 per dataset with the default configuration and also visualizations of CHiQPM with Tw = and n = 30 trained on CUB-2011 and explained in Figures [add] Figure 14 first contains the global class explanation between the Red-winged blackbird and Bronzed Cowbird, which are jointly predicted in Figure 2 The CHiQPM faithfully communicates that differentiates them based on their unique attributes:
The red eye or wing. Additionally, Figure 21 contrasts the explanation of Figure 2 with one for an
35

---

Table 17: Rough total time in hours spent on training with reference hardware NVIDIA RTX 2080 Ti as GPU and AMD EPYC 72F3 as CPU
Experiment
GPU
CPU
Main Experiments (Table 15
185
110
Ablations (Table 16
487
297
ImageNet Main Experiments (Table
14
350
33
Preliminary / Failed experiments
300
600
Q-SENN 40)
training as competitor
60
0
Σ
1382
1040
Bronzed Cowbird Features
Train Images
Feature 7
Feature 8
Feature 17
Feature 4
Bronzed Cowbird
winged Blackbird
winged Blackbird
Figure 14:
Global explanation comparing Bronzed Cowbird and Red Winged Blackbird. The CHiQPM trained CUB-2011 with features per class and 30 in total, explained in Figures [ a determined the differentiating factor, their red eye or wing, without any annotations and can communicate its behavior faithfully.
easy example. Further contrastive global class explanations with human understandable differences on CHiQPM trained on CUB-2011, 1, Stanford Cars and ImageNet- IK are shown in Figures 15 20 Additionally, Figure[11 displays displlas the extensive local explanations and an example where CHiQPM dynamically predicts the appropriate set of classes based on visible evidence, as globally explained by 2 and 21 Figures 23 and 24 show the corresponding graphs without the limitation only including classes that share the top feature. Finally, Figures 25 to 30 show more exemplary local explanations of our CHiQPM, including the novel hierarchical explanations and saliency maps that also transport activation, described Appendix C.2) They demonstrate how CHiQPM can provide uniquely comprehensive local interpretability for single test image, faithfully following its global explanations.
Figure 17 Supporting Figures
P.1 Feature Generalization Visualizations
This section provides supplementary visualizations demonstrating that the features learned by our CHiQPM are not merely class-specifie detectors but function general concept detectors across diverse range of images. Figures
34 visualize how the features of the probed CHiQPMs to 16 and to 28 generalize across a huge range of images and classes that share all one feature with one class. We choose
explained in Figures I 2 classes. For that, we visualized Shiny Cowbird for Figures 31 and 32 as it has large neighborhood in the probed CHiQPM's class representations and is also used the main paper Figures T and Additionally, we show the features of all classes similarly represented to Arctic Tem in Figures
14
33 and 34 as it is another class
36

---

White necked Raven Festures
Train images
Feature 29
Peature
Feature 23
Featare
Feature 32
White nocko Favin
Common Roven Fealures
Figure 15: Global explanation comparing White necked and Common Raven. CHiQPM trained on CUB-2011 determined the differentiating factor, the white neck, without any annotations and can communicate its behavior faithfully. Local explanations of this model are shown in Figures 25 and 26
Clay colored Sparrow Features
Train images
Feature B
Feature
Feature 33
Feature 25
Feature 39
Peature 44
City colored Spartw
Spartow
Figure 16: Global explanation comparing Clay colored and Brewer Sparrow. CHiQPM trained on
CUB-2011 1 determined the differentiating factor, the distinct head patterns, e.g. the white crown stripe for Clay colored Sparrow, without any annotations and can communicate its behavior faithfully. Local explanations of this model are shown Figures 27 and 28
with large neighborhood in both models. We further supplemented Figure 32 with the Raven classes from Figures 15 25 26 showcase that the features localize consistently, not just directly around the chosen class. These visualizations demonstrate that similarly represented classes by CHiQPM indeed similar in reality, measured by Structural Grounding, and that the features are general concept detectors.
37

---

Table 18: Accuracy and Compactness using Resnet50; CHiQPM shows the highest accuracy among interpretable models. Figure shows the increasing gap when further raising compactness. Among more compact models, the best result is marked in bold, second best underlined.
Method
Accuracy CARS
Total Features)
Features/ / Class
CUB
IMGNET
CUB
CARS
IMGNET
CUB
CARS
IMGNET
Baseline Resnet50
86.610.2
92.110.11
76.1
2048
2048
2048
2048
2048
2048
glm-sagas
78.020.4
86.820.6
58.0+0.0
80918
8074100
1627
5
5
5
PIP-Net
82.0+0.3
86.5+0.3
731=19
669+13
12
11
ProtoPool
79.4+0.4
87.5+00.2
202
195
202
195
SIDD-Model
84.5+0.2
91.1+0.11
72.7++0.0
50
50
5
5
Q-SENN
84.6.10.3
91.8.10.3
74.3:10.0
50
50
50
5
5
QPM
85.1+0.33
91...0.3
74.2110.
50
50
50
5
5
CHiQPM (Ours)
85,3+-0.3
91.9#0.2
75.3++0.0
50
50
50
5
5
5
Table 19: Comparison on QPM 41 Interpretability metrics with Resnet50. Requiring annotations, Structural Grounding can only be computed for CUB-2011, Among more compact models, the best result is marked in bold, second best underlined.
Method
SIDE51 CARS
Independence CARS
Contrusivennes" CARS
Grounding?
IMGNET
IMGNET
IMGNET
CUB
Baseline Resnet50
57.7.004
54.4.0.3
37.1
96.0.10.0
97.8.10.0
99.4
74.410.1
75.1.10.1
71.6
34.010.3
glm-sogas
55,4+00.5
51.8-0.3
358+00.0
97.++0.3
97.6=00
99.4-0.0
74.0=0.1
74.5=00.
71.7=0.0
2,5+1.0
PeP-Net
99.210.1
99.0.0011
75.6.10.4
62.911011
9..6.0.00
99.7:10.0
6.7.10.9
ProsoPood
24.5
3077234
96.920.1
96.020.
76.7=:100
78.9220
13.91:09
SLDD Model
88.2110.2
88.6.0.6
64.710.7
96.210.11
95.5.0.11
9..6.0.0
87.3.0.2
89.710.
93.-0.11
29.214.0
O-SENN
03.2440.4
94.3=0.3
95.5+0.1
94.22011
930-00.3
03.9-00
20.8+-4.1
QPM
90.1410.3
6:.44
64.110.7
97.0:10.0
96.5:000
90.1:000
97.7:0.
89.3:0.1
47,912.7
CHIQUM (Oues)
38.1#00.5
88.==10
42.9:0.9
94.1#0.3
93.5=0.1
98.7=0.0
100.0:000
99.9=0.0
75.04222
Table 20: Accuracy and Compactness using Resnet34: CHiQPM shows the highest accuracy among interpretable models. Among more compact models, the best result is marked in bold, second best underlined.
Method
Accuracy 1
Total Features
Features Class
CUB
CARS
CUB
CARS
CUB
CARS
Bascline Resnet34
85.7+0..3
91.5=0.2
2048
2048
2048
2048
glm-sagas
1.0
82.0=0.6
442:
453=6
5
SLDD-Model
83.2+10.3
90.7
50
50
5
5
Q-SENN
83.7+0.2
91.3+1..
50
50
5
5
QPM
83.0+0.22
91.3=000
50
50
5
5
CHiQPM (Ours)
83.7+0.22
91.5+
50
50
5
5
Table 21: Comparison on QPM [1] Interpretability metrics with Resnet34. Requiring annotations, Structural Grounding can only be computed for CUB-2011. Among more compact models, the best result marked in bold, second best underlined.
Method
SID@5 f
Class-Indepnndence t
Contrastiveness*
Structural Grounding CUB
CUB
CARS
CUB
CARS
CUB
CARS
Baseline Resnet34
62.110.3
56.610.4
97,910.0
97.710.0
76.410.1
77.910.2
39.610.2
gim-saga,
59.9+0.4
55.3+0.3
97.9=0.0
97.7+0000
76.5+0.00
77.8+0.2
7.6+2.22
SLDD-Model
90.1++0.8
86.7+2.5
97.5+0.0
97.6+-0.2
86.0+110
83.3++4.6
24.5+2.7
Q-SENN
86.6+:1.4
80.1+228
96.6+0.1
96.0++0.2
9.0
94.3++1.4
25.7+2.0
QPM
90.5++0.5
89.1=1.1
97.5+0.0
96.9--0.1
95.5++0.2
94.7+1.11
39.0+2.99
CHiQPM (Ours)
87,9+0.5
88.4+1.0
94.1+0.11
82,9++4.5
99.9++0.0
100.0+0.0
68.5+4.1
38

---

Table 22: Accuracy and Compactness using Inception-v3: CHiQPM shows the highest accuracy among interpretable models. Among more compact models, the best result is marked in bold, second best underlined.
Method
Accuracy t
Total Features.
Features Class CUB
CUB
CARS
CUB
CARS
CARS
Bascline Inception-v3
86.1
92.6:1.22
2048
2048
2048
2048
glm-sagas
79.2::0.5
89.310.3
81419
795
SLDD-Model
83.1+0.4
91.1+0.2
50
50
Q-SENN
83.8
91.6:10.2
50
50
QPM
84.2+t0.4
91.7+0.11
50
50
CHiQPM (Ours)
8..3++0.
92.1+-0.2
50
50
5
Table 23: Comparison on QPM [41 Interpretability metrics with Inception-v3. Requiring annotations, Structural Grounding can only be computed for CUB-2011. Among more compact models, the best result marked in bold, second best underlined.
Method
SID@s f
Class- Indepeeneeee f
Contrastivenesst
Structural Grounding
CUB
CARS
CUB
CARS
CUB
CARS
CUB
Baseline Inception-v3
38.9=0.3
33.110.2
96.11:0.0
95.710.0
89.6100.2
91.7:10.2
7.199.6
saga,
39.3=0.2
34.0+0.44
95.4+000
95.0+10.0
91.34+0.3
93.4:+:.2
0.3++0.4
SLDD-Model
58.1=1.2
52.1
92.610011
92.110.1
93.0.10.3
94.4100.2
24.4422.3
Q-SENN
55.6=:0.9
48.2:10.
92.2:10.2
91.5:10.1
94.6:10.3
95.14:0.4
19.7112.6
QPM
48.6=0.9
42.8+088
95.1+0011
94.7+00.0
93,44+0.1
94.3++0.1
34.8+:3.4
CHiQPM (Ours)
45.4-1.33
37.812.2
93,440.1
92.740.2
100.0:0.0
100.0:1.00
52.8:1455
Table 24: Accuracy and Compactness using Swin Transformer small: CHiQPM shows the highest accuracy among interpretable models. Among more compact models, the best result is marked in bold, second best underlined.
Method
Accuracy +
Total Features.
Features / Class CUB
CUB
CARS
CUB
CARS
CARS
Baseline Swin Transformer small
87.0+0.1
90.6110.6
768
768
768
768
glm-sagas
0.4
1.2
572+
559+
5
5
SLDD-Model
85.3
89.110.77
50
5
QPM
85.0==0.4
88.7+0.5
50
50
5
5
CHiQPM (Ours)
85.9
89.0+1.6
50
50
5
Table 25; Comparison on QPM annotations, Structural Grounding can only computed for CUB-2011. Among more compact models, the best result is marked in bold, second best underlined.
41 Interpretability metrics with Swin Transformer small. Requiring
Method
SID@5 f
Class Indepeeddneee f CARS
Contrastivenesst CARS
Structural Grounding CUB
CUB
CARS
CUB
Baseline Swin Transformer small
26.4.10.1
26.0.10.1
96.8.10.0
96.61.0.0
98.3.10.1
98.8.10.1
24.5
glm sagas
26.4.10.2
26. 26.1.001
96.6.0.00
96.410.00
99.1
99.6.10.0
8.8.1..8
SLDD-Model
38.0=10.5
35.621.1
93.4=0.1
93.310.2
99.010.2
99.420.2
37.223.4
QPM
33.62:04
320--033
93.2=0.0
94,740.0
08 38.3
09.1202
45.1=3.2
CHiQPM (Ours)
46.1440.9
45.8=4.0
94.2-0.1
93.7440.1
99.9+-0.0
99.9-1.11
65.4=3.6
39

---

Asion Markin Vi Vertiage Corvertible 2012 Teatures
Train Images
Feature
Reature
Reature 16
Reature 21
Feature 45
Festure 9
Ascin Mortin VS Convertali 2012
Acton Mart va 2012
Aston Nartir Verkage Coupe 2012 Features
Figure 17: Contrastive global Explanation, comparing the class representations of two cars that only differ in Coupe or Convertible for CHiQPM trained on Stanford Cars that represents every class with 5 of 50 features. They are differentiated based on features activating where the windows would be.
Aston Martin Vantage Corvertible 2012 Features
Train inaiee
Feature 45
Feature
Peature
Beature 16
Pesture 21
Festure
Aston Kartin ve Convertzte 2012
Adtinn Nortin Converti'e 2012
Acton Martin Virage Convertile 2012 Features
Figure 18: Contrastive global Explanation, comparing the class representations of two Convertible Aston Martins for CHiQPM trained on Stanford Cars that represents every class with 5 of 50 features. They are differentiated based on human perceivable deviating features like the fender vent.
Barnare mountain
002
Greaker Swiss Nountain 200
Greater Swiss Hourtain dog Festures
Figure 19: Contrastive global Explanation, comparing the class representations of two mountain
dogs for CHiQPM trained on ImageNet-1K K that represents every class with of 50 features. They are differentiated based on human perceivable deviating features like the different ear fur. Local explanations for this model are shown in Figures 29 and 30
40

---

white stork Features
Imaga
Reature
White stork
Block stork
Figure 20: Contrastive global Explanation, comparing the class representations of white and black stork for CHiQPM trained on ImageNet-IK 1K that represents every class with 50 features. They are differentiated based on one broadly activating feature.
41

---

n=1
n=2
Horned Puffin, +-6
Common Raven,
winged Blackbird,
Yellow headed Blackbird
American Crow
Bronzed Cowbird Features
Bronzed Cowbird
Test Images
Feature
Feature
Feature 17
Feature
Uniknown
Unknown
n=1
Shiny Cowbird Features
n=2
Least Auklet, +7
Black throated Blue
Groove billed Ani
n=3
Red winged Blackbird, +3
Warbler
Black capped Vireo,
Shiny Cowbird Bronzed Cowbird
Figure 21: Exemplary local explanations provided by our CHiQPM, with the global explanation in Figure [ for two test images of Bronzed Cowbird. The first row an easy example where all features are found, including the red eye. Therefore, our calibrated model only predicts Bronzed Cowbird. The red eye is not visible the second image, leading the reasoning of our CHiQPM along its dynamic class hierarchy identifying it as one of the black bird species and predicting all of them, including the Bronzed Cowbird.
42

---

BMW M3 Coupe
2012, +10
Cadillac Escalade EXT Crew Cab 2007
2007
Jeep Cherokee SUV 2012
Aston Martin Virage Convertible 2012
Aston Martin va Vantage Coupe 2012
Aston Martin ve Vantage Convertible 2012
Teat Imnger
Feature
Festure
Fusture 15
Feuture
Naturi 43
Posture 3
Aston Vartin va Vantage Features
BMW 2012, +6
Hyundal Genesis Sedan
GMC Hybrid
Jeep Grand Cherokee SUV 2012
Volvo XC90 SUV 2007
Cadillac Escalade EXT Crew Cab 2007
Aston Martin Virage Convertible 2012
Aston Martin V8 Vantage Aston Martin V8 Vantage Coupe 2012 Convertible
Figure 22: Exemplary local explanations provided by our CHiQPM, with the global explanation in Figure [7 for two test images of the Convertible. The first row an easy example where all 5 features are found, because the top is down. Therefore, our calibrated model only predicts the Convertible. The top is up in the second image, leading to the reasoning of our CHiQPM along its dynamic class hierarchy identifying it as one of the cars, either Coupe Convertible because both window features are only barely activating. As the global explanation in Figure exxplanss the probed CHiQPM does not rely on the fabric top, and hence predicts the set of both cars interpretably.
43

---

Rock Wren. +15
Whip
Homed Puttin,
Least +10
Western Gull
Common Raven,
Red winged Blackbird, +3
Yellow headed Blackbird
Groove billed Ani
American Crow
Bronzed Cowbird
Figure 23: Full graph including all activations for top graph in Figure 21
44

---

Homed Putfin.
Blue Jay. Grebe,
Brown Pelican, 14
Chuck will Widow, +8
Least Auidet, +7
Blackbird
Groove
Red winged Blackbird
Brandt Cormorent
Winter Wren, Black throated Blue Warbler Fox Sparrow
Capa Glassy Starling,
Black capped Vireo, +1
Blackbird
Rusty Blackbiird
Whip poor Will
Bronzed Cowbird Shiny Cowbird
Figure 24: Full graph including all activations for graph in Figure
45

---

White necked Raven Features
Test Image
Feature 29
Feature 16
Feature 23
Feature
Feature 32
Unknoww
Least Auklet, +2
Black capped Vireo
Eastern Towhee 1
Orchard Oriole
Rose breasted
Grosheak throated Spartow
Homed Puffin
Red winged Blackbird, +3 Scott Oriole, +1 American Three toed Woodpecker
Pigeon Guillemot
Black throated Blue Warbler Common Raven, +1
Shiny Cowbird, +1
Groove billed Ani
White necked
Bobalink
Figure 25: Exemplary local explanation for White necked Raven labeled test sample of CHiQPM with global explanation in Figure [5] All features recognized, hence only the true label is predicted and the features of the predicted class visualized. The tree further visualizes the learned class similarities, with violet and red leading to black throated birds, whereas violet and yellow seem to indicate Oriole.
46

---

White necked Raven Features
Test Image
Feature 29
Feature
Feature 23
Feature 30
Feature 32
Unknowwn
Gray Catbird, +3
Homed Puffin, +1
Frigatebird
Yellow headed Blackbird
Pileated Red winged Blackbird, +3 Woodpecker
Shiny Cowbird, +1 Bobolink
Fish Crow Common Raven
White necked Raven
American Crow
Figure 26: Exemplary local explanation for White necked Raven labeled, presumably mislabelled, test sample of CHiQPM with global explanation in Figure L5 Due to missing evidence, especially the white neck, our calibrated CHiQPM predicts a set black birds. We show saliency maps for features of White necked Raven the reader can have learned about its class representation already. Note that the gray feature for Common Raven is shown in Figure too. When CHiQPM determines that it needs to predict set, one might want to visualize all the features of all classes that are predicted in the coherent set. Alternatively, all active features could be visualized. This is ultimately up to the level of detail desired.
47

---

Clay
Image
Feature
Unbmown
1
Wren, +1
Horned Lark, +
essper Spamew
Yark eyed Jundo
slsld Sparrow
Nelson Sharp tailed parrow
Conte Sparrow
Marsh Wren
Bewick Wren
White crowned Sparrow
Savennah Sparrow
Swainson Warbler
Green tailed Towhee
Brewer Sparro
Song Sparrow
Hous Sparrow, +1
Clay colored Sparrow
Chipping Sparrow Field Sparrow
Tree Sparrow
Figure 27: Exemplary local explanation for Clay colored Sparrow labeled test sample of CHiQPM with global explanation in Figure [6] All features recognized, hence only the true label is predicted and the features of the predicted class visualized. Interestingly, the clearly visible white crown stripe, detected by the red feature, is sufficient for CHiQPM distinguish it from most other sparrows early on in the hierarchy, as opposed to Figure 28 where blue and green determine a set of sparrows first.
48

---

Clay colored
Image
Feature
Unkrown
Green tailed Towhee Dark eyed Junco, +1
Palm Warblen
Rock We
Horned La
Field Sparrow +1
Northem Waterthrush
Henslow Sparrow, Louisiapa Waterthrush
Nelsion Sharp tailed Sparrow
Bewick Wren
House Sparrow, +1
Song Sparrow
Seaside Sparrow
White crowned Sparrow White throated Sparrow Savannah Sparrow
Marsh Wren Le Conte Sparrow
Baird Sparrow Vesper Sparrow
Brewer Sparrow
Clay colored Sparrow
Figure 28: Exemplary local explanation for Clay colored Sparrow labeled test sample of CHiQPM with global explanation in Figure 16 Both head features activate evenly, thus both sparrows are predicted jointly.
49

---

dog
Image
Unkoown
sea lion, +52
wine,
Tibetan mastiff, 43
black swan, +1
drake, +3
hot pot, +14
basset
Appenzeller, +1
wok, +2
ladle, +3
Rottweiler
oystercatcher
Greater Swiss
Mountain dog
Bernese mountain dog
Figure 29: Exemplary local explanation for Bernese mountain dog labeled test sample of CHiQPM with global explanation in Figure 19 All 5 features recognized, hence only the true label is predicted, The tree further gives a glimpse into how the 1000 classes of ImageNet-I1e IK are organized.
50

---

Bernese mountain dog Reatures
Test Image
Feature 2
Feature
Feature
Feature 44
Feature 46
lion,
Tibetan mastiff
grey whale, German shepherd
8
buckle,
Loafer, red wine, +4
hot pot, 16
bla ck-aan--tan coonhound otterhound
wok, +2
kelple
black swan
teapot,
Wa ker hound, +1
basset
drake, +1
bluetick
Rottweiler
beagle. +1
Appenzeller
Bernese EntleBucher mountain dog Greater Swiss Mountain dog
Figure 30: Exemplary local explanation for Bernese mountain dog labeled test sample of CHiQPM with global explanation in Figure [9 All features are recognized, hence only the true label is predicted. However, due to the sideways pose, feature 46, focusing on the frontal face, is less activated. The tree further gives glimpse into how the 1000 classes of ImageNet- K are organized.
51

---

Lesve Chactors
CM SC
Figure 31: Visualizations for classes similar to Shiny Cowbird using the model with 3 features per class, e.g. explained Figures] and
Figure 32: Visualizations for classes similar Shiny Cowbird using the model with nww 5 features
per class, e.g. explained in Figures 15 T6 and 25 to 28 We included the Raven classes from
Figures 15 25 and 26 for reference.
52

---

Figure 33: Visualizations for classes similar to Arctic Tern using the model with fwwe features
per class, e.g. explained in Figures T
2 14 and
Talk Tegis
Festre 33
Texture 23
Festre 24
Texture 26
Festar 35
Ferur 42
Miic hen
Doper 9m
Figure 34: Visualizations for classes similar to Arctic Tern using the model with Tlwe features
per class, egg. explained in Figures [5
16 and 25
28
53


Reviews:
[{"review_id": "TaKgoJQxPH", "reviewer": "Reviewer_kBM9", "summary": "The paper introduces Calibrated Hierarchical QPM (CHiQPM), an interpretable image classification framework that merges the compact global explanations of Quadratic Prototype Models (QPM) with *hierarchical* local explanations and built-in *conformal prediction* (CP).\nAt training time, a dense backbone network is first diversified, after which a discrete quadratic program with new hierarchy constraints selects a small, fixed set of concept-like features per class. Those features are then fine-tuned with a Feature Grounding Loss and an additional ReLU so that each activates only on a single, human-aligned visual concept. Finally, the model is *calibrated* by CP, enabling it to traverse its sample-specific hierarchy and output coherent class sets.", "questions": "## Questions\n- Can you provide formal statements for the proposed method as raised in the above?\n- Justification of hyper-parameter selection is needed. Can you give more discussion?\n- Can you state whether the reported experimental settings satisfy the assumption? and can you provide several counterexamples?\n\n## Additional suggestions\n- Matrix \\textit{M} denotes a feature map, but the letter $M$ is reused for class-class similarity, risking confusion.\n- L655: “calibrated cohrerent set predictions” → coherent\n- L839: “These visuzalizations demonstrate…” → visualizations\n- “we temporarily save every model … to speed up metric **calculation**.” The preceding sentence says *metrics* (plural). Singular/plural mismatch.\n- “auxiliary Feature Diversity Loss $\\mathcal{L}_{\\text{div}}$ [33]” – no definition or formula is given in the current manuscript. Readers must look up ref. [33].", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- The proposed pipeline is easy to follow.\n- Non-trivial empirical gains while keeping backbone accuracy.\n- Interpretable hierarchy visualizations and qualitative examples. Figures illustrate that the learned tree edges correspond to semantic super-/sub-classes and that prediction sets indeed trace these edges upward when uncertainty is high.\n\n## Weaknesses\n- Essential steps lack any formal analysis. There is no approximation bound, convergence argument, or sample-complexity discussion. Theoretical novelty is therefore thin.\n- Since the proposed method appears to be somewhat sensitive to a hyper-parameter $\\rho$, theoretical discussions on the optimality of hyper-parameter selection strengthen this study. Moreover, the authors set $\\lambda_{\\text{feat}} = 3$ in their numerical experiments, but there are no discussion on how to choose this value, and no justification.\n- Exchangeability is assumed in the manuscript, but there is no discussion as to when this assumption is satisfied and when it is violated (how realistic an assumption it is). At least there should be a discussion of the assumptions made with respect to the experimental setup reported by the authors.", "was_revised": true, "final_justification": "Although theoretical concerns remain, other points were addressed by the rebuttal.", "rebuttal": "Thank you for your review, appreciating several strengths and raising several questions that allow us to clarify our contributions. We also thank you for the detailed suggestions on the text and will of course correct the typos, rename the class-class similarity to an unused symbol, and include the formula for the Feature Diversity Loss in the revised manuscript. \n\n> Essential steps lack any formal analysis. There is no approximation bound, convergence argument, or sample-complexity discussion. Theoretical novelty is therefore thin. \n> \n> Can you provide formal statements for the proposed method as raised in the above? \n\nOur work follows other work in interpretable machine learning, such as PIP-Net, ProtoPool, Q-SENN or QPM, where the primary contribution is a model that offers novel forms of interpretability that are thoroughly empirically validated. As such, the paper’s main contribution is not theoretical. Instead, our main novelty is the introduction of hierarchical explanations and the adaptation of the theoretically well-grounded theory of conformal prediction to predict along them. This enables the first form of built-in interpretable conformal prediction and local explanations that answer an unprecedented range of questions simultaneously while having non-trivial empirical gains (as rightly pointed out):\n\n1.  What meaningful features of which classes are found in an image? (Quantified by multiple metrics including Structural Grounding and Feature Alignment in Tables 1, 3, & 16.) \n1. How does each feature narrow down the set of potential predictions into increasingly similar classes? (Visualized in our hierarchical explanation and measured by Set Coherence in Figure 6.) \n1. Which set shall be predicted to guarantee a configurable average accuracy? (Guaranteed by our conformal prediction framework, with results in Table 2.) \n1. Which features would have needed to activate stronger in order to predict a smaller set with sufficient certainty? (Demonstrated by the paths in our hierarchical explanations.) This notion of counterfactual interpretability has recently gotten more attention [1] and is included by-design. \n\nNote that questions 2.-4. are only enabled through our novel contributions, while the answer to 1. is greatly improved compared to what QPM offers. \nAlongside the improved global interpretability, we consider answering these questions in an explanation a significant step to advance the field of interpretable machine learning. We will include this phrasing and the recent counterfactual paper as we believe that it clarifies our contribution and why it is significant without novel formal statements. \n\n> Exchangeability is assumed in the manuscript, but there is no discussion as to when this assumption is satisfied and when it is violated (how realistic an assumption it is). At least there should be a discussion of the assumptions made with respect to the experimental setup reported by the authors. \n> \n> Can you state whether the reported experimental settings satisfy the assumption? and can you provide several counterexamples? \n\nYou correctly pointed out that we assume exchangeability between calibration data and test data for our conformal prediction experiments, which is required for the guarantees of Split Conformal Prediction to hold. We argue that exchangeability is certainly holding for our evaluation, as we \n\n1. Calibrate on the first 10 samples per class of the test data (lines 325-326). This ensures the calibration and test samples are i.i.d., a stronger condition than exchangeability. \n2. Very closely match the coverage on the test data for which we calibrate, indicating that the test was exchangeable with the calibration data. \n\nWe will explicitly state that we assume exchangeability in the main paper and reference an expanded section on exchangeability in Section J, which contains the surrounding clarification. As our experiments are designed to satisfy exchangeability, there is no counterexample in our experiments. However, there exists work [2, 3, 4] on applying split conformal prediction under conditions where exchangeability does not hold. Investigated scenarios without exchangeability are time series datasets [2, 3] or calibrating on ImageNet and then testing on datasets with distribution shift [4], such as ImageNetR or ImageNetA.\n\n> Since the proposed method appears to be somewhat sensitive to a hyper-parameter $\\rho$, theoretical discussions on the optimality of hyper-parameter selection strengthen this study. Moreover, the authors set $\\lambda_{feat}=3$ in their numerical experiments, but there are no discussion on how to choose this value, and no justification. \n> \n> Justification of hyper-parameter selection is needed. Can you give more discussion? \n\nWe thank the reviewer for the question about selecting $\\rho$ and are happy to clarify our reasoning. As described in Sections 3.1, 4.3 and D, $\\rho$ should be set using the calibration data and depends on several factors, most notably how many classes are similar in the dataset and if a specific coverage is targeted. Varying coverage goals relate to real-world scenarios, since a bird classifying app can be calibrated for a lower coverage, while a medical application should certainly guarantee smaller error rates. \nWe selected $\\rho=0.5$. As shown in our ablation studies (Figures 7 and 8), this value represents a 'sweet spot' for the CUB-2011 dataset. It is high enough to induce a rich semantic hierarchy that improves set prediction performance, yet low enough that it does so without sacrificing point-prediction accuracy. However, we will extend Section D to emphasize that there is no single optimal $\\rho$ for all CHiQPMs. \n\nYou are right to point out that an ablation on $\\lambda_{feat}$ is missing. It was chosen based on the initial experiment on CUB-2011 shown below showcasing Accuracy, Sparsity and Feature Alignment with respect to the weighting. Evidently, alignment and sparsity improve with increasing $\\lambda_{feat}$. The hyperparameter was set to the highest value for which the accuracy does not decrease, hence resulted in $\\lambda_{feat}=3$. We will include this ablation in the appendix. \n\n| Metric \\ $\\lambda_{feat}$ | 0.031 | 0.1  | 0.31 | 1.0  | 3.0  | 10.0 | 30.0 |\n| :-------------------- | :---: | :--: | :--: | :--: | :--: | :--: | :--: |\n| Accuracy              | 85.3  | 85.1 | 85.2 | 85.2 | 85.3 | 85.2 | 84.5 |\n| Feature Sparsity      | 38.8  | 37.2 | 32.6 | 26.4 | 22.3 | 21.2 | 22.9 |\n| Feature Alignment     | 3.1   | 3.2  | 3.4  | 3.7  | 3.8  | 3.9  | 4.0  |\n\n\nWe thank you again for pointing out valid points of criticism and believe that the resulting changes will improve the manuscript. \n\n[1] Dominici, Gabriele, et al. \"Counterfactual Concept Bottleneck Models.\" ICLR. 2025. \n\n[2] Barber, Rina Foygel, et al. \"Conformal prediction beyond exchangeability.\" The Annals of Statistics 51.2 (2023): 816-845. \n\n[3] Oliveira, Roberto I., et al. \"Split conformal prediction and non-exchangeable data.\" Journal of Machine Learning Research 25.225 (2024): 1-38. \n\n[4] Alijani, Shadi, and Homayoun Najjaran. \"WQLCP: Weighted Adaptive Conformal Prediction for Robust Uncertainty Quantification Under Distribution Shifts.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "MCVmQEpGN5", "reviewer": "Reviewer_7M7N", "summary": "This work introduces the Calibrated Hierarchical QPM (CHiQPM). The strength of CHiQPM lies in its ability to provide both global and local explanations. The model employs contrastive explanations for majority classes and offers hierarchical explanations that align with human reasoning; a unique feature that also enables interpretable conformal prediction (CP) through traversable decision paths. Empirical results demonstrate that CHiQPM achieves near-state-of-the-art accuracy (retaining 99% of non-interpretable models' performance) while maintaining interpretability.", "questions": "1. Given the proposed method can do both global and local explanations; if the author consider to compare with some common XAI benchmarks, e.g., OpenXAI.\n\n2. Please provide some discussions around scalability of the proposed approach.\n\n3. In practice, can the authors give some examples of how a conformal set of features is better than, say, the use of top-1 or top-k features for decision-making.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n1. the idea of integrating global, local XAI and CP apears to be novel.\n2. the paper is technically sound.\n\nCons:\n1. while the overall idea is novel, the work is incremental to the original QPM work.\n2. the writing is a bit hard to follow if without prior knowldege of QPM (i.e., not self-contained), I have to quick read the QPM paper as well.\n3. the evaluation miss some common metrics for local XAI, e.g., delete/insert scores, Pointing Game;", "was_revised": true, "final_justification": "The authors addressed my concerns in their response and thus I increase my score.", "rebuttal": "Thank you for your review and appreciating the novelty and soundness of our paper. We also thank you for the valuable feedback regarding clarity. We will revise the paper to make it more self-contained, particularly by providing more background on QPM [1] and how it differs from prototypical methods, as also suggested by Reviewer 7DuL. Regarding the questions:\n\n> Please provide some discussions around scalability of the proposed approach. \n\nWe gladly elaborate on the scalability of CHiQPM. As shown in Table 1, our model scales exceptionally well to the large-scale ImageNet dataset, achieving a remarkable accuracy of just 0.8 percentage points less than the dense, non-interpretable baseline. This performance is achieved using just 50 features for all 1000 classes. \nAs we described in Section J, scaling to a dataset of this magnitude introduces certain limitations for interpretability. For instance, the learned concepts become more abstract, saliency maps less faithful, and some features exhibit polysemanticity.  Nevertheless, our results show that CHiQPM successfully identifies the high-level features needed for accurate classification.\nFinally, the time it takes to solve the QP scales exponentially with the number of features to select [1]. But remarkably, just 50 features are sufficient for the very complex ImageNet dataset, making this a very manageable limitation in practice.\nIn conclusion, CHiQPM demonstrates robust scalability. While its interpretability does not scale as excellently as its accuracy, it still offers the best global interpretability on ImageNet. We will summarize these limitations in scaling in a dedicated section in Section J. \n\n> the evaluation miss some common metrics for local XAI, e.g., delete/insert scores, Pointing Game; \n> \n> Given the proposed method can do both global and local explanations; if the author consider to compare with some common XAI benchmarks, e.g., OpenXAI. \n\nWe would like to point out that we do not introduce a new saliency map method whose faithfulness should certainly be evaluated using delete/insert or similar [2] scores, that alleviate some OOD Problems of the former metrics. Instead, the proposed method mainly offers improved local interpretability as one can provide saliency maps via any of the available methods for each of the individually meaningful, general and diverse features separately, as well as via the novel hierarchical explanations.  \nHowever, as visualizations of the activations of individual features are shown, it is a good idea to evaluate their faithfulness. Following your suggestion, we evaluate a form of the Pointing Game, similar to the initial form in [3], using segmentation masks provided for CUB200 [4]. Specifically, we calculate the fraction of the activation of the GradCAM saliency map that is focused on the segmented bird S. \n\n$\\phi = \\frac{GradCAM}{\\sum GradCAM} \\odot S$ \n\nAs the bird can be considered the region of the image responsible for the classification, a higher overlap with the segmentation indicates that the saliency maps localize more faithfully. Importantly, one would not necessarily expect an overlap of 100%, as the edge region is relevant to describing the shape, causing activations both on and off the segmentation. Across the entire test dataset, CHiQPM’s GradCAM focusses to 83.4% on the bird, whereas the Dense baseline only does so with 75.8%. Hence, CHiQPM has activation maps with improved faithfulness on CUB-2011, validating their use for saliency maps of individual interpretable features. We will include this experiment in the appendix.\n\n| Model  | Accuracy | Average $\\phi$ = GradCAM Overlap with Segmentation |\n|:-------|:--------:|:--------------------------------------------------:|\n| Dense  | 86.6%    | 75.8%                                              |\n| CHiQPM | 85.3%    | 83.4%                                              |\n\n> In practice, can the authors give some examples of how a conformal set of features is better than, say, the use of top-1 or top-k features for decision-making. \n\nWe gladly elaborate on why using our approach is preferential to using top-k features for decision-making. We understand “conformal set of features” as the set of features $\\hat{F}_i^c$ (Eq. 8), from which the descending classes in the hierarchical explanation are predicted, e.g. {Green, Blue} in Figure 2. There are two main differences to just using top-k features for decision-making. \n\n1. The conformal set of features is determined via conformal prediction which is considered an effective tool for supporting experts [47,48]. The level in the hierarchy is determined dynamically for each test sample via our novel interpretable conformal prediction method to guarantee that the target coverage is met. That way, the conformal set of features also transports certainty via its set size. In contrast, predicting with a fixed k, e.g. top-1, also returns large inefficient sets, when the test sample can be easily classified.  \n2. Practically, using top-k features is not suitable for building a hierarchy, as those top-k features might not belong to any one class together. This also causes top-k to be slightly less efficient than predicting with a fixed level n in the sample specific hierarchy. To demonstrate this, we compare between predicting with top-k features and at level n.\n\n|                | Level 1  | Level 2 | Level 3 | Level 4 | Level 5 |\n|:---------------|:--------:|:-------:|:-------:|:-------:|:-------:|\n| **Set Accuracy** | 96.5   | 93.8   | 91.6  | 89.5   | 85.3   |\n| **Set Size** | 20.17  | 4.43  | 2.12  | 1.45 | 1.00  |\n\n|                  | Top-1   | Top-2  | Top-3  | Top-4  | Top-5  | Top-6  | Top-7  | Top-8  |\n|:-----------------|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| **Top-K Accuracy** | 96.3  |93.2 | 90.8 | 89.0 | 85.5 | 85.4 | 85.3 |85.3 |\n| **Set Size** | 20.17 | 4.48 | 2.19 | 1.51 | 1.08 | 1.02 | 1.01| 1.01 |\n\nNote that predicting with top-1 feature will always return the large set of all classes with that feature which is even less accurate than predicting at Level 1 as there simply might be no class that has the remaining dominant features apart from Top-1. A concrete example can illustrate this failure case: A male bird of the species “Scarlet Tanager” is red, while females are olive-green. The feature \"red\" might be the top activated feature for a male, but CHiQPM may not use \"red\" to represent the class Scarlet Tanager because it's not a consistent feature. Thus, relying on the top-1 feature \"red\" would lead to an incorrect prediction. We will add this experiment and discussion to the appendix. \n\nWe again thank you for your review and suggestions that strengthen our paper. \n\n[1]: Norrenbrock, Thomas, et al. \"QPM: Discrete Optimization for Globally Interpretable Image Classification.\" The Thirteenth International Conference on Learning Representations. 2025. \n\n[2]: Zheng, Xu, et al. \"F-Fidelity: A Robust Framework for Faithfulness Evaluation of Explainable AI.\" The Thirteenth International Conference on Learning Representations. 2025. \n\n[3]: Zhang, Jianming, et al. \"Top-down neural attention by excitation backprop.\" International Journal of Computer Vision 126.10 (2018): 1084-1102. \n\n[4]: Farrell, R. (2022). CUB-200-2011 Segmentations (1.0) [Data set]. CaltechDATA."}, {"review_id": "xamjjSG9a1", "reviewer": "Reviewer_TSHT", "summary": "QPM introduce a way to make convolutional networks globally interpretable by representing each class as the logical \"or\" of a very small, fixed set of feature maps. \n\nThis yields ultra‐compact, class‐wise explanations but lacks fine‐grained, instance‐level detail and makes no provision for calibrated uncertainty in predictions. CHiQPM, builds directly on QPM:\n\n(a) The discrete quadratic program is augmented to force many class pairs to share nwc – 1 features, creating a tree‐like hierarchy of explanations rather than a flat, per‐class OR.\n\n(b) After solving the QP, the model is fine-tuned with a novel Feature Grounding Loss and ReLU-based regularization that:\n\n(i) equalizes feature activation strength within each class representation and \n\n(ii) suppresses spurious activations, dramatically improving how well features align to human‐understandable concepts.\n\n\n(c) By traversing the learned hierarchy and using a tailored nonconformity score, CHiQPM produces calibrated prediction sets whose size meets a desired coverage target while keeping the returned classes semantically coherent (e.g. birds of similar shape), a capability missing in both QPM and flat set predictors.\n\nExperiments on CUB‑2011, Stanford Cars and ImageNet‑1K across four backbones show (a) accuracy kept similar to a dense CNN (b) higher structural grounding than QPM.", "questions": "1. Are there other hierarchy construction choices? \n2. Grounding is measured via concept annotations; however, do humans actually find the hierarchical paths helpful? A small-scale user study (e.g 20 domain experts evaluating 100 samples) could validate the claimed usability of the explanations.\n3. What is the sensitivity to λ_feat and maximum depth of the hierarchy? Please add a grid.\n4. Are there limitations regarding class imbalance?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. CHiQPM builds on QPM’s adds a hierarchical tree structure, giving both a concise global summary and sample-specific decision paths.\n2. The discrete QP formulation forces most class pairs to share nwc – 1 features, yielding an explanation hierarchy rather than a flat feature assignment.\n3. Introducing L₍feat₎ plus a ReLU-based fine-tuning step sharply improves alignment between learned features and human concepts.\n4. Strong experiments\n\nWeaknesses:\n1. Point‐accuracy improvements over QPM are very small and even negative on some backbones, limiting appeal outside interpretability‐focused audiences.\n2. Dense notation (e.g. reusing “M” for multiple matrices) and scattered rationale for key design decisions (why ReLU vs. soft-threshold) hinder readability.", "was_revised": true, "final_justification": "Authors response is great, my rating doesn't change.", "rebuttal": "Thank you for your positive review, acknowledging several strengths and posing interesting questions. We will of course try to improve readability at the points mentioned and answer each question separately.\n\n> Are there other hierarchy construction choices?\n\nOur used hierarchy based on order of the features of the predicted class has significant advantages compared to other choices. Its sample specific nature lends itself to a local explanation, as one can consider the order of the features as how dominant they appear in the image. Additionally, that enables traversing up the hierarchy in a meaningful way, since the least certain features get omitted first, causing accurate and efficient sets at each step. We believe that the main alternative is a class specific fixed order of features, as that would enable a global hierarchical explanation for each class. The comparison between a fixed order of features for each class, computed based on the average order on training data and our prediction up the hierarchy is shown below. Evidently, predicting with such a fixed order causes less accurate inefficient sets compared to our sample specific dynamic hierarchy. For example, our dynamic hierarchy reaches 91.6% accuracy with an average set size of 2.1, while the fixed hierarchy never reaches that accuracy. We believe that this is likely due to the unique feature, e.g. the red eye of the Bronzed Cowbird, being on average quite important to that class and thus causing prediction sets that are too specific to the top-1 prediction. This experiment showcases the superiority of using our sample specific dynamic hierarchy and will therefore be added to the appendix, as it further validates our method. Many thanks for this suggestion which is further strengthening our contribution.\n\nComparison to fixed order of features per class based on average order of Features:\n\n**Sample Specifc Dynamic Feature order as used by CHiQPM:**\n\n| | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| **Set Accuracy** | 96.5   | 93.8   | 91.6  | 89.5   | 85.3   |\n| **Set Size** | 20.17| 4.43 | 2.12 | 1.45 | 1.00 |\n\n**Static Feature Order based on Average Order of Features on Training Data for each class.**\n\n| | Level 1 | Level 2 | Level 3 | Level 4 | Level 5 |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| **Set Accuracy** | 89.7 | 87.4|86.2 |85.6 | 85.2 |\n| **Set Size** | 18.52 | 2.85 | 1.30 | 1.09 | 1.00 |\n\n> Grounding is measured via concept annotations; however, do humans actually find the hierarchical paths helpful? A small-scale user study (e.g 20 domain experts evaluating 100 samples) could validate the claimed usability of the explanations.\n\nWhile we agree with the reviewer that such a user study would be very valuable, we are unfortunately unable to accommodate that in this timeframe. We will, however, add it to a newly added future Work section in the appendix.\n\n> Are there limitations regarding class imbalance?\n\nThank you for this intriguing question. We believe that CHiQPM can be more robust to class imbalance than baseline methods for two reasons:\n\n1. The quadratic problem is solved with no weighting based on prevalence of the class in the training data.\n2. As underrepresented classes get the same number of feature assignments, the class will be predicted if all its general features are activating.\n\nHowever, for that to work, the assignment between features and classes has to be good. When some classes are underrepresented, the constants in the QP might not be good enough to obtain an assignment well suited for fine-tuning. Empirically, these effects seem to balance out, indicating no quantifiable limitation regarding class imbalance. We calculated the Pearson correlation coefficient between number of training samples and test accuracy for each class on the slightly imbalanced datasets Stanford Cars and ImageNet. As expected, there is a positive correlation between the number of training samples on both datasets, with a negligible reduction for CHiQPM. We will add this discussion and experiment to the appendix. Additionally, we add robustness to class imbalance as a direction for future work as we believe in the aforementioned potential. Future work might include considering the number of samples per class during the QP as a form of certainty or use some additional form of supervision, e.g. via the class names, to make the constants more robust.\n\n**Pearson Correlation Coefficient between number of training samples and test accuracy per class:**\n\n| Model | Stanford Cars | ImageNet |\n| :--- | :---: | :---: |\n| **Dense** | 0.12 | 0.18 |\n| **CHiQPM** | 0.11 | 0.17 |\n\n> What is the sensitivity to $\\lambda_{feat}$ and maximum depth of the hierarchy? Please add a grid.\n\nThank you for that question, as we indeed missed including the ablation. The maximum depth of the hierarchy is the number of features assigned to each class $n_{wc}$. Therefore, we swept $n_{wc}$ and $\\lambda_{feat}$ and report accuracy, sparsity and alignment. As generally expected, $\\lambda_{feat}$ improves alignment and sparsity across various sparsities. Additionally, features become sparser and more aligned with decreasing $n_{wc}$, as they must activate on fewer samples and can therefore learn less abstract shared concepts. We gladly add these grids for their valuable insights to the appendix.\n\n**Accuracy:**\n\n| $n_{wc}$ \\ $\\lambda_{feat}$ | 0.031 | 0.1 | 0.31 | 1.0 | 3.0 | 10.0 | 30.0 |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **6** | 85.2 | 85.2 | 85.1 | 85.2 | 85.0 | 85.1 | 84.5 |\n| **5** | 85.3 | 85.1 | 85.2 | 85.2 | 85.3 | 85.2 | 84.5 |\n| **4** | 85.2 | 85.1 | 85.1 | 84.9 | 85.1 | 85.1 | 84.8 |\n| **3** | 84.9 | 84.7 | 84.8 | 84.9 | 85.2 | 85.3 | 84.5 |\n\n**Sparsity:**\n\n| $n_{wc}$ \\ $\\lambda_{feat}$ | 0.031 | 0.1 | 0.31 | 1.0 | 3.0 | 10.0 | 30.0 |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **6** | 42.6 | 41.0 | 35.5 | 29.0 | 24.8 | 23.5 | 25.0 |\n| **5** | 38.8 | 37.2 | 32.6 | 26.4 | 22.3 | 21.2 | 22.9 |\n| **4** | 34.8 | 33.4 | 29.2 | 23.7 | 19.8 | 18.5 | 20.3 |\n| **3** | 30.8 | 29.2 | 26.0 | 20.7 | 17.1 | 15.8 | 17.3 |\n\n**Feature Alignment:**\n\n| $n_{wc}$ \\ $\\lambda_{feat}$ | 0.031 | 0.1 | 0.31 | 1.0 | 3.0 | 10.0 | 30.0 |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **6** | 2.8 | 2.9 | 3.1 | 3.3 | 3.5 | 3.6 | 3.7 |\n| **5** | 3.1 | 3.2 | 3.4 | 3.7 | 3.8 | 3.9 | 4.0 |\n| **4** | 3.6 | 3.7 | 3.9 | 4.3 | 4.5 | 4.6 | 4.6 |\n| **3** | 4.3 | 4.4 | 4.6 | 5.0 | 5.3 | 5.3 | 5.3 |"}, {"review_id": "HPmK6lyTa9", "reviewer": "Reviewer_7DuL", "summary": "This paper proposes a globally and locally interpretable image classification model that builds upon QPM with improved contrastive class representations and hierarchical explanations. CHiQPM encourages class-wise feature sharing and introduces a built-in interpretable conformal prediction mechanism by traversing explanation trees formed from shared feature activations. Experiments on CUB, Stanford Cars, and ImageNet-1K demonstrate that CHiQPM maintains high accuracy while offering calibrated predictions.", "questions": "1. How is the interpretability of local features ensured? While contrastiveness is enforced, it does not imply semantic consistency. How do we know the activated features correspond to concepts meaningful to humans?\n2. What is the generalization ability of the model structure? If applied to a complex dataset with highly abstract or noisy classes, how would CHiQPM perform? What happens if class pairs do not share enough features to construct a meaningful hierarchy?\n3. How does CHiQPM compare with prototype methods like ProtoTree [Nauta et al., 2021]? What are the key advantages of your approach?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper aims to provide global explanations that are encouraged, especially the design of interpretable models rather than relying on post-hoc explanations.\n2. The idea of inducing hierarchical explanation paths by enforcing shared features between similar classes is novel.\n3. The paper is easy to follow, making the objectives accessible to a broad audience.\n\n**Weaknesses**\n1. The model heavily relies on learned mid-level features, which are not guaranteed to correspond to human-interpretable concepts. While the authors introduce a contrastive objective and grounding loss, these only encourage sparsity but not semantic alignment. This leaves the interpretation paths inherently fragile and possibly misleading. Moreover, relying on a small set of local features per class may hurt performance, especially if global context is important.\n2. The proposed framework assumes that similar classes will share a few features. However, for more abstract or complex classification tasks, a larger number of features per class may be necessary. As the feature pool expands, feature overlap between classes may decrease, undermining the construction of hierarchical structures and weakening the interpretability claims.\n3. The paper lacks analysis of how the depth and width of the explanation trees vary under different configurations (e.g., feature pool size or class granularity). Such ablations are crucial to understand how robust or shallow the constructed explanation paths truly are.", "was_revised": true, "final_justification": "While CHiQPM offers a more flexible hierarchy for model explanations, it provides no guarantee on what these features actually represent. Therefore, I remain unconvinced and maintain my borderline reject assessment.", "rebuttal": "Thank you for your review, acknowledging several strengths and bringing up important points to discuss. As some arguments build on each other, we would like to answer the questions with answers building on each other.\n> How does CHiQPM compare with prototype methods like ProtoTree [Nauta et al., 2021]? What are the key advantages of your approach?\n\nThe distinction to prototypical methods is a crucial point so we would like to elaborate. CHiQPM offers global interpretability as it has interpretable class representations that are built from shared, general, contrastive and diverse features which empirically align well with human attributes on CUB-2011. Therefore, one can understand the class representations of a CHiQPM globally. \nFor example, the CHiQPM explained in Figure 1 represents a Bronzed Cowbird as:\n* A Bird with a red eye\n* A Bird with head and belly of black birds\n* A Bird with the neck of black birds\n\nIn contrast, Prototypical methods represent classes via prototypes which typically (barring PIP-Net) are exact patches from the training data. The crucial difference is that these prototypes are class specific. This originates from training the feature (or prototype) extractor with classification loss, while allowing it to learn at least one prototype per class. Consequently, the class representation of a prototypical method for a Bronzed Cowbird might be:\n* A Bronzed Cowbird\n* … (They typically have significantly more prototypes assigned per class. Depending on the model, the prediction either depends on just one class-specific feature, or all/most assigned prototypes are class-specific.)\n\nWhen judging prototype visualizations, humans typically suffer from confirmation bias and think that the model indeed represents the class with the human concept, while it is a class detector. This can be quantified via Class-Independence [35], with many prototypes of PIP-Net being only (> 99%) similar to samples of just one class. Additionally, human studies have shown that humans cannot predict the presence of such prototypes [16,19]. Figure 7 in the ProtoTree paper [31] itself illustrates this very effectively: When the height of the ProtoTree enables one prototype per class (height = 8; $2^8 = 256 > 200$ = number of classes in CUB-2011), the accuracy jumps to a competitive level and does not increase further with increasing height. Notably, even with more allowed prototypes at a height of 9, allowing $2^9 = 512$, ProtoTree still just learns about one per class with 201.6 (see Table 2 in the paper [31]). Finally, ProtoTree learns a deep decision tree, which might seem similar to our hierarchical explanations. However, as its features are not general, the usefulness is very limited.\n\nCHiQPM follows the line of Work of Q-SENN [34] and QPM [35], where classes are represented with shared general features instead of class-specific ones to enable global interpretability. Crucial elements for that are to learn less features than there are classes in the dataset and to ensure that features are equally important for multiple classes. Therefore, typically no more than 50 features are learned, while their weights to classes are binary in QPM and CHiQPM. That way, one feature on CUB has to activate on general, shared concepts of its assigned classes and cannot detect class-specific properties. As shown in [35], this strong restriction on the space of learnable features already causes the features to become class-independent and contrastive.\n\nWe thank the reviewer for highlighting this crucial point. We agree that clarifying the distinction to prototypical methods is essential for understanding the significance of our work. We will add this detailed comparison to the appendix and emphasize the core differences in the main paper.\n> The model heavily relies on learned mid-level features, which are not guaranteed to correspond to human-interpretable concepts. While the authors introduce a contrastive objective and grounding loss, these only encourage sparsity but not semantic alignment. This leaves the interpretation paths inherently fragile and possibly misleading.\n> \n> How is the interpretability of local features ensured? While contrastiveness is enforced, it does not imply semantic consistency. How do we know the activated features correspond to concepts meaningful to humans?\n\nAs we discussed in the limitations Section J, it can indeed be considered a limitation of CHiQPM that it is not enforced to learn human-interpretable concepts. Instead, it by-design learns general and contrastive features that classify the dataset well. However, the feature alignment metric quantifies that CHiQPM empirically does learn features that correspond to concepts meaningful to humans, with an average increased feature value of 3.8 times its mean when the most aligned concept is present (Table 3). It seems that the concepts that humans have annotated for the birds in CUB-2011 are among the most effective general and contrastive concepts, which is why CHiQPM learns to detect them.\nNotably, this metric also verifies the qualitative impression that the saliency maps of individual classes give. For example, the features 7,8 and 17 of Figure 1 indeed return high alignment values of 2.2, 3.9 and 2.5 for the attributes Red Eye Color, Black Belly Color and Black Throat Color their saliency maps indicate. Please note that the novel Feature Grounding Loss improves the alignment with these attributes, but as you correctly mentioned does not generally prefer the hard-to-define category of human-interpretable concepts, but rather further encourages general contrastive features.\n\nFinally, we would like to point out that restricting the learnt features to relevant concepts that are known to mankind a priori is not necessarily an advantage, as discussed in Point 3 of Section J. The open setting of learning general features that are shared (or the main difference) between classes could lead to scientific discovery, as one can learn where patterns are that are suited for differentiation.\n> Moreover, relying on a small set of local features per class may hurt performance, especially if global context is important.\n\nWe acknowledge that it might be possible that the few features per class become too few in certain conditions. However, previous and this work show that the accuracy on already quite challenging datasets tends to saturate after 5 features per class. For reference, on CUB-2011, 3 features per class already seem sufficient for CHiQPM:\n\n| $n_{wc}$ | Accuracy |\n| :---: | :---: |\n| 3 | 85.2 |\n| 4 | 85.1 |\n| 5 | 85.3 |\n| 6 | 85.0 |\n\nWe further want to clarify that we do not enforce local features, but rather bias the model towards learning such features via the Feature Diversity Loss and feature selection bias in the QP. However, if a more global feature is required, it is still learnt, such as Feature 8 in Figure 1. We would also like to point out that CHiQPM performs remarkably well on ImageNet, where the scale of features is drastically varying across images and prototypical models with prototypes restricted to one patch are generally not even converging. We will include this ablation in the appendix and add your point as limitation.\n> The proposed framework assumes that similar classes will share a few features. However, for more abstract or complex classification tasks, a larger number of features per class may be necessary. As the feature pool expands, feature overlap between classes may decrease, undermining the construction of hierarchical structures and weakening the interpretability claims.\n> \n> What is the generalization ability of the model structure? If applied to a complex dataset with highly abstract or noisy classes, how would CHiQPM perform? What happens if class pairs do not share enough features to construct a meaningful hierarchy?\n\nWe consider ImageNet as already quite complex with very noisy labels [1] and deduce that CHiQPM’s predictive performance generalizes very well to such a complex dataset with just 5 features per class and 50 in total. However, as described in Section J, the interpretability of the features does not scale as well. For a more thorough discussion, please see our first response to Reviewer7M7N.\n\nAs discussed in Section 3.1, 4.3 and D, the tree density should ideally be set using the calibration data, as it is hard to know how many class pairs share sufficient concepts. If that analysis determines that $\\rho$ should be set to 0, the resulting CHiQPM will lose some of its interpretability, but will still be the most accurate model with global interpretability. Nevertheless, as mentioned in the Limitations (lines 750-753), if classes truly share nothing or are very few in total, CHiQPM is not applicable.\n> The paper lacks analysis of how the depth and width of the explanation trees vary under different configurations (e.g., feature pool size or class granularity). Such ablations are crucial to understand how robust or shallow the constructed explanation paths truly are.\n\nThe depth of the explanation tree generally equals the number of features assigned to each class, as every activating feature is part of the tree. We understand the width of the tree as the number of leaf nodes in the tree and have measured that width of the total explaining tree as in Figure 23 and the tree below Level 1, as e.g. in Figure 2. As one expects, with increasing number of features, the resulting trees become smaller, as fewer classes are assigned to each individual feature. We will add this ablation to the appendix.\n\n| Number of Features  | 20 | 30 | 40 | 50 | 60 |\n| :--- | :---: | :---: | :---: | :---: | :---: |\n| # Leafs in total Tree | 73.9 | 64.8 | 57.9 | 53.5 | 50.2 |\n| # Leafs in Tree below Level 1 | 24.9 | 17.8 | 14.1 | 11.9 | 10.3 |\n\nWe hope these clarifications and new results address your concerns.\n\n[1] Shankar, Vaishaal, et al. \"Evaluating machine accuracy on imagenet.\" International Conference on Machine Learning. PMLR, 2020."}]

Author rebuttal:

Meta-review: This paper received mixed reviews. After discussion, reviewers agreed to accept this paper.  The authors are encouraged to address the major concerns in the camera-ready version.
1. Add formal analysis for essential steps.
2. Improve the writing to make it easier to follow
3. Accuracy improvement over QPM is incremental.
4. Streamline notations.

The authors’ rebuttal and messages were carefully ready, discussed, and considered.
Decision: Accept (poster)