QPM: Discrete Optimization for Globally Interpretable Image Classification
Authors: Thomas Norrenbrock
Timo Kaiser
Sovan Biswas
Ramesh Manuvinakurike
Bodo Rosenhahn
Venue: ICLR 2025 Poster (2025)
Keywords: explainable AI
Global Interpretability
Sparse Representations
Compactness
Local Interpretability
Image Classification
Discrete Optimization
Primary area: interpretability and explainable AI
TL;DR: Using Discrete Optimization to get a Model with faithfully globally interpretable class representations with SOTA Accuracy among interpretable models.

Abstract: Understanding the classifications of deep neural networks, e.g. used in safety-critical situations, is becoming increasingly important. While recent models can locally explain a single decision, to provide a faithful global explanation about an accurate model’s general behavior is a more challenging open task. Towards that goal, we introduce the Quadratic Programming Enhanced Model (QPM), which learns globally interpretable class representations. QPM represents every class with a binary assignment of very few, typically 5, features, that are also assigned to other classes, ensuring easily comparable contrastive class representations. This compact binary assignment is found using discrete optimization based on predefined similarity measures and interpretability constraints. The resulting optimal assignment is used to fine-tune the diverse features, so that each of them becomes the shared general concept between the assigned classes. Extensive evaluations show that QPM delivers unprecedented global interpretability across small and large-scale datasets while setting the state of the art for the accuracy of interpretable models.

Published as a conference paper at ICLR 2025
QPM: DISCRETE OPTIMIZATION FOR GLOBALLY INTERPRETABLE IMAGE CLASSIFICATION
Thomas Norrenbrock, Timo Kaiser & Bodo Rosenhahn Institute for Information Processing (tnt) L3S Leibniz Universität Hannover, Germany (norrenbr, kaiser, rosenhahn @ent. uni haanovhn.neeeeeee
Sovan Biswas & Ramesh Manuvinakurike
Intel Labs, USA (sovan. biswas, ramesh..e hww.ess...maiiaaaaaieeeeeo......
ABSTRACT
Understanding the classifications of deep neural networks, e.g. used in safety- critical situations, is becoming increasingly important. While recent models can locally explain single decision, to provide faithful global explanation about accurate model's general behavior is a more challenging open task. Towards that goal, we introduce the Quadratic Programming Enhanced Model (QPM), which learns globally interpretable class representations. QPM represents every class with binary assignment of very few, typically 5, features, that are also assigned other classes, ensuring easily comparable contrastive class representations. This compact binary assignment found using discrete optimization based on predefined simi- larity measures and interpretability constraints. The resulting optimal assignment is used to fine-tune the diverse features, so that each of them becomes the shared general concept between the assigned classes. Extensive evaluations show that QPM delivers unprecedented global interpretability across small and large-scale datasets while setting the state of the art for the accuracy of interpretable models.
Rottweiler Features
Class Images
Rortweiler
Doberman
Doberman Festures
Figure 1: Faithful global interpretability of QPM: Without any additional supervision, QPM learns to represent Rottweiler and Doberman using 5 diverse and general features. QPM faithfully explains that differentiates them exclusively via their visibly distinct head.
INTRODUCTION
Deep Learning has made remarkable advances in various fields, such image classification, segmen- tation or generation (Krizhevsky et al. 2012: Kirillov et al. [2023] Rombach et al., 2021 Ramesh et al. 2022). For high-stakes decisions, e.g. applying image classification in the medical domain. legis ltioon moves towards requiring a certain level of interpretability (Veale & Zuiderveen Borgesius 2021), whose measurement is fairly open task its own. However, some desirable and measurable

---

Published as a conference paper at ICLR 2025
Features
Classes
Features
Classes
1.8
Maximize: x
" Qx + c²xx
Ideselected)
Feature Selection Sparse Assignment
x=
Dense
QPM
Figure 2: Exemplary Application of the QP dense model with just features and classes with the aim of selecting 3 features and assigning 2 per class. The different weights are indicated by thickness of connection and color indicates sign. The result is a binary assignment of selected features to classes. Typical values for Resnet50 (He et al. (016)) on CUB-2011 (Wah et al. 2011 are selecting 50 features out of 2048 and assigning each of the 200 classes
qualities of explanations have been identified (Miller 2019). Human- explanations should be contrastive (Lipton| 1990), diverse (Alvarez Melis & Jaakkola 2018), general and compact (Read & Marcus-Newhall 1993). As humans can consider 7 + 2 cognitive aspects at once (Miller 1956). an explanation size of up to is desirable. Additionally, an explanation should faithfully explain the model, which is where many post-hoc methods fail (Kindermans et al. 2019 Adebayo al. 2018
Daras & Dimakis 2022). Therefore, we focus on models that are interpretable by design with built-in faithful explanations.
Previous works, such as SENN (Alvarez Melis & Jaakkola 2018). Q-SENN (Norrenbrock et al.
2024), Concept Bottleneck Model (CBM) (Koh et al. 22020), Label-free CBM (Oikarinen et al. 2023)
PIP-Net (Nauta et al., 2023), ProtoPool (Rymarczyk et al. 2022), ProtoTree (Nauta et al. 2021 }
ProtoPNet (Chen al.] [2099) the SLDDD-Model (Norrenbrock al. (022) rely on combining
understandable features in an interpretable manner. However, while most models can offer convincing
local explanations for single decision, they struggle with the global explanation of their behavior in general. Some models with global interpretability do not show competitive accuracy (Oikarinen
ct al.
/ Koh et al. 2020) and it is debated (Molnar) 2020). if ensembles of very deep decision
2023
(Nauta et al. (021) or dense high- high-dimensional linear layers (Koh et al. 2020 Rymarczyk ct al. 2022 Alvarez Melis & Jaakkola 2018) are truly intrinsically interpretable as they lack desired qualities like compactness. For that reason PIP-Ner focuses on learning sparse class representations.
tress
These representations lie in a high dimensional feature space, which causes PIP-Ner's s features to be connected to very few or only one class each. This leads to the emergence of features that are already detecting the class and no general concept. The sparse representations of PIP-Net thus have classes are represented with themselves. To alleviate that issue, the SLDD-Model and Q-SENN reduce both dimensions of compactness: They not only reduce the number of features per class Tww,, which in isolation leads class-specific features but also the number of features in total n to be significantly below the number of classes Nlg- That causes each of the fewer features to be assigned multiple classes, which prevents the emergence of class detectors. However, these models still have shortcomings when it comes to global interpretability. Their class representations are real-valued, or ternary for Q-SENN, include bias, and are composed of varying number of features. Therefore, the global class explanations are hardly comparable or contrastive. In this work, we introduce the Quadratic Programming Enhanced Model (QPM) that offers inter- pretable class representations and sets new state of the art for the accuracy of compactness-baas based interpretable models. It represents every class with the binary assignment of a low user defined number of features nww. which themselves 5, in line with previous work (Norrenbrock et al., 2224; 2022), to accommodate for human limita- tions (Miller 1956). As shown in fig. QPM offers built-in faithful global explanations for classes
no interpretable meaning,
contrastive, general and diverse. We typically choose
2

---

Published as a conference paper at ICLR 2025
Table 1: Properties of class representation for class i, y, wif + b, for CUB-2011: Only QPM represents each of its classes with the binary assignment of fixed number of general features (quantified in table 3 and no class Bias. Therefore, classes can also be represented as set of 5 feature indices S,, y/ jjes, Sj. These contrastive class explanations enable faithful global interpretability. If applicable, all methods are configured to nowe = 5 and n 50.
Method
Size of Wi
Equal Class Sparsity
No Class Bias
Contrastive Representation
Baseline Resnet50)
W; R
glm-sugas
Wi R809
PIP-Net
Wi E R
X
ProtoPool
Wi E R202
X
SLDD-Model
Wi E R
X
X
Q-SENN
Wj E { a, 0, a.05
X
X
X
QPM (Ours)
Wi E 10,1)
S. [1, 50]
and enables the intuitive comparison of different learned class representations. These easy compar- isons between compact binary class representations even enable reasoning about the differentiating feature between the classes, like the head in fig. The improvements in faithful global interpretability of class representations are summarized in [able The crucial step training a QPM is solving a binary QP, applied to a dense black-box model, which jointly finds an optimal solution to both the selection of reduced subset of the model's features and the sparse assignment between the features and classes, as shown in fig. 2 It maximizes the similarity between features and their assigned classes, while minimizing the similarity of jointly selected features. Further, the linear term can steer the selection towards desired biases, while the desired interpretability is incorporated via constraints. This optimal solution is then fixed for the following fine-tuning during which the features adapt to their assigned classes. As every class is assigned to the same number of features, each of the features detects shared general concepts between its assigned classes instead of also detecting the entire class. This leads to state-of saaee-of-thee ti-th--art accuracy. Finally, the as- signments are not maximizing inter-class distance, resulting in more similar representations for similar classes and form of structural grounding. Code: httpt:://///..... / thub.
Our main contributions are as follows:
We propose the Quadratic Programming Enhanced Model (QPM), which incorporates an optimal feature selection and their binary assignment of a few, e.g. features per class. is found by formulating the quadratic problem and solving it optimally.
We demonstrate improvements in accuracy, compactness and structural grounding of QPM on multiple benchmark datasets and architectures for image classification, including ImageNet- IK (Russakovsky et al.] 2015). Due to optimally using the given capacity, QPM sets the new state of the art for compactness-basee globally interpretable models.
We show that the learned features exhibit several desired quantifiable properties, such as contrastiveness, generality and diversity, and can be steered towards user-defined criteria.
Representing classes as contrastable compact set of these general features makes QPM faithfully globally interpretable, while further closing the accuracy gap to black-box models.
2 RELATED WORK
Research towards Interpretable machine learning includes the direct design of models providing inter- pretability by themselves (Alvarez Melis & Jaakkola [0018] Sawada & Nakamural [002 Norrenbrock
ct al. 2022, Nauta et al. 2023 | Rymarczyk et al.] 2022 [ et al.) [022 Marconato et al.
2022 Koh et al. |020 Rymarezyk et 2021 Chen et al. 2019) or to find post-hoc methods which
aim to explain the decision process or single features of the model (Kim et al. 20188] Bau et al. 2017]
McGrath et al. 2022 Fel et al. 2023 Yuksekgonul et al. 2022 Kalibhat et al. 2023] Oikarinen & Weng 2023). As our method is designed to find compact set of human-unddersandable features, our work can be assigned to the former type, which we focus on within this section. However, the alignment of the learned features of our proposed QPM with human attributes can be guided by the post-hoc methods. When considering the interpretability of a model, a distinction is made between local interpretability, which refers to the explanation of single decision, and global interpretability,
3

---

Published as a conference paper at ICLR 2025
Train Dense Model
Compute Feature-Claan Similarity A, Feature-Feature Similarity R and Feature-Bias b for QP
Solve QP for Feature Selection and Assignment
Finetune Features
QPM
with Laiv
Figure 3: Overview of our proposed pipeline construct QPM
which describes the holistic behavior of the model over the entirety of dataset (Molnar 2020). For local interpretability, B-Cos Networks (Böhle et al. (023) already offer faithful explanations the form of saliency maps. Therefore, this work focuses on the more challenging global interpretability, which also improves local interpretability. In the social sciences (Miller 2019), fuienn-rriendly explanations are contrastive (Lipton] 1990), concise and general (Read & Marcus-Newhal 1993). Further, SENN (Alvarez Melis & Jaakkola 2018) describes diversity and grounding as desirable attributes for features of interpretable model. Grounding refers to the alignability with any human concept and is very difficult to quantify, as one would need a full dataset of potentially learned concepts. Problematicallyy. deep neural networks typically exhibit superposition and polysemantic neurons (Scherlis et al. [2022] Elhage et al. 2022 Templeton 2024). which is why we focus on more clearly quantifiable aspects in this work.
Models such as Prototree (Nauta et al. (021). ProtoPNer (Chen et al. 2019). ProtoPShare (Rymar czyk et al., 2021). ProtoPool, and PIP-Ner aim to learn prototypes from data by employing deep feature extractors. These prototypes' similarities are subsequently integrated into interpretable models. However, the extent of their interpretability remains debatable, as Kim et al. (2022) and Hoffmann et al. (202T) reveal gap between human and computed similarities. Similar to this work, PIP-Ner also aims for compactness via sparse weights in the final decision layer. However, they apply a local optimization that aims for sparsity solely, resulting in big set of used features with many of them being slass-specific. Norrenbrock et al. 2022 2024) additionally select a compact feature set for their SLDD-Model and Q-SENN, where class is to be related to only a few features. Their diversity is ensured through the Feature Diversity Loss Laiv. which incurs a higher cost when highly activated and weighted features localize the same region. For both feature selection and the computation of the sparse layer, glm-saga (Wong et al. (02T) is used. It locally and iteratively optimizes the problem, leading to suboptimal feature selection and continuous weights. In contrast, our global optimization with user-defined stcerable criteria jointly finds an optimal selection of the required number of features and computes their binary assignments. This leads to a more effective use of the allocated capacity and built-in easily interpretable class representations for global interpretability. Another line of research is based on the Concept Boutlencc Model (CBM) which initially predicts the labeled concepts within a given dataset and subsequently leverages a basic model to predict the target category based on these identified concepts. This approach remains an area of active exploration and development (Sawada & Nakamura 2022 Zarlenga et al. 2022] Marconato et al.) 2022; Oikarinen et al.] 2023), but is limited by the annotations, or in case of the Label-free CBM by the vasion-language model, resulting in subpar accuracy and compactness. Finally, Rosenhahn (023 applies discrete optimization to obtain sparse neural networks (Glandorf* et al. 2023).
METHOD
Our proposed QPM is designed for the interpretable classification of an image as a class E Ca.,2....,Co Cn. )- The QPM uses deep feature extractor & to compute feature maps xhu of width wM and height hM and averages them into a feature vector f* E Rd7 The classification result y E R of the QPM is the matrix multiplication between the sparse binary matrix W E (0, m and the features f* formalized as y = W f
M
The pipeline of our proposed method is shown in fig. 3 and is motivated by (Norrenbrock et al. 2022 2024). following their presentation and notation. It starts with training a conventional boo
model with initially ny features using the feature diversity loss Laiv (Norrenbrock et al. 2022), as high diversity of features is desired for interpretable models. A detailed explanation of Caiv is included in appendix M Using the black-box model as starting point, we aim to find a selection of n out of the initial ny features and their sparse binary assignment W to the classes to enable downstream interpretability. The feature extractor 4 is then fine-tuned with this solution fixed, so that the features adapt to the sparse solution and become a shared concept of the assigned classes. This is encouraged through selecting fewer features than there classes, n7 ne, and representing every class with the same number we: typically 5, of features. Using the same number of features for every class is beneficial for the interpretability in multiple ways. The class representations do not
4

---

Published as a conference paper at ICLR 2025
Rwe, while the composing features focus on detecting general concepts. Since we aim to optimize binary variables under constraints with a clear objective, we formulate it as discrete optimization problem to get the optimal solution. As indicated in fig. 2 we define the constants A, R. and of the resulting QP so that in the global optimum different (R)). localized (b) features selected and assigned to classes for which they have high predictive power (A). These fixed simple binary class representations then lead to the emergence of interpretable features during fine-tuning. How the quadratic problem with A, R and is formulated to ensure this goal is discussed in the following sections.
need bias and can be contrasted as St fl....,
........;.. n5} "
3.1 QUADRATIC PROBLEM
We consider the problem of selecting the nz out of n features and assigning them to the classes as a binary quadratic problem, that can be solved globally optimal. Specifically, the feature selection E {0. 1 and assignment between features and classes W E {0, 1)"">xX,/ are jointly optimized, with W being W for the selected features. Given similarity matrix A E R="'"y the main objective is to maximize the similarity Z A between the selected features and their assigned classes
The
Z = Σ aa o u.r)²
with o indicating the Hadamard product. Here, 8 indicates whether a feature is selected and W describes feature is assigned the class. Note that we use index classes and d for features. The sparsity and low-dimensionality are formulated as constraints for the optimization:
nij
d=1
n Σ
We,dSd = Twe Ve [I.....,ne]
To allow the QPM the differentiation between all classes and enable effective fine-tuning, we additionally add constraints that two classes are assigned to the same set of features:
(we we)) ) << Tww Vc, {l.....ncc} Note that the constraints in eqs. (3) and 4 technically define quadratically constrained quadratic program (QCQP). To make the QCQP computationally tractable, the constraints are relaxed and added iteratively for classes that violate the constraints. The efficient implementation is discussed in detail in section 4.1.I] The general formulation of the problem allows us to add further nuance to the optimization and include more desiderata. Since high representational capacity is desired for the selected features, the cross-feature similarity matrix R E 2 xny is incorporated to reduce the similarity between the selected features:
(4)
ZR - Rs Additionally, the selection of specific features can be guided via selection bias b g"
(5)
ZB = bTs. s, where a higher value b, leads to preferred selection of the feature i. The combination of all these objectives leads to:
(6)
max Z = max ZA + ZR + ZB W.a
W.
The formulation in standard form for quadratic problems 1 Qx + c²'x with Q capturing the
quadratic terms ZA and RR, and incorporating the linear term Z n is included in appendix O
3.2 CLASS-FEATURE SIMILARITY
The class-feature similarity matrix A with entries a.d should reflect how beneficial the assignment of feature d to class is for the classifier. As every feature gets assigned to multiple classes, which themselves become assigned to multiple features, the metric should focus on a robust positive relation between the activation and likelihood of sample being of the respective class. This is captured by the Pearson correlation coefficient ar between the feature distribution f.A and the label vector Ie E (0, 1)"r, in which for all ny training images 1 indicates the label being
5

---

Published as a conference paper at ICLR 2025
3.3 FEATURE- FEATURE SIMILARITY
Just maximizing eq. can lead to very similar features being selected which is neither beneficial for interpretability nor for accuracy as representational capacity is lost and multiple features develop towards the same concept during fine-tuning. To prevent this, selecting similar features in A should be penalized in the objective. We choose the cosine similarity between the class similarities of two
features d + d' in A for R with ra,d' ReLU |a.e|la..aa
using ReLU to focus on preventing
redundant features and Td,d' 0 for d = d'. As we are only interested in preventing the selection of highly similar features, we can clip all entries in R below an e to enable fast solving of the QP. The details are discussed in section 4.1.1
3.4
FEATURE-BIA
The Feature- Binturee-Bass b describes the benefit of selecting each feature. This can be used to steer the model towards specific desiderata. As diversity is generally preferred (Norrenbrock et al. [2022]
Alvarez Mclis & Jaakkola 2018) for interpretable models, a bias towards more local features is used,
ba
TVT Σg S3dd mar mar(S)){3a
(8)
Here Sy is the softmax over the spatial dimensions of the d-th feature map for the image j. Scaling the feature bias by their activation leads to the selection of features that are more localized when their activation is high. Alternatively, the bias can be used to steer the selection towards other criteria the practitioner might identify as relevant, which demonstrate in the appendix. We center b and scale the maximum absolute value to be X, whose strength defines the priority put on the bias.
4 EXPERIMENTS
Following prototype-based methods we applied our method to CUB-2011 (Wah et al. 201T and Stanford Cars (Krause et al., 2013). To showease QPM's broad applicability, we also include results on the large-scale dataset ImageNet- 1K (Russakovsky et al.] 2015). to which most interpretable methods are not applicable. Notably, CUB-2011 contains annotations of human concepts which we use to measure Structural Grounding. An overview of the used datasets is shown in Suppl. table As our method is independent of the used backbone, we evaluated it across various architectures, but focus on Resnet50 (He et al.] (0166) in this paper. Similar results on Resnet34, Inception-v3 (Szegedy al. 2016) and Swin Transformer (Liu et al. 202T), as well detailed results with standard deviations, are included in Suppl. appendix L We do not apply our method to other interpretable models like PIP-Ner (Nauta et al. (202)). as QPM an alternative way of inducing compactness and the features of PIP-Net are not general, thus ill-suited for broad assignment.
4.1 IMPLEMENTATION DETAILS
We generally followed PIP-Net for the data preparation. Specifically, the images are first cropped to the ground truth bounding box for CUB-201 and TravelingBirds (Koh et al.) 2020). For all datasets, the images are resized to 224 x 224. Following PIP-Net, TrivialAugment (Müller & Hutter 2021) is used and the strides of ResNets are also to to obtain more fine-grained feature maps. The remaining parameters, including dense training for 150 epochs on fine-grained datasets and directly using the pretrained model on ImageNet- IK with subsequent 40 epochs of fine-tuning, mirror the SLDD-Model and are described in appendix [ Note that QPM is trained more efficiently than and
Q-SENN, as does not use multiple training iterations during fine-tuning. We set nw n = 50 for QPM, unless stated otherwise. We demonstrate the impact of changing the parameters in the ablation studies but choose these, as it is in line with prior literature (Norrenbrock et al., 2024
2022), ni; < Th., and it enables sufficiently compact explanations (Miller 1956). The shown results, e.g. tables 2 and S are the mean across seeds, with the exception for ImageNet-IK, 1K, PIP-Ner and ProtoPool. For comparison, all models are exclusively pretrained on ImageNet- IK. This change did affect ProtoPool, but even with iNaturalist (Van Horn et al. 2018) pretraining, we could not reproduce the reported results by Rymarczyk al. (2022).

---

Published as a conference paper at ICLR 2025
4.1.1 QUADRATIC PROBLEM
This section presents details on how the described quadratic problem with eq. 77) as objective is solved using Gurobi (Gurobi Optimization, LLC, 2023) We incorporated deduplication and the assignment of an equal number of features to all classes eqs. (3) and (4) using an iterative approach with relaxed constraints. Specifically, the model is optimized without these constraints, but instead 1 W' 8 = fewelt. Then, after each iteration, all violated constraints are added to the model, but only limited to running set of features I E {0. 10.1 j which gets extended during the iteration. Next to the features, we also maintain a set of classes Cduplicates that were equal at one itcration and classes Cssarse that ever had too few features assigned. Instead of eqs. and the relaxed constraints
w Sr 2 Towe
VC Csparee
(9)
(we W < Towe
V., c' Cduplicates
(10)
are added, where W describes indexing We where r'= 1. Additionally, we set the start solution for the next optimization to a good, usually optimal. feasible solution for the currently selected set of features. As we need multiple iterations to enforce all constraints, we limit the time spent on one iteration to hours and set the gap to optimality to 10-4 In our experiments, the global optimum for the relaxed problem is usually found in less than 4 hours for fine- fine-grained datasets, and roughly 11 hours for ImageNet- 1K using a CPU like EPYC 72F3. While eq. (9) changes the desired optimization problem, the resulting objective is very close (achievable gap of less than 1 1%) to the global optimum, which is infeasible to compute and does not lead to an improved model. The experiments to verify this claim included in Suppl. appendix N Finally, alongside experiments, previous work (Hornakova et al. 2021) shows that the exact global optimum is not always preferred for relevant metrics. To make the relative weighting of the multiple objectives Za, ZR and ZB easier, A is scaled with and we to have a maximum of for n. 200 and Tww = 5. Since n features need be chosen, all entries below in R are set to O, where is the highest value, for which there still exists selection with Z R 0. This is equivalent finding the maximal for which the graph described by G with
0 if Ta,d' 26 else,
Id
(11)
has a maximum clique of size n/. We used approximations (Pattabiraman et al. 2015 Boppana & Halldórsson 1992) and sufficiently sized approximated maximum clique the start value for 8. Additionally, the remaining nonzero values R are scaled to have a maximum of 1. For scaling the bias b. clipped outliers, centered the remaining values around 0 and scaled the maximum absolute value to be X = VIO. which is empirically found.
4.2
METRICS
Bronzed Crowhird Fratures
Images
Broovee Cumbrdd
Shiny Cowbiid
Shiny Crowbird Features
Figure 4: Contrastive faithful class explanations for QPM trained on CUB-2011: Without any additional supervision, QPM learns to differentiate Shiny and Bronzed Cowbird (1/91 -0.97) using the red eye just like humans do, as the annotations in CUB-2011 or the screenshot in fig. 24 show.
Following PIP-Net as recent work, we evaluate the accuracy and compactness. measured as number of total features nž and number of features per class nlwe- Additionally, our QPM learns interpretable
7

---

Published as a conference paper at ICLR 2025
CMM Comporati 0 Componeni
DRN Companon.)
250
430
Companone
200
330
230
100
20
Feature Viice
Flature Waken
(a) Baseline Resnet50 with Contrastiveness = 41.8%
(b) QPM with Contrastiveness 99.99%
Figure 5: Extreme Examples for feature distributions and their Contrastiveness on CUB-2011.
class representations, summarized in table that are composed of features. As discussed in section 2 diversity, contrastiveness, generality and grounding are desired aspects of explanations. While we believe that our sparse binary assignment is very well suited for detailed analysis and alignment of the learned features, as it likely prohibits superposition, polysemantic neurons are still likely to occur and hard to measure for QPM and all end-to-end trained interpretable models. Therefore, measuring the grounding of features and instead focus on contrastive, general and diverse as desirable and quantifiable qualities of features as building blocks of our interpretable class representations, whose Structural Grounding we estimate using the attributes contained in CUB-2011. 1. Specifically, every class e in CUB-2011 is annotated with vector a, E [0, 1,312, where de.j indicates the fraction of images with label c, in which a human perceives the attribute j to be present. With these vectors, we compute the ground truth structural class similarity T с [0, 1 with v/2. being the cosine similarity between a and ae' Similarly, T Model E [_=,,]">>" "e is based on the class vectors in the interpretable classification layer. We then report the similarity for the top 25 most similar unique pairs of classes Csim reality
omit
StructuralGrounddng EedeCoo
odel Wee
(12)
Ee,' c,c'
Models with high Structural Grounding offer an interpretable human-like class-similarity, e.g. using the apparently different head to differentiate between Rottweiler and Doberman in fig. [or differenti- ating shiny and bronzed cowbird by its only separating attribute, shown in fig. 4
To measure the contrastiveness of features, a Gaussian mixture model with two components fit to every feature distribution f.d. resulting in the normal distributions A²d and N?1 visualized in fig. 5 We then compute the Contrastiveness as average of all features using the overlap (Inman & Bradley 1989) between the two distributions:
Contrastiveness
- Overlap(N (,g)
(13)
as bi-modal contrastive features can be represented by two non-overlapping distributions. The binary quality of the features is also indicated in figs. [ and 4 as the features are normed per column. Additionally, the features should capture a general concept, instead of a class-specífic one. This can be measured via the (laas--nddpenddece T:
- min 1 max Σ - f.a) n
(14)
ΣØ (f.. min f.a)
It measures which fraction of the zero-based feature activation across the entire dataset is not focussed on the most related class. A model with high Llass-nndeeeeee has features that recognize a shared concept for multiple classes, like the 4 central features in figs. [ and 4 Notably, opposed to Dependence (Norrenbrock et al. 2024), Class-Independeen can capture the assignment of multiple class detectors to the same class. For measuring the spatial diversity of the features, diversity@5 (Norrenbrock et al. 2022) has been proposed. The diversity 5 however suffers from the non-linear behavior of the softmax, resulting in
8

---

Published as a conference paper at ICLR 2025
Table Comparison on compactness and accuracy with Resnet50: QPM shows increased accuracy and compactness. The cenppactness-accuraaae trade-off is shown in fig. 7
Among more interpretable
models, the best result is marked in bold, second best underlined.
Method
Accuracy f
Total Features
4
Features Class
+
CUB
CARS
INET
CUB
CARS
INET
CUB
CARS
INET
Baseline Resnet50)
86.6
92.1
76.1
2048
2048
2048
2048
2048
2048
glm-sagas
78,0
86.8
58.0
809
807
1627
5
5
5
PIP-Net
82.0
86.5
731
669
12
11
ProtoPool
79.4
87.5
202
195
202
195
SLDD-Model
84.5
91.1
72.7
50
5
Q-SENN
84.7
91.5
74.3
50
50
50
5
5
5
QPM (Ours) 85.1
91.8
74.2
50
Table 3: Comparison on Interpretability metrics with Resnet50. Due required annotations, Struc- tural Grounding (abbreviated SG) can only be computed for CUB-2011.
Method
SID@S f
messs--ndeeeneecce t
Contrastiveness t
SG f
CUB
CARS
INET
CUB
CARS
INET
CUB
CARS
INET
CUB
Baseline Resnet50
57.7
54.4
37.1
98.0
97.8
99.4
74.4
75.1
71.6
34.0
glm-sagas
55.4
51.8
35.8
97.6
99.4
74.0
74.5
71.7
2.5
PIP-Net
99.1
99.0
75.6
62.9
99.5
99.5
6.7
ProtoPool
24.5
30.7
96.9
96.0
76.7
78.9
13.9
SLDD-Model
88.2
88.6
64.7
96.2
95.6
98.6
87.2
89.7
93.4
29.2
Q-SENN
93.3
94.4
82.0
95.5
94.8
98.7
93.0
94.2
92.6
23.4
QPM (Ours)
90.1
89.6
64.1
97.0
96.5
99.1
96.0
97.7
89.3
47.9
scale-dependency (table 8).
Therefore, we propose Scale Gecue-Invariant-iiiee Invvrrann--iversiyyeee (SID@5)
M
e A3
My
$
(15)
warh Σ
M
Em,n
Md
RM Disl
MM 1=
max
22 m
i..j
SID@S @5
(16)
5
where S refers to the result of softmax applied to the d-th highest weighted feature map M scaled by its absolute mean. A high SID @5 is visible figs. [and as the 5 features used for each class, localize on very different regions in the image.
4.3 RESULTS
This section discusses the experimental results. The usual metrics for compactness-basee globally interpretable models are shown in table For the fine-grained datasets, QPM among the most compact models while showing the highest accuracy, thus setting the state of the art for interpretable models. On ImageNet-IK. where prototype-basee methods are not even applicable, QPM is only marginally beaten by Q-SENN, which uses inteesiee iterations and negative reasoning for some classes, which significantly hinders interpretability. A runtime analysis is shown in appendix The results for the interpretability metrics shown in table 3 Note that glm-sagas and PIP-Net are hardly comparable, as glm-sagas uses the uninterpretable features of black-box model and PIP-Net learns very localized class-detectors, with some features activating single class. In contrast, QPM achieves excellent values across all metrics and datasets in this multicriterial task of self-explaining neural networks, summarized fig. 6 Its interpretable class representations. composed of diverse, general and contrastive features, mirror reality, as measured by Structural Grounding. Note that QPM learns grounded representations as shown in figs. [ and g without any additional supervision and is able to communicate the only differentiating factor it uses. QPM's local behavior then follows its faithful global explanations, which leads to trustworthy classifications and predictable errors when the differentiating factor is not present, as in fig. g The appendix contains more visualizations, including a discussion of failure cases in appendix EE] discussion polysemantic features (appendix H). an extension of Structural Grounding to ImageNet- IK (appendix M and a discussion limitations and future work (appendix|K)
99% on just

---

Published as a conference paper at ICLR 2025
ProtoPool - 0.93
SID@5
Baseline Resnet50 1.20
PIPNet 1.26
Accuracy
glms 1.47
SLDD-Model
1
Q-SENN 2.27
(67) Cac
Y1
QPM(Ours) 2.62
D-SENN (n) with 1
DPM Uned of
Class-ndeepndence
Q-SENN (rad with n2-55
S4
n7-50
122
1
Twe
Figure 7: Compactness-accuracy trade-off compared with Q-SENN (dashed) and SLDD-Modes (dotted) with Resnet50 on CUB-2011, With increasing compactness, QPM's op- timal usage of the set nt; and Towc becomes more beneficial.
Structural Grounding
Contrastiveness
Figure 6: Radar plot across all considered metries for CUB-
2011. Metrics which are preferred to be lower (ny, Towe) are encoded as and every value is given as fraction of the maximum. Values in legend are the area of each radar plot. Table 4: Impact of including (V) additional objectives
Za (eq. (6)) for locality and ZR (eq. (5) to reduce correlation alongside A in eq. (7) on CUB-2011 with Resnet50. Corre- lation is measured
the average maximally similar feature
according to cosine similarity, formulated appendix C.B
ZB
ZR
Accuracy f
SID@51
Correlation +
X
84.6
89.0
33.9
Figure 8: Misclassified example of a QPM, explained in fig. [ Pre- dictably given the explanation, the model classifies a Doberman as Rottweiler due the absent head.
V
84.4
90.3
33.5
X
/
85.0
88.5
22.7
/
85.1
89.6
24.6
4.4 ABLATION STUDIES
This section validates the impact of the individual objectives in the quadratic problem in table 4 and presents the compactness trade-off in fig. 7 We focus CUB-2011 but observed similar results for other datasets. The compactness-accuraat
tradeoff for QPM compared with Q-SENN and the SLDD-Modes is visualized in fig. The global optimization clearly leads to
effective use of the defined capacity, with the highest uplift in the very high compactness regime, e.g. 1.5 percent points at ; 20, where good selection and assignment naturally has more impact.
The impact of the ceature-feature similarity matrix R and feature selection bias b is shown in table 4 Incorporating a bias b for local feature maps further increases the SID@5. On the other hand, reducing feature similarity through R effectively reduces the correlation between the resulting features, which improves accuracy, as the model uses its capacity more effectively. In summary, the inclusion of the secondary objectives ZR and ZB beneficial for the resulting model, improving the desired aspects not just after solving the QP but also in the resulting model after fine-tuning.
The appendix contains further ablation studies support our claims, demonstrating the ability to steer (appendix [D). validating the choice of correlation as metric for A (appendix (1 and showing the benefits of enforcing exactly nww features per class (appendix G
CONCLUSION
In this paper, we introduced the Quadratic Programming Enhanced Model (QPM). It uses discrete optimization to find an optimal feature selection and assignment of just 5 to each class. With this casy-to-understand assignment, the resulting QPM is more interpretable than previous methods, as has contrastive faithfully interpretable class-representationss. shows Structural Grounding, is steerable, and its features have excellent SID@5, Class- Ilass-Indeeendeece and Contrastiveness. Additionally, further closes the accuracy gap to the drastically less robust uninterpretable baseline. Figure 6 shows that only QPM excels in all metrics, thus setting a new state of the art for compactness-aase interpretable models, while delivering unprecedented global interpretability even ImageNet-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the Federal Ministry of Education and Research (BMBF), Germany, under the AI service center KISSKI (grant no. 01IS22093C), the Deutsche Forschungsemeinsccatt (DFG) under Germany's Excellence Strategy within the Cluster of Excellence PhoenixD (EXC2122)) the European Union under grant agreement no. 101 136006 XTREME. The work has been done in
collaboration and partially funded by the Intel Corporation. This work was partially supported by the German Federal Ministry of the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (GreenAutoMl 4FAS project no. 67K132007A).
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
David Alvarez Melis and Tommi Jaakkola. Towards robust interpretability with self-explaining neural networks. Advances in neural information processing systems, 31, 2018
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: vrXivv1607.006 (6070064500 2016.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection:
Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541 16549, 2017.
Alexander Binder, Grégoire Montavon, Sehastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In Artificial Neural Networks and Machine Learning-CAANN 2016: 25th International Conference on Artificial Neural Networks, Barcelona, Spain, September 6-9, 2016, Proceedings, Part Il 25, pp. 63-71. Springer, 2016.
Moritz Böhle, Mario Fritz, and Bernt Schiele. Holistically explainable vision transformers. arXiv preprint arXiv:2301.0880 008669, 2023.
Ravi Boppana and Magnús M Halldórsson. Approximating maximum independent sets by excluding subgraphs, BIT Numerical Mathematics, 32(2):180-196, 1992.
Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances in neural information processing systems, 32, 2019.
Giannis Daras and Alexandros G Dimakis. Discovering the hidden vocabulary of dalle-2. arXiv preprint Xiiv:22060001699, 2022.
Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hadielld-Dddss, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint ar arXiv:2209. 20652 2022.
Thomas Fel, Agustin Picard, Louis Béthune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi
Cadène, and Thomas Serre. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2711-2721, June 2023.
Patrick Glandorf*, Timo Kaiser* Bodo Rosenhahn, and (*contributed equally). Hypersparse neural networks: Shifting exploration to exploitation through adaptive regularization. In International Conference on Computer Vision Workshops (ICCVW), October 2023.
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2023. URL https:: //ww /www gurobi com
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it?
shortcomings of latent space prototype interpretability deep networks, 2021.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
Andrea Hornakova, Timo Kaiser, Paul Swoboda. Michal Rolinek, Bodo Rosenhahn, and Roberto Henschel. Making higher order mot scalable: An efficient approximate solver for lifted disjoint paths. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6330- 6340, 2021.
Henry F. Inman and Edwin L. Bradley. The overlapping coefficient as a measure of agreement between probability distributions and point estimation of the overlap of two normal densities. Communications in Statistics Theory and Methods, 18100)3851-3374, 1989. doi: 10.0/ 03610998908833127.
Neha Kalibhat, Shweta Bhardwaj. C. Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi Identifying interpretable subspaces in image representations. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15623-15638. PMLR, 23-29 Jul 2023. URL httpt:::pppoceeee//eeeoo mlr. wwwpsppessp.... 2a2ess/v222//aaiiattz222
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: evaluating the human interpretability of visual explanations. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pp. 280-298. Springer, 2022.
Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-2800 Springer, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et Segment anything. arXiv preprint arXiv:2304.02663, 2023.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp- 5338-5348. PMLR, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Peter Lipton. Contrastive explanation. Royal Institute of Philosophy Supplements, 27:247-266. 1990.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022. 2021.
Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proot concept- based models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL hctps:// oppenreview.o hww.nrevieeenee//ooumiiiieeee (ttee//foorummiddda771993555551
12

---

Published as a conference paper at ICLR 2025
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47)::2206221111 2022.
George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):11, 1956.
Tim Miller. Explanation in artificial intelligence: Insights from the social sciences. Artificial intelligence, 267:1-38, 2019.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Samuel G. Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-art data augmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 774-782, October 2021.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine-grained image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14933-14943, 2021.
Meike Nauta, Jörg Schlötterer, Maurice van Keulen, and Christin Seifert. Pip-net: Patch-based intuitive prototypes for interpretable image classification. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Thomas Norrenbrock, Marco Rudolph, and Bodo Rosenhahn. Take 5: tion with a handful of features. In Progress and Challenges in Building Trustworthy Embodied AI,
Interpretable image classifica-
2022.
Thomas Norrenbrock, Marco Rudolph, and Bodo Rosenhahn. Q-semn: Quantized self-explaining neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 21482-21491, 2024.
Curtis G. Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. In Proceedings of the 35th Conference on Neural Information Processing Systems Truck on Datasets and Benchmarks, December 2021.
Tuomas Oikarinen and Tsui- Wei Weng. CLIP-dissecd Automatic description of neuron repre- sentations in deep vision networks. In The Eleventh International Conference on Learning Representutions, 2023. URL utthttps: wwtp//peeeeeeeeeeoooeeeee net/
Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXi::2304.06/29 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL BRIttppp/ppapessseeerrittt www..pp.ccp.pp.p.p........... 9015
ibrary.
pdf
Bharath Pattabiraman, Md Mostofa Ali Patwary, Assefaw Gebremedhin, Wei-keng Liao, and Alok Choudhary. Fast algorithms for the maximum clique problem on massive graphs with applications to overlapping community detection. Internet Mathematics, 11(4-5):421 1448, 2015.
Vitali Petsiuk, Abir Das, and Kate Saenko. Rise; Randomized input sampling for explanation of black-box models. In Proceedings of the British Machine Vision Conference (BMVC), 2018.
13

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.019522 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06/25 2022.
Stephen J Read and Amy Marcus-Newhall. Explanatory coherence in social explanations: A parallel distributed processing account. Journal of Personality and Social Psychology, 65(3):429, 1993.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Bodo Rosenhahn. Optimization of sparsity-constrained neural networks as mixed integer linear program: Nn2milp. Journal of Optimization Theory and Applications, 199(3):311-54, 2023,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision (IJCV), 115 13)2211-252, 2015. doi: 10.1007/s1 126000100-22--
Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zielinski. Protopshare: Prototypical parts sharing for similarity discovery interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1420-14300 2021.
Dawid Rymarczyk, Lukasz Struski, Michal Górszezak, Koryna Lewandowska, Jacek Tabor, and Bartosz Zieliński. Interpretable image classification with differentiable prototypes assignment. In European Conference on Computer Vision, pp. 351-368. Springer, 2022.
Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10:41758 41765, 2022.
Adam Scherlis, Kshitij Sachan, Adam S Jermyn, Joe Benton, and Buck Shlegeris. Polysemanticity and capacity in neural networks. arXiv preprint arXiv:2210.01892, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localiza- tion. International journal of computer vision, 128:336-359, 2020.
Brian Sullivan. Shiny and bronzed cowbird comparison, 2024. URL https: //w/ /www a illaboutbirds htt.://t...dd..ooooo//.///////// http://wwww...iaoooooooooo/////a 2024-09-27.
compare/ Accessed:
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pp. 3319-3328. PMLR, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 121182226 2016.
Adly Templeton. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. Anthropic, 2024.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769-8778, 2018.
14

---

Published as a conference paper at ICLR 2025
Michael Veale and Frederik Zuiderveen Borgesius. Demystifying the draft artificial intelligence act act-anaae antlysinng the good, the bad, and the unclear elements of the proposed approach. Computer Law Review International, 22(4))97-112, 2021.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning. pp. 17205-11216, PMLR, 2021.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PStR*2ttruct: Privacy, Accountabiliry, Interpretabiliry, Robustness, Reasoning Structured Data, 2022. / net weeee/oorrmmlddHHMMo.
.g
Mateo Espinosa Zarlenga, Pictro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini. Michelangelo Diligenti, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, et al. Concept embedding models. In NeurlPS 2022-36th Conference on Neural Information Processing Systems, 2022.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
15

---

Published as a conference paper at ICLR 2025
Table 5: Statistical overview of datasets. TravelingBirds is used exclusively in the appendix.
Dataset
CUB-2011
Stanford Cars
TravelingBirds
ImageNet-IK
Classes ne
200
196
200
1000
# Training
5994
8 144
5994
1881 167
# Testing
5774
8 8041
5774
50000
A APPENDIX
This appendix contains additional details of the implementation details, more results with standard deviations, further experiments steerability, discussion of failure cases and the formulation of the Feature Diversity Loss Ldiv. Further, the quadratic problem is presented in its standard form and the optimality of the found solutions is discussed.
B TRAVELINGBIRDS
We use Traveling as an additional dataset validate our method and for our steering experiments in appendix D It is based CUB-2011 and designed to allow the measurement of the robustness of spurious correlations. Specifically, the background of every class in the training set is replaced with an image of constant class of Places365 (Zhou et al. 2017). For the test set, backgrounds of random classes are used, thus measuring if the model learned to rely on the spuriously correlated background.
C IMPLEMENTATION DETAILS
We first describe the implementation on the fine-grained datasets CUB-2011, Traveling and Stanford Cars. All deviating details for ImageNet-IK: are included in appendix C.A. The implemen- tation details are similar to the SLDD-Model (Norrenbrock et al. (022), but use the default data for prototype-basee based methods. Specifically, we use the exact same dense model for both models in our experiments and only alter the following parts of the pipeline with the same hyperparameters
for fine-tuning. Therefore, the improved metrics can be attributed to the superior selection and assignment.
Architectures We implement our method using PyTorch (Paszke et al.) 2019) and its ImageNet- IK pretrained models as feature extractors. For Resnet50 and Resnet34 follow PIP-Net and use smaller stride size of for the two last blocks.
Data For training with CUB-2011 and TravelingBirds, the images are first cropped to the ground truth segmentation, following prototype-base babded methods Nauta et al. (2023); Rymarczyk et al. (202). After cropping, they are resized to 224 x 224 (299 x 299 for Inception-v3). For Stanford Cars and steerability experiments in table 11 random crop after resizing
side to the target image size is used instead. Then normalization, random horizontal flip, jitter and TrivialAugment Müller Hutter (202T) applied. At test time, augmentation used and only cropping, random crop replaced by center crop, resizing and normalization is maintained.
Dense Training We fine-tune the pretrained models on the fine- aie--grained datasets using stochastic gradient descent with batch size of 16 for 150 epochs. The learning rate starts 5 10-3 for the pretrained layers and 0.01 for the final linear layer and gets multiplied by 0.4 every 30 epochs. We momentum to 0.9, Ez-regularization to 5 10-4 and apply dropout with rate 0.2 the features. The weighting 3. included in eq. 33) of the Feature Diversity Loss Norrenbrock et al. (2022) is set 0.196 for the Resnets, 0.049 for Inception-v3 and 0.0245, the highest value we tried for which all dense models converged, for Swin Transformers. Note that the values are scaled with the number of patches in the feature maps, leading to numerical values that do not align conveniently with powers of 10.
Fine-tuning After solving the quadratic problem, the model is trained with the final layer fixed to the sparse assignment of selected features W for 40 epochs. The learning rate starts 100 times the

---

Published as a conference paper at ICLR 2025
final learning rate of the dense training and decreases by 60 % every 10 epochs. During fine-tuning momentum is increased to 0.95 and dropout on the features reduced to 10%. For Swin Transformers, the batch size is set to and Layer normalization Ba al. (2016) is turned off after the dense training has finished, ensuring more unrelated features. All other parameters equal the dense setting.
As the feature maps the result of ReLU Nair& & Hinton (2010), might expect its values to be strictly 2 0. However, just like for the SLDD-Model, the features of QPM are normalized with a fixed mean and standard deviation before fine-tuning begins, resulting in the sub-zero min (frz).
Reproducibility For reproducibility, all our experiments with 5 seeds use the integers 16 to 20, ending at 18 for the 3 ImageNet-IK runs, as seed for all random processes.
Scaling the Objective To keep a similar relative weighting across changing Twe and nee we also scale the main objective for the quadratic problem Zд with them
1000 ZA
Z1
(17)
Tbo
ensuring no additional scaling for ne = 200 and Towe 5.
Choice Pretrained Weights We use the pretrained Resnet50 weights V/ of PyTorch for our experiments, as the default V2 has very class-specific features already, with a class-nndependence of 92.6%. For V2, a sparse model computed by glm-saga [Wong al. (02TT) with just features per class can already achieve 66% accuracy ImageNet-1K, demonstrating the class-specificness of its features. For Resnet34 and Inception-v3, we use the only available set of weights from Py PoTochh For Swin Transformers, we used the original provided weights of Py Pyoorh, they are suitable for the used image resolution.
C.A IMAGENET-IK
Due to computational constraints, we follow the SLDD-Modes skip the dense training on ImageNet- 1K and directly use the pretrained model as dense model. To facilitate the comparability of metrics between the dense model and our experiments, we use the default strides. For augmentation, we use Lighting noise and omit TrivialAugment. Finally, the learning rate of the fine-tuning starts at 100 the value used for the fine-grained datasets account for the increased size of the dataset.
I of
C.B CORRELATION METRIC
For measuring the effect of reducing correlation between selected features in table the Correlation is used:
1
n;
Stafraa frallffaa
Correlation
max dedd'
(18)
n2 Aml
C.C QUADRATIC PROBLEM
This section presents further details on the quadratic problem and the start solution WSStart for the next iteration solving the quadratic problem with updated constraints. The start solution is a good, usually optimal, feasible solution for the currently selected set of features A. To simplify the initial iterations, only eq. is considered. The deduplication of eq. TOPp is only included after a solution is found that satisfies eq. (9.. The start solution is constructed from n which contains nw assignments for each class to the most similar features in A. If the equal distribution of assignments per class still exclusively optimized for, Startt = W is already the start solution. Else, take care of all classes with equal assignment Coqual in Wme Specifically, remove all
2

---

Published as a conference paper at ICLR 2025
69
TravelingBirds Crop
91.2
68
91.0
Accuracy [%)
90.8
90.6
66
90.4
90.2
65
90.0
CUB-2011
89.8
10
10
10
10?
Scaling Factor
Figure 9: Steerability of the proposed QPM: When increasing the weighting of the bias X, the desired metrics, accuracy or $ID@5 improve.
duplicate pairs (c, c') E Cequal:
if
(c, c' Cequal &
(c, d) Maxi(c, c'),
Deduplication
if
(c, d) E Coqual
(19)
c.d
(c, d) Mini(c,
else
Start
W + W Deduplication
(20)
Here, Mini(c,e') returns the indices remove, that is of the current assignment with lowest similarity:
h (a. o =. + max(a, w="")) (1 s)
(21)
Mini(c, d') c, argmin(h") c' argmin(h")) else
min(h s min(h")
(22)
Here, is the selection vector and ensures that all changes only apply the selected features. Similarly, Maxi(c, e') returns the indices the assignment to add, which has the highest similarity of the the not currently assigned features:
h = (a. o cand (c)) o
(23)
c. argmax(h")) if max(h" 2 max(h=))
Maxi (c. d)
(24)
c' argmax(h" else
The candidate function cand(c) (c) (1-w w":)) wbnd(c, w"*))
(25)
checks that the assignment is not made yet and the would-be-no-dun -odullilee function wbnd(c, W E 10,1} further ensures that the addition of the assignment of class c to fea- ture d would introduce duplicate, returning in that case. While this technically does not guarantee an optimal solution, first only finding the solution with nwe assignments per class and then deduplicating ensures that the number of duplicates quite low already, which usually leads to finding the optimal feasible start solution.
D STEERABILITY
This section concerned with the ability the practitioner to steer the model towards desired biases using the feature bias b. For example, if human recognizes the erroneous focus the background

---

Published as a conference paper at ICLR 2025
of trained QPM, enabled through global interpretabilityy, the feature bias Center (eq. 26 can be used to steer the model towards more centered features.
by Center
R& وو de dM() Sid
(26)
where de computes the distance between the maximum of the j-th sample's map Ma (2, y) and the closest edge:
de (M min([æ warl 1. ly har 1.
(27)
The resulting improved accuracy on TravelingBirds with À 10 shown in table demonstrates this steerability. Setting À allows precise weighting of the emphasis put the bias. This direct control for both the center and diversity bias is visualized in fig. and allows the incorporation of any feature-level bias b.
E FAILURE CASES
This section presents examples where QPM predicts wrongly. For that, fig. 10 shows exemplary images of Rottweiler and Doberman with classification results of the probed QPM trained on ImageNet-IK and with global explanations figs. and and 21 to 23 Note that the accuracy across
the two classes is 87%, well above the average, reflected in correct classifications across poses. backgrounds and settings in figs. and Tob Additionally, fig. IT shows the GradCAM (Selvaraju et al., 2020) visualizations and demonstrates that QPM always focuses on the dog in the image. For the erroneous predictions, the model behaves just like the global explanations would indicate. Rottweiler and Doberman may be swapped, if the head is occluded in figs. Toc and Tog in a difficult pose to gauge the shape, shown in figs. [od and TOh Since the Black and tan coon hound is assigned both head features of Rottweiler and Doberman, they can also be confused when primarily the head is visible, demonstrated in figs. 10e and TOi Finally, figs. 10f and [0] seem to contain of the many (Northcutt et al. 2021) wrongly labeled samples in ImageNet-TK. QPM also robustly classifies wrongly labeled data, as the global explanation would suggest. Figures 12 and 13 show the feature activations of Greater Swiss Mountain Dog and Rottweiler on fig. [oi and other class examples, further suggesting that it is indeed typical Greater Swiss Mountain rather Rottweiler for the probed QPM, as the features of the former localize on the expected regions, whereas most Rottweiler features barely activate. Finally, fig. [4] shows further test examples for the model explained in fig. 4 and demonstrates that the model does not predict Bronzed Cowbird the differentiating red eye is not present in the image. In summary, QPM's 's local behavior robustly follows the faithful global explanations, which can lead to predictable faulty classifications case of occlusion or difficult pose.
F RUNTIME ANALYSIS
This section discusses the time it takes to obtain a QPM, compares to competing models and discusses the impact of n on it. Figure 15 demonstrates that the optimization time strongly increases when increasing n However, for the probed datasets, going beyond 50 features seems not to be necessary, as the accuracy only improves negligibly, while the interpretability harmed: Features become less general and there will be fewer class representations with high overlap, which allow for the most intuitive interpretation. One can further optimize this using suitable priors, which we do not include in this work, as the interpretability and additional accuracy decreases with increasing n", It is however an avenue for future work, when datasets with sufficient complexity are published. Table compares the time to obtain the interpretable model between QPM, Q-SENN and SLDD-Model. Q- Q-SENN and SLDD-Model start with a feature selection, that takes 15 minutes on CUB-2011 I and roughly 500 minutes ImageNet- IK. They both use glm-saga for feature selection and computing the sparse matrix and are thus scaling with number of samples ny, which QPM invariant to, as that dimension is summarized in the constants.
4

---

Published as a conference paper at ICLR 2025
(a) Correctly Classified Doberman Examples
(b) Correctly Classified Rottweiler Examples
(c) Wrongly classified as (d) Wrongly classified as (c) Wrongly classified Rottweiler
(f) Classified as Rottweiler
Rottweiler
Black tan coonhound
(g) Wrongly classified as (h) Wrongly classified Doberman
Wrongly classified (j) Classified as Greater Black & tan coon hound Swiss Mountain dog
Doberman
Figure 10: Examples for correctly and wrongly (according to ground truth labels) classified examples of the QPM with global explanations shown in figs. and 21 show Doberman labeled images. Figures 10a and T0g to 10 roww 2 and 4) display Rottweiler labeled images. The resulting classifications match the expected behavior based on the global explanations. As the explained QPM uses the head to differentiate between Doberman and Rottweiler (fig. T they can be confused when is occluded (figs. [0c and [Oc) or in difficult pose (figs. [Od]andd and 10h As the black and tan coon hound is assigned the same head features (fig, 23). they get confused, il only
23 Figures Figures[ObJtor [Or (rows 1 and 3)
the head is visible (figs. 10e and TOi). Finally, the probed QPM correctly classifies according to its explanations (figs. 10a and 10b also wrongly labeled samples (figs. [Of 10j..
5

---

Published as a conference paper at ICLR 2025
(a) Correctly Classified Doberman Examples
(b) Correctly Classified Rottweiler Examples
(c) Wrongly classified as (d) Wrongly classified as (c) Wrongly classified Rottweiler
(f) Classified as Rottweiler
Rottweiler
Black tan coonhound
Wrongly classified as (h) Wrongly classified Doberman
Wrongly classified
(j) Classified as Greater Swiss Mountain dog
Doberman
Black & tan coon hound
Figure 11: Gradcam Visuliazations for fig. 10
G IMPACT OF EVEN SPARSITY
This section discusses the impact of enforcing exactly Thee features per class, rather than on average. For that, we trained a model without this constraint, but instead with 1 TWs. nw enforcing an average sparsity. To counteract the uneven number features per class, every class got a bias, that is linear to the number of features it is below the average. In prior experiments, various forms of counteracting the uneven assignment with a bias have performed similarly. Table Table shows that the assignment is beneficial for the accuracy. Further, the assignment boosts interpretability leads to more classes that can be contrasted easily and does not introduce an unintuitive bias term. Additionally, fig. 16 demonstrates that classes, which are assigned fewer features, cause these

---

Published as a conference paper at ICLR 2025
Exampie Images
Frature
Feature 2
Feature
Feature
Feature
Rottweller
Greate Swiss Mountain dog
Figure 12: Features of Greater Swiss Mountain Dog and their localization the sample in fig. 10j (first row), that is presumably falsely labeled Rottweiler. The 4 lower rows contain examples of Greater Swiss Mountain Dog and the features consistently localize around semantically similar regions, also on the Rottweiler labeled one.
Table 6: Time in minutes between finishing dense training and obtaining the final model. First value is time for optimization. second time spent fine-tuning. Following Q-SENN, we use the fast setting for ImageNet-IK. Q-SENN trains for 70 epochs, instead of 40, in total during fine-tuning and does iterations of glm-saga. Note that every method runs exclusively on GPU server, except for the QP optimization, which can be done on just CPU.
Method
CUB
INET
SLDD-Model
(15 + 22) + 78 115
(500 + 3000) + 3600 = 7100
Q-SENN
(15 + 4 22) 78 7/4 Rz 240
(500 4 100) + 7/4 3600 7200
QPM (Ours)
210 + 78 298
660 + 3600 4260
features to become less general for QPM and Q-SENN, which hurts interpretability and potentially accuracy. Figure [7 also visualizes that Q-SENN always learns to represent classes with a huge variety in the number of assigned features, necessarily leading to hardly interpretable representations. Nevertheless, the impact is disparate on the two datasets and the accuracy increase is not significant on Stanford Cars. Future work might investigate if datasets with classes of varying complexity will benefit from representing classes with a suitable number of features and how this can be combined with contrastive globally interpretable class representations.
7

---

Published as a conference paper at ICLR 2025
Example Images
Feature
Feature 2
Feature
Feature
Feature
Rottweller
Rottweller
Rottweller
Rottweller
Rottweller
Figure 13: Features of Rottweiler and their localization, scaled by column, the sample in fig. 10 (first row), that is presumably falsely labeled Rottweiler. The 4 lower rows contain examples of Rottweiler. The color code is consistent with fig. and features and are shared with Greater Swiss Mountain Dog.
Table Accuracy with or without exactly thw features per class (eq. Instead, on average Thwc features class are used.
Method
CUB
CARS
without eq. (3)
0.2 85.1++0.3
91.6++0.3
QPM (Ours)
91.8+
H POLYSEMANTIC FEATURES
This section discusses the phenomenon of polysemantic features and how it relates QPM. Like all deep learning models (Scherlis et al. 2022) not specifically designed to prevent polysemanticity. QPM learns polysemantic features. refers individual neurons activating on not just one concept but rather on seemingly unrelated ones. While it is an active area of research, their emergence can likely be attributed to being an effective solution to the training objective. many training samples, the impact on the loss can be fairly low, if polysemantic feature activates on any of its n meanings. The only exception occurs, when it activates on samples, where its activation contributes significantly to class that is already showing lot of activation. While this is typically very difficult to analyze, the interpretable structure of QPM can offer more insights, as it enables a reliable metric on which to gauge how strongly the activation another concept would affect the loss: The similarity in QPM's class representation space. Our hypothesis is that QPM learns features that are locally monosemantic, while being globally polysemantic. Around class, e.g., Bronzed Cowbird, expect the features
8

---

Published as a conference paper at ICLR 2025
Example Images
Frature
Feature 2
Feature
Feature
Feature
Brorzed Coubird
Bronzed Caweird
Shiny Cowbing
American Crow
Figure 14: Features of Bronzed Cowbird, explained and compared with Shiny Cowbird in fig. 4 the predictions of the QPM, and their localizations, normed across column, on Bronzed Cowbird labeled test samples. When all features, including the red eye (feature 1) are visible (rows and 2), the model correct. However, as expected from the global explanation, without the red eye can be wrong and confuse e.g. Shiny Cowbird with it. The probed QPM represents American Crow with features 2,4.5 and further not shown features, that localize on wing and beak of crows.
20
15
10-
Time
20
40
50
70
50
100
Number
Figure 15: Time it takes optimize QP for models with varying nnin fig.
to only activate on one of the n concepts that they activate on across the entire dataset. As this is generally fairly difficult to measure, show anecdotal evidence for this in fig. [8] It shows the Feature Aligmment metric from D..EEN(NNorenbreg Bronzed Cowbird, measured the number of its features that classes do share. Specifically, given the training features F E Raw R
et al. (2024) relative to the similarity to the
1st
ptrain iJ
F
ptrain ij
(28)
a.j
Patt
Pa-
tEpat
tEpa-

---

Published as a conference paper at ICLR 2025
Average Class-Indepen Dass-IndepeedencC of Assigned Features
97.0
96.5
96.0
95.5
Q-SENN
QPM without Eq. 3
94.5
QPM
10
20
Nonzero Weights per
Figure 16: Average Class- Class-Independence of Assigned Features on CUB-2011 as function of the number of features assigned to the class. The distribution of the sparsity is shown in fig. 17
10a
Q-SENN
QPM without Eq. 3
QPM (Ours)
10-1
Density
10
10-3
5
10
15
20
25
Nonzero Weights per Class
Figure 17: Distribution of Nonzero Weights per Class on CUB-2011. For each method, all 1000 classes from 5 seeds shown.
describes the average difference in activations when an annotated attribute is present, encoded in Pa+ or Pa- for absent. Norrenbrock al. (2024) then scales the difference by the average zero based activation and reports the average maximum per feature:
ptrain miny Etrain max max.44t .5t
nT
(29)
D1=1
For our analysis, we limit these formulas to just the attribute red eye color red - eye and only consider the one feature & detecting it for Bronzed Cowbird:
Tred (x)
RT
Ast re egg. (x))
(30)
LK
min, I,k
The x-axis additionally describes a filtering applied to the features and attributes based on the similarity of the label, where sample is considered for computing A & (x) if the annotated label shares at least 5 - x features with Bronzed Cowbird. Figure [88 demonstratss that the feature clearly detecting the red eye of the Bronzed Cowbird is indeed quite sensitive its presence when the ground truth label is similar to the class, while it globally loses that sensitivity as it also detects other concepts of classes further away from Bronzed Cowbird.
I STRUCTURAL GROUNDING ON IMAGENET
This section is concerned with evaluating a metric similar Structural Grounding on ImageNet- IK. It is based comparing the class similarities in reality Tee with the & Model ones learned by
10

---

Published as a conference paper at ICLR 2025
4.0
3.5
Aignment with rd eye color
3.0
2.5
2.0
1.5
1.0
0.5
0.0
Label Distence Bronzed Cowbird
2024 for the red-eye feature (marked in red) in fig. 4 and the attribute red eye color Fred-ege (x)) The x x-axis describes to which samples the computation limited, e.g. x describes computing the metric only on samples whose label is represented using up to 2 other features. On the right, = 5 refers to the usual global feature alignment. The probed QPM learned polysemantie, but locally monosemantic feature. When differentiating between bronzed and shiny cowbird only (æ 1), the feature value increased by almost 4 times its mean, the attribute annotated to be present.
Figure 18: Feature Alignment metric from Q-SENN Norrenbrock
model. Structural Grounding relies on the annotations of CUB-2011 to compute T However, there such annotations a fine-gres fininne scale for ImageNet- IK. Therefore, use the similarity of the text-names in CLIP (Radford et al. (021) as proxy to obtain our ground-truth class similarities. Specifically, we compute the cosine similarity Te E [-1, 1, 1e=eX between the text embeddings of the class names, obtained from the powerful pretrained ViZ-L-14, that is broadly used, e.g. to condition Stable Diffusion PoPPoeel et al. (2023). We always take the first description given for every class.
When inspecting the most similar classes, several issues are apparent. Many of them include shared tokens or words, e.g., ski and ski mask, lion and sea lion, rule and stole or digital clock and wall clock. While some of these indeed describe a similar class, e.g., giant Schmauzer and Standard Schnauzer, others do not. Including classes without high similarity as ground-truth similar classes harms the quality of the evaluation drastically and demonstrates the value of having human annotations. Another issue in the clip similarities is that fine -eeeggaaied knowledge about the classes seems to be less dominant than literal exact matching, as with higher similarity only more commonly used terms are correctly associated with similar terms, e.g., Orangutan and Gorilla, but not Rottweiler and Doberman. Therefore, the number of similar classes consider is set 1250, as the latter pair ranks at position 828 and we definitely consider it as a similarity worth measuring. Notably, this pair is ranked behind pairs such hog and lank, lemon and yawl hamster and snail, further demonstrating the weakness of the language model to exactly model the similarities. The final apparent issue lies in the ambiguity of class names which leads crane appearing twice as class name, once referring to birds, once to a machine on a construction site. Notably, the distribution of class sparsity has significant impact. While QPM is limited at least one feature with all features sharing the same weight and every class being represented by Tw features, SLDD-Modes S-NNN and glm-sagas all exhibit multiple (14, and 5) class pairs, that have class similarity of above 99%. The SI.DD-Mode! for instance repeatedly represents classes with one feature with positive weight and one with an extremely low negative weight, resulting even in cosine similarities of due to floating point precision. While this generally hurts interpretability. it can be beneficial for Structural Grounding.
class similarity of up 80%, due deviating in
Despite these issues, Table 9 shows that QPM still performs comparatively to SLDD-Model and Q-SENN with their extremely high similarities and learns significantly more aligned representations than the dense bascline. even ImageNet- 1 K. Future work might incorporate more fine- fine-graine grained class hierarchy, building upon the very general WordNet, into this metric or profit off of further improved language models.
11

---

Published as a conference paper at ICLR 2025
Table Results for diversity @5 (Norrenbrock al., 2022)
demonstrating its weakness capture
the locality of the by-design very local features of PIP-Net. Note that 20 is the worst possible value.
Method
CUB
CARS
Baseline Resnet50
61.1
57.4
glm-sagas
55.3:10.5
52.6110.3
PIP-Net
20.5+0.0
20.5:10.0
ProtoPool
25.5:10.4
23.4
SLDD-Model
79.2=0.3
81.910.9
Q-SENN
87.010.5
89.6+0.3
QPM (Ours)
89.9
91.410..
Table 9: Structural Grounding based Clip Similarities ImageNet
Method
Structural Grounding
Dense Resnet50
17.9=0.0
glm-sagas
10.3++0.0
SI.DD-Model
36.9+
Q-SENN
332
QPM (Ours)
34.5++0.6
J IMPACT OF CLASS-FEATURE SIMILARITY METRIC
This section contains ablation study the choice of Pearson correlation metric for the feature- class similarity matrix A. While it captures the desired linear relationship, that is also utilized during the following predictions, intuitive alternative is the Area under the receiver operating characteristic curve (AUROC), which is highly non-linear and frequently used to capture the predictive power with varying threshold. Table shows that AUROC also suitable but inferior the simple correlation.
K LIMITATIONS AND FUTURE WORK
This section discusses limitations for the proposed QPM and avenues for future work.
In this work, QPM is applied to the generally available and typical datasets for image classification, with ImageNet- IK indicating broad applicability. However, QPM's high interpretability is especially beneficial for high-stakes applications such the medical domain or autonomous driving, where each individual situation can not be accessed by an expert. Rather, after training the QPM and before be obtained gain insights into whether it is right
deploying it cars, its class explanations for the right reasons and if these are robust to all deployment conditions. Thus, applying QPM to suitable high-stakes applications is promising avenue for future work. However, to our knowledge. there is no suitable dataset from these domains published yet.
limitation of our QPM in its current form lies in its inability model negative assignments. Compared to the OLdD-Model and Q-SENN, which use negative weights, it is evident that the varied datasets used in this paper, not require it. Further, while we believe that it is generally prefer- able to represent classes only using positive assignments, e.g., also done by recent prototypical models (Nauta et al.) 2023 Rymarczyk et al., 2022), one can think of other datasets where negative reasoning may be superior. If, c.g., all classes in a dataset containing birds had a black beak, except for one with all other colors, it would likely be the most efficient solution to represent that one with feature activating positively assigned to it, which the current QPM might do. Thus, future work may incorporate assignments into the optimization, which might lead to even more compact representations.
negative assignment
black beaks, rather than have every other class
negative
As discussed in appendix H the learned features of our QPM are generally polysemantic, while potentially being monosemantic locally. For aligning them with human concepts, all post-hoc
12

---

Published as a conference paper at ICLR 2025
Table 10: Accuracy with different criteria used Feature-Clast Class Similarity matrix A
A Metric
CUB
CARS
AUROC
84.8::.2 85.1+1..3
91.6:1.22
Correlation (Ours)
91.810.1
methods, such as TCAV (Kim et al. 2018), Clip-Dissect (Oikarinen & Weng 2023), or the alignment methods from SLDD-Model or Q-SENN concepts is more beneficial for QPM than it is for e.g., black-box models, as they are used in an intuitively interpretable way. Further, the interpretable assignment can even help with alignment, as shown in appendix H. Nevertheless, polysemantic features are a challenge for interpretability and future work in this direction can focus on preventing their emergence while still using them in an interpretable way or robustly measuring alignment to multiple concepts.
be applied. Notably, aligning feature with their human
For many explanations from our QPM, saliency map for its individual features is used. While we typically just visualize each individual feature map via upscaling, resulting in a comparable resolution to GradCAM (Selvaraju et al. 2020), other saliency methods, like Integrated Gradients (Sundararajan et al.
2017). LRP (Binder et al.) 2016) RISE (Petsiuk et al. 2018) can be applied. Because QPM is backbone independent, even models with built-in more faithful saliency maps such as B-cos Networks (Böhle al.) 2023) can be used. Since the use of these features is easy-to-interpred evaluating the localizations of our model should focus on the feature explanations rather than class- level ones. Future work might incorporate these faithful saliency maps to measure insertion or deletion methods, akin to those used for class-level saliency maps (Petsiuk et al. (2018). Ideally, one able overcome the issue of moving dusti--dittiibution with removing pixels (Hooker al.) (209)). Finally, the contrastive nature of QPM's features might lead to an intuitive threshold that can be used during the removal of pixels, similar how previous metries try change the class prediction.
L DETAILED RESULTS
This section contains detailed results with standard deviations, including experiments with Resnet34, Inception-v3, Swin-Trane Transformer-small and Swin-Tran Transformer-tiny. Suppl. table table 22 The good results across architectures demonstrate an independence between backbone and our proposed method. They further seem robust as the difference in mean is usually large compared to the standard deviation. Further, figs. 21 and 22 show how the features of fig. T continue to localize the same human attribute across different poses. Additionally, we included the activations of these features on images of another class in fig. 23 showcase the global interpretability enabled through the binary assignment of more interpretable features. Instead of the blue and green feature, this probed QPM recognizes the Black and Tan Coonhound through both doberman-like and rottweiler-like head features, as well as a neck that is also assigned to pandas or bears. Figures T9 and 20 additionally include examples for contrastive class representations learned on Stanford Cars and TravelingBirds. Finally, table [ contains results for diversity @5, to quantify its inability to capture the high spatial diversity of PIP-Nets class detectors.
M FEATURE DIVERSITY LOss
This section further describes the Feature Diversity Loss Laiv, proposed in Norrenbrack et al. (2022). is defined per sample, for which the model predicted the class è = arg max(y) and ensures a local diversity of the used feature maps M x xhar.
exp(mg))
fa wa.
(31)
M Ej' exp(mgy) max f |w2ll22
hM
Laiv
max ($)
(32)
Equation 31 employs the softmax function to normalize the entries m of the feature maps M across spatial dimensions. It then scales the maps emphasize visible and significant features, maintaining
13

---

Published as a conference paper at ICLR 2025
age Coupe 2012 Convertible 2012
Aston Martin Virage Convertible 2012 Features
Astio Mariin Vir- Aston Martin Vir
AUKE
Aston Martin Vinage Coupe 2012 Features
Figure 19: Faithful global interpretability of our QPM trained on Stanford Cars: Without any additional supervision, QPM learns represent the Convertible and Coupe Variant using diverse and general features.
Black Billed Curkoo Features
Class Images
Slack billed Carkoo
Yelnow billed O Ouceso
Yellow billed Cuckoo Features
Figure 20: Faithful global interpretability of our QPM trained on TravelingBirds: Without any additional supervision, QPM learns to represent the Yellow and Black billed Cuckoo using diverse and general features, correctly ignoring the correlated background.
the relative mean of M while weighting them according to the predicted class. Equation 32 then applies
of the normalized and scaled feature maps S. The result is negatively weighted and thus encourages the model to learn features that localize on different image regions. The resulting total training loss is
Linal LOE + BLaiv:
(33)
with 3 E R as weighting factor.
N OPTIMALITY OF SOLUTION
In order test the optimality of our solution, try solve the problem without our relaxation in eq. (9) with more compute and time. We used 3 days and 250 GB on an AMD EPYC 72F3 to solve the problem globally across seeds on CUB-2011 with target gap to optimality of 1% to ensure sufficient deduplication. The time limit was left to 3 hours for one iteration, as otherwise multiple iterations would not finish. Across the seeds used for QPM, the average obtained objective value for the global problem was 0.5% above the one computed with our simplifications. Similar to our ablations in table 4 the resulting accuracy for the extensively optimized model was not improved, but even 0.1 percent points lower. As mentioned in section 1.1. the objective does not perfectly correlate with downstream metrics, as the constants A, and only approximate the desired behaviour. However, the average gap the best bound was still 3. 3.2%. with only negligible progress during the final iteration, suggesting that longer time limit would not significantly improve it. Note, that the best bound might be violating constraints, already added or not. In summary, the
14

---

Published as a conference paper at ICLR 2025
gap between easy- cusy-to-compute solution and obtainable solution of the global problem is 0.5%, which leads to no improved model, with an upper bound on the gap of 3.7% (372 to 386).
STANDARD FORM FOR QUADRATIC PROBLEM
The quadratic problem, described in section 3.1
be expressed in the standard form for quadratic programming problems. The aim is to optimize quadratic problems of the form xx Qx + c x with respect specified constraints. To describe define the variables x, Q, as well as the constraints. For notation, 0 and 1 describe a vector with x zeros ones respectively and 0m.." describes a m n matrix of zeros.
quadratic problem in standard form, therefore
Variables
Let x be the binary decision variable vector, combining & and the vectorized form of W:
vec E 10.1)"**ee
Objective Function
The standard objective function includes all objectives:
Maximize:
xxx +xxxxxxx
Here
R
17 the
(34)
stack
0 Re.
combines all quadratic objectives and
(35)
the linear term. Here
diag(a1) (11)
diag (2)
Astack
(36)
diag an
connects the vectorized entries of W with A.
Constraints
1. Constraint for the number of selected features (eq. (2
T
1R
07
x = 12
(37)
2. No assignments on unselected features:
SeatureSum Ony-Re] (1 (mo+)) - -x) =0
(38)
featureSum = =0" FeatureSel"!]
(39)
where FeatureSel"/ E {0, 1 is matrix of zeros with FeatureSel,, = 1 where (j i) mod ny = 0. The vector featureSum captures the total number assignments
per feature.
3. Constraint for the number of assignments per class (eq. 33)
UBD
Towe
(40)
Where the upper block diagonal matrix
nn
0
0
0
1
/
(41)
0
1n
is a block diagonal matrix with n, ones per row, one 1 in each of the ny n, columns and ne total rows.
15

---

Published as a conference paper at ICLR 2025
Table 11: Accuracy without background removal based on the Ground-Truit with Resnet50. QPM less susceptible to the spuriously correlated backgrounds with and without Center Bias & Center NA indicates no convergence.
Method
CUB
TRAVEL
Baseline Resnet50
84.2
33.8:40.6
glm-sagas
75,0
35,6
PIP-Net
7..9
59.4
ProtoPool
75.0=1..3
NA
SLDD-Model
82.2++0.1
62.6:E1.6
Q-SENN
82.8
67.0
QPM (Ours)
82.
64.7++0.7
w/ Center Bias Center
82.4+0.3
68.9+0.5
4. No duplicated classes (eq.
(
Eq
On (mst)))))) 00 Vo # fl....,, ne}
(42)
Equ
[enytt - Fe -ee Ad {1, (l..,,,,,
(43)
Table 12: Ablation Studies investigating impact of incorporating feature-feature similarity through R and locality bias b on CUB-2011 with Resnet50.
R
Accuracy t
SID@5 f
Correlation +
84.6+10.4
89.5+0.2
33.9
84.4+-0.2
90.4+-0.3
33.5+19.4
85.0+10.3
89.410..3
22.7
85.1
90.1
24.6
16

---

Published as a conference paper at ICLR 2025
Table 13: Comparison on compactnes and accuracy with Resne150: QPM shows increased accuracy and compactness The compactness accuracy trade-ofi off is shown in fig. 7 Among more interpretable models, the best result is marked in bold, second best underlined.
IMGNET
2048
10
Features Class TRAVEL
2048
202
10
CARS
2048
11
195
10
CUB
2048
5
12
202
5
5
10
IMGNET
1627#1
2048
50
50
Total Features.] TRAVEL
781
744=15
2048
202
50
50
50
CARS
2048
807
6691: 13
195
50
50
50
809
731=19 19
CUB
2048
202
50
50
50
IMGNET
58.0=0.0
72.7+00.0
74.2+0.0
74.5
76.1
TRAVEL
59.5
58.0=1.2
74.0=04
38. 1..
75.6++0.2
75.7++0.8
75.2++0.6
Accuracy
CARS
12.11
86. 66.6
36.5+0.3
79.4 87.5
84.5+-0.2 1+0.1
91.8++0.
9..1
86.6+=0.2
78.0 88.040.4
82 92.0+0.33
85.1 1..33-0..
85.7
CUB
Baseline Resnet50
Yhure: 10 (Ours)
SI SL.DD-Mocs Model
QPM (Ours)
glm- sagas
ProtoPool
Method
PIP-Net
17

---

Published as a conference paper at ICLR 2025
Table 14: Comparison on Interpretability metrics with Resner50. Due to required annotations, Structural Grounding can only be computed for TravelingBirds and CUB-2011. Among more interpretable models, the best result marked in bold, second best underlined.
Strectura Grounding CUB
TRAVEL
32. 22.1200.2
4.44-2.8
6.911.3
7.6+:2.5
30.7=3.1
54.3=4.0
62.9288
34.0=0.3
2.54:.0.
6.710.99
13.9-00.9
29.2:44.0
47.9
52.3
IMGNET
71,740.00
93.4
89 89.3+-0.1
87.40.11
71.6
TRAVEL
74.4+0.1
73.8-0.1
99.7 7...0
85.2
86.3
94.0=0.3
94.2-0.3
Contrastiveneass CARS
75.1+
74.540.1
99.7100.0
78.9+2.0
89.710.3
97.7+0.4
98.6:10.2
74.4+00.1
74.0=20.1
99.610.0
76.7+11.0
87.320.2
96.0=0.4
95.910.5
CUB
IMGNET
99.420.00
98.620.00
99. 19.1000.0
0..
99.4
Class- Indu-nedeeneeeee ? TRAVEL
98.0=00.0
97.8+0.0
73.410.11
95.5+001
96.5=0.1
97,2+0.0
98.1000.0
CARS
97,8+00.0
97.6+10.
62.910.1
96.0=00.5
95.5#0.1
96.5+00.0
98.0:0011
98.0=0.0
97.8-00.0
75.6110.4
96.9=0.1
96.2-0.1
97.0=0.0
08.1
CUB
IMGNET
35.8+0.00
64.720.7
04.120.7
80.110.9
37.1
TRAVEL
59.9+-0.4
56.04-0.8
98.710.11
31.5+:1.6
87.51:0.4
89.74-0.2
95.11:0.3
SI@@1 +
CARS
54.4+0.3
1-.3--
19.0110.1
30.7+:3.44
98.6610.66
89.6 99.6++0..4
96.6.10.55
57.7=0.4
55.4=:0.5
09.2.1.11
24.5=00.8
08.
90.1 0.3
95.
CUB
Baseline Resner50
Rac 10 (Ouns)
SLDD-Model
QPM (Ours)
glm- mmn-sagas
ProtoPool
Method
PIP-Net
18

---

Published as a conference paper at ICLR 2025
Table 15: Comparison on compactness and accuracy with Resnet34: QPM shows increased accuracy and compactness. Among more interpretable models, the best result marked in bold, second best underlined.
CARS TRAVEL
2048
10
Features CClass
2048
5
10
CUB
2048
5
5
5
10
TRAVEL
2048
443+6
50
50
50
Total Features.
CARS
2048
45316
50
50
50
CUB
2048
4424.
50
50
50
TRAVEL
61.3104
53.5+0.88
74.0
75.1+00.3
75.74-0.6
Accuracy CARS
91.5+:02
82.0+:066
90.710..
91.3+000
91.7++0.1
85.7+0..3
1.00+110
83.2+0.33
83.0
83.9+-.11
CUB
Baseline Resner34
Towe = 10 (Ours)
glm sagas SLDD-Model
QPM (Ours)
Method
19

---

Published as a conference paper at ICLR 2025
Table 16: Comparison on Interpretability metrics with Resner34. Due to required annotations, Structural Grounding can only be computed for TravelingBirds and CUB-2011. Among more interpretable models, the best result marked in bold, second best underlined.
Structural Groundingt
TRAVEL
36.0+0..4
9.4=3.8
29. 29.415.2
49.7
60.3++2.0
39.61:0.2
7.6
24.5
39.0-12.99
54.7+-3.8
CUB
TRAVEL
76.0=10.1
76.0=0.1
82.0
94.3
95,4++0.2
Contrastivenesst CARS
77.910.2
77.8=0.2
83.314.6
94.7+11.1
98.7+00.5
76.410.1
76.5+:0.0
86.0
95.5
96.9+-0.3
CUB
TRAVEL
98.0=10.0
97.911.0
97.8=10.0
97.6=10.0
96.1+0.1
Class- Class-Indepentt Inss-nnneenddcce 1
97.7+0.0
97.7+0.0
97.6
96.911.11
98.4+-0.0
CARS
97.9:1.00
97.910.0
97.5
97.5+10.0
97.9++0.1
CUB
TRAVEL
64.1
60.6+:0.4
87.3+0.3
89.7+-0.7
98.3+-0.0
56.6++0.4
55.3+0..3
86.7+22.5
1..1
97.0++0.3 98.1+-0.5
SID@5 CARS
62. +0333
59. $9.940.44
90.1 90.1
90.5
CUB
Baseline Resnet34
Ylaee = 10 (Ours)
SLDD-MooE Model
QPM (Ours)
glm sagas
Method
20

---

Published as a conference paper at ICLR 2025
Comparison on compactness and accuracy with Inception-v3. QPM shows increased accuracy and compactness. Among more interpretable models, the best result is marked in bold, second best underlined.
CARS TRAVEL
2048
5
5
5
10
Features /Class
2048
5
5
10
CUB
2048
10
CARS TRAVEL
813+19
2048
50
50
50
Total Features
2048
7951:8
50
50
CUB
2048
8141-9
50
50
50
TRAVEL
68. 74.044
63 63..5
69.910.2
71.5
70.8+
Accuracy 1 CARS
92.61-222
89.310.3
91.11:0.2
91.7.0.11
91.7++0.2
86. 1
79. 09.250..5
83. 13.140..4
84.2-0..4
84.4+004
CUB
Baseline line Inception-v3
Tboee = 10 (Ours)
LLDe-M
QPM (Ours)
glm- sagass
Method
Table 17:
21

---

Published as a conference paper at ICLR 2025 5
Table 18: Comparison on Interpretabiig metrics with Inception-v3. Due to required annotations. Structural Grounding can only be computed for TravelingBirds and CUB-2011. Among more interpretable models, the best result marked in bold, second best underlined.
Structural Groundingt TRAVEL
24.1=:0.3
2.5+2.88
27.2+5.2
44.343.88
50.3+1.6
7.149..6
0.340.4
24.412.3
34.8133.4
41.9+3.0
CUB
TRAVEL
91.7 89.8110.1
91.2+0.11
92.310.3
93.411.1
92.6+0222
Contrastivenesst CARS
93,4+:.22
944102
94.310.1
93.6+-0.2
89.6:1.22
91.3+0.3
93.0
93.4
92.6+0022
CUB
TRAVEL
95.9++0.0
95.3+00.0
92.6+0.22
95.1+0.1
96.9+0.11
Class- Independence CARS
95.7++0.0
95.0+:0.0
92.1
94.7+0.0
96.8+00.0
96.1+..00
95.4+00.0
92.610.1
95.1
96.
CUB
TRAVEL
40. 40.74:0.4
41.0+...3
1.3
50.2+0..4
15.0++144
33. 13.12:0.2
34.01:..4
52.11:.5
42.8
47.1
SID@5 CARS
38.910..3
39.3 09.32:0.2
1.2
48.6
+-.6
CUB
Baseline Inception-v2 v3
Toere = 10 (Ours)
SLDD-Model
QPM (Ours)
glm sagas
Method
22

---

Published as a conference paper at ICLR 2025 5
Table 19: Comparison on compactness and accuracy with Swin Transformer small: QPM shows increased accuracy and compactness. Among more interpretable models, the best result marked in bold, second best underlined.
CARS TRAVEL
768
5
5
10
Features Class
768
5
5
5
10
CUB
768
5
5
10
CARS TRAVEL
561+11
768
50
50
50
Total Features
55918
768
50
50
50
CUB
768
572
50
50
50
TRAVEL
59.0+0.4
16.4+1.0
60.3+1.0
61.0
60.9--1.1
Accuracy 1 CARS
90.6:10.6
1..5+1.2
89.1=10.7
88 88.7-0.5
89.0=0088
87.0=0.11
76.5+0044
85.3
85.0+10.4
85.4++0.3
CUB
Baseline line Swin Transformer small
Torc = 10 (Ours)
SLDD-Model
QPM (Ours)
glm- sagaa
Method
23

---

Published as a conference paper at ICLR 2025
Table 20: Comparison on Interpretability metrics with Swin Transformer small. Due required annotations, Structural Grounding can only be computed for TravelingBirds and CUB-2011, Among more interpretable models, the best result is marked in bold, second best underlined.
Structural Grounding CUB
TRAVEL
23.6+00.9
6.9+
40.6+2.4
43.1
52.5+2.0
24.5
8.8+2.8
37.2+34
45.1 45.1+332
52.1
Contrastiveness TRAVEL CARS
97.4+0.1
98. 18.3+0.11
99.3+0044
99.1+00.3
94.7+110
98. 18.8+1.11
99.6+0.0
99,4+0022
99.1 19.120.2
97.710.
98.3+0.1
99.1+0.1
99.0+0022
98.5+0.33
95.5+0.3
CUB
TRAVEL
96.9+0.1
96.7++0.0
93.6+:0.
95.2+0.11
9691
Class Srs---ndeeeedence f
19.66+0.0
96.4+0.0
93.3++0.2
94,7+0.00
96.6110.0
CARS
96.8 +6.0
96.6++0.0
93.4+00.1
95.2+00.0
96.9
CUB
TRAVEL
29.
30.0++044
43.2+1.4
37,9+0.7
37.3+1.7 44.6111.
SID@51 CARS
26.0=0.1
26. +0.1
35.6=1.11
32.0#2.33
16.4
26 46.2
38 08.0=0.5
33 33.6204
40.400.7
CUB
Base line Swin Transfore formerr small
Row = 10 (Ours)
SLDD-Model Model
QPM (Ous)
glm-san sagass
Method
24

---

Published as a conference paper at ICLR 2025
Table 21: Comparison on compactness and accuracy with Swin Transformer tiny: QPM shows increased accuracy and compactness. Among more interpretable models, best result marked in bold, second best underlined.
TRAVEL
768
5
5
5
10
Features Class
CARS
768
5
5
5
10
CUB
768
5
10
CARS TRAVEL
569+10
768
50
50
50
Total Features
55919
768
50
50
50
5591
CUB
768
50
50
50
TRAVEL
60.1 60.1600.6
1.5
58.3
59.3:10.9
59.8+=0.9
Accuracy CARS
90.3:0..3
70. 70.8+1.44
88.4
88 1.30
18.3-+1.00
86.6:10.2
71 71.8+-2.0
84.4
84.5
84,6--04
CUB
Baseline Swin Transformer tiny
Yloe:: = 10 (Ours)
SLDD-Model Model
QPM (Ours)
glm -agass
Method
25

---

Published as a conference paper at ICLR 2025
Table 22: Comparison on Interpretability metrics with Swin Transformer tiny. Due to required annotations, Structural Grounding can only be computed for TravelingBirds and SUB-2011, Among more interpretable models, the best result marked in bold, second best underlined.
Structural Groundingt CUB
TRAVEL
23.1100.4
6.1
41.2+
51.412.1
54.5
26. 110.2
12. 12.1+3.9
13.855.7
50.91499
54.7
TRAVEL
98.410.1
98.9+0.1
98.9
98410.2
92.8
Contrastiveness CARS
99.010.1
99.5+0.1
19.3+0.4
98.710.44
97.4=0.8
98.8.10..
99.3+-0.0
99.2++0.3
982-104
93.8+00.2
CUB
TRAVEL
96.910.0
96.7+0.0
93.6=20.2
95.310.1
97.0
Class- Iee--nndeeddnnce
96.6.10.0
96.5+0.0
92.9=1.1
94.6.10.1
96.3=0.0
CARS
96.8
96.6++0.0
93.3+022
95. 95.1 11..1
96.9+0.0
CUB
TRAVEL
30.910.2
29.9+0.3
46.6
41.810.3
48.5#0.99
26.310.1
25.9+0.11
35.41..6
31.51114
37.5=2.44
SID@5 CARS
27.3
26.4+:0.2
38.9+0066
36.91044
44.6+:0.5
CUB
Baseline Transformer tiny
Tlwe = 10 (Ours)
SLDD-Model
QPM (Ours)
glm- singsts
Method
26

---

Published as a conference paper at ICLR 2025
3.0
399
3.5
2.1
2.6
-07
0.6
05
08
Figure 21: Exemplary Activations of Features in fig. Jon further Rottweiler images. The feature values after normalization are written on the images. Note that all shown activations are scaled from 0 to 1, resulting in arbitrary localization of the brown feature detecting the Doberman-liie head.
27

---

Published as a conference paper at ICLR 2025
-.5
008
3.
3.1
4.5
2.5
2.4
2.0
3.5
3.6
3.0
Figure 22: Exemplary Activations of Features in fig. on further Doberman images. The rounded feature values after normalization are written on the images. Note that shown activations are scaled from 0 to 1, resulting in arbitrary localization of the red feature detecting the Rottweiler-like head.
28

---

Published as a conference paper at ICLR 2025
1,8
355
0.5
05
Figure 23: Exemplary Activations of Features in fig. on Black and Tan Coonhound images. The rounded feature values after normalization are written the images. Note that all shown activations are scaled from 0 to I, resulting an arbitrary localization the two not assigned and barely activated blue and green features. The fifth assigned feature is shared with dog types such Newfoundlands, bears and pandas, localizing on the neck region.
29

---

Published as a conference paper at ICLR 2025
Figure 24: Screenshot of the Cornell-La, Lab website (Sullivan 2024 describing how the similar species shiny and bronzed cowbird differ: The only differences explained are in the size, which is not usable for our QPM due cropping, and the eye color, which our QPM QPM learned without any supervision and explained in fig. 4 as the differentiating factor.
Male Bronded Cowebods are largor and have aed whereen Shiny Condrid have ddkk
trin Suttoan Mocadda Libory Tooms Appi 20, 2011
Simlarr Species Bronzed Bronzed Cowbird
Adult male
Churky with coniccll bit Males glane black evenl.
Brinn Suftran Mitaadayy lenry Cheir Church March-0 2016
Main Species
Shiny Shiny Cowbird
Male
30


Reviews:
[{"review_id": "ftzcfrilfy", "reviewer": "Reviewer_vzLV", "summary": "The paper introduces the Quadratic Programming Enhanced Model (QPM), an approach for globally interpretable image classification. Unlike traditional models that primarily focus on local explanations, QPM achieves a compact, interpretable global representation of each class by assigning only a few features, typically five, per class through discrete optimization. This binary feature assignment ensures contrastive and general class representations, which allows for clear, globally understandable explanations. The model sets a new benchmark in both interpretability and accuracy for compact, interpretable models across small and large datasets, including ImageNet-1K. The approach shows robust performance by using optimally selected features that enhance global interpretability and structural grounding while maintaining high classification accuracy.", "questions": "- Presentation-wise, I think it is better to explicitly show what the features are in human language for the figures, so that it is more friendly to the readers.\n\n- Structure ground is only evaluated on CUB-2011, and the authors also explained why. How much effort would it take to achieve the evaluation metric on a larger dataset, such as ImageNet?\n\n- How long is the optimization of “4 hours for fine-grained datasets, and roughly 11 hours for ImageNet”? Are these times better than previous methods?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper studies global  an important question in explainable machine learning, providing global interpretation through user defined features.\n\n- The paper motivation is well justified, well-written and easy to understand.\n\n- The results show that QPM achieves superior performance in structure grounding, which signifies that QPM’s learned class representations align with human-understandable features. This is very crucial in explainable machine learning.", "weaknesses": "- The paper operates on only a few features, which is fine for natural images. However, I think in more complex domain applications such as medical imaging, more features might be required. How would more features affect the model’s optimization of feature selection in terms of time and accuracy?\n\n- On that note, the paper lacks experiments related to high-stake domain applications such as healthcare, autonomous driving, etc.\n\n- In QPM, Pearson correlation serves as a fundamental similarity measure that aids in identifying which features are most relevant to each class. By focusing on linear relationships, it allows the model to assign interpretable, contrastive, and compact features to classes. However, because it overlooks non-linear relationships, it may not capture all nuances in complex datasets, potentially impacting accuracy in applications where non-linear patterns are significant. Could the author elaborate on this matter?\n\n- QPM demonstrates inferior performance on SID@5, Class-Independence, and Contrastiveness compared to state-of-the-art models, suggesting that its compact binary feature assignment strategy, while beneficial for interpretability, may limit the model's ability to capture diverse, independent, and highly contrastive feature representations across classes. This trade-off reflects QPM's focus on simplicity and interpretability at the expense of some nuanced, dataset-specific feature diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3aCwzTsnnN", "reviewer": "Reviewer_x2rh", "summary": "To develop inherently interpretable image classification, the authors propose to use a set of five diverse features for each class to perform classification. Specifically, the proposed method leverages discrete quadratic programming to extract the features that are distinct, contrastive, and generalizable across similar classes from pretrained vision models, e.g., ResNet-50. The paper provides extensive empirical results on prediction accuracy and interpretability metrics.", "questions": "See Weakness for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea is intuitive and easy to understand.\n2. The paper is well-organized and easy to follow.\n3. Different from most interpretable models, the proposed method didn’t compromise too much on the prediction accuracy.", "weaknesses": "1. Although the proposed method can provide five different heatmaps to explain the prediction of a class, the semantic meanings of these explanations are subjective. This is different from Concept Bottleneck Models (CBMs) and Testing with Concept Activation Vectors (TCAV) that can provide human-understandable concept explanations with semantic meanings. Although the authors propose structural grounding that uses dense annotations of bird parts from the CUB dataset to interpret some of the heatmaps, the concept coverage might be restrictive and cannot reflect what the model has learned.\n2. The choice of five features per class might be arbitrary. Some classes might be more complicated than others, while some classes could be very simple. Would it be better to use a different or dynamic number of concepts for each class? This is actually related to the Weakness 1 above. The restricted number of five concepts makes the model learn polysemantic features, which means it might represent different concepts within a single feature. This can make the interpretation of the functionality of the features even harder. For example, as shown in Fig. 12, the green feature can highlight bird beaks on one image and wings on the other image. This might defeat the purpose of making the model more interpretable.\n3. The proposed method highly relies on the quality of extracted features from the pretrained models, whereas they are not designed to be explainable. Since the base models are normally trained, they can also extract biased features which would significantly affect the performance of the proposed model.\n4. Experiments using standard metrics for explanations are missing, such as the explanation fidelity using the deletion/insertion method [1]. The proposed contrastiveness metric cannot fully reflect the explanation quality in terms of faithfulness. The authors propose Scale-Invariant-Diversity@5 (SID@5), however, the proposed method cannot achieve the best or second-best performance on the three datasets.\n\n[1] Petsiuk, V. \"Rise: Randomized Input Sampling for Explanation of black-box models.\" BMVC 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vATOfbbBak", "reviewer": "Reviewer_GMvn", "summary": "With the rise of AI in several high-risk areas such as autonomous driving and medical diagnosis, being able to interpret the decisions of deep neural networks is paramount for safety. The authors focus on providing faithful global explanations as opposed to the simpler task of giving local explanations. This is achieved via their Quadratic Programming Enhanced Model (QPM) via learning interpretable class representations using only 5 features and binary assignment. This simple decision logic makes interpreting classifications feasible by humans. The authors optimize for this representation and predictive power using discrete optimization on diverse pretrained features. They evaluate their method on a variety of datasets and achieve competitive accuracy and interpretability compared to popular interpretability methods. The authors show qualitative results of their model’s interpretability and quantitative for metrics on complexity, spatial diversity, class-indepence of features, contrastiveness of features, and their structural grounding to human given concepts.", "questions": "- You provide misclassifications of your model in the Appendix in figure 10, but not the explanation for the classification visually. I think being able to understand the misclassification from your method’s explanation is important to your method’s strength. Could you show the explanation visualization for several misclassified samples? I think image (j) would be interesting plus images from other datasets.\n- See weaknesses for other questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Global explanations are simple and easy for humans to interpret. Weights from features are binary and only positive. Each class only has 5 features (in most experiments).\n- Robustness and generalizability of their method is given based on evaluation on a variety of datasets and models\n- Good use of discrete optimization and constraints to select diverse, localized, and predictive features from a pre trained set of features. Clever relaxation of constraints for feasible optimization while maintaining accuracy and interpretability.\n- Ability for concepts/features to be used for predicting multiple classes.\n- Competitive or SOTA accuracy and interpretability compared to other popular methods. Qualitative results show the ease of interpretability of their method.\n- Ablation experiments show effectiveness of $Z_B$ and $Z_R$ objectives.", "weaknesses": "Minor:\n- Writing can be unclear at times:\n  - L282: Are you keeping the entries that are being clipped? Between 0 and epsilon.\n  - L127-129: This sentence is unclear. How does the balanced binary assignment cause general binary concepts?\n  - L372-273: You say you omit measuring the grounding of features, but provide results…Do you mean that you don’t focus on it since you only do this for CUB & TravelingBird due to lack of part annotations in other datasets?\n\nWorth noting:\n- While your method doesn’t use negative reasoning, which makes it more simple, it is a restriction/limitation.\n- Your method’s performance isn’t that much above Q-SENN, but you mention the computation limitation on Q-SENN. A comparative analysis, such as a table, that shows the speed of training differences between methods would be desirable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)