Take 5: Interpretable Image Classification with a Handful of Features
Authors: Thomas Norrenbrock
Marco Rudolph
Bodo Rosenhahn
Venue: Submitted to ICLR 2023 (2023)
Keywords: xAI
interpretability
fine-grained image classification
sparsity
image classification
interpretability by design

Abstract: Deep Neural Networks use thousands of mostly incomprehensible features to identify a single class, a decision no human can follow. We propose an interpretable sparse and low dimensional final decision layer in a deep neural network with measurable aspects of interpretability and demonstrate it on fine-grained image classification. We argue that a human can only understand the decision of a machine learning model, if the input features are interpretable and only very few of them are used for a single decision. For that matter, the final layer has to be sparse and – to make interpreting the features feasible – low dimensional. We call a model with a Sparse Low-Dimensional Decision “SLDD-Model”. We show that a SLDD-Model is easier to interpret locally and globally than a dense high-dimensional decision layer while being able to maintain competitive accuracy. Additionally, we propose a loss function that improves a model’s feature diversity and accuracy. Our interpretable SLDD-Model only uses 5 out of just 50 features per class, while maintaining 97% to 100% of the accuracy on four common benchmark datasets compared to the baseline model with 2048 features.

Under review as a conference paper at ICLR 2023
TAKE 5: INTERPRETABLE IMAGE CLASSIFICATION WITH A HANDFUL OF FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Deep Neural Networks use thousands of mostly incomprehensible features identify single class, decision no human can follow. We propose an interpretable sparse and low dimensional final decision layer in a deep neural network with measurable aspects of interpretability and demonstrate it fine-grained image classification. We argue that human can only understand the decision of a machine learning model, if the features interpretable and only very few of them used for a single decision. For that matter, the final layer has to be sparse and to make interpreting the features feasible low dimensional. We call model with Decision "SI.DD-Modee". We show that a SLDDD-Model
Sparse Low-Dimension is easier interpret locally and globally than dense high-dimensional decision layer while being able to maintain competitive accuracy. Additionally, we propose loss function that improves a model's feature diversity and accuracy. Our more interpretable SLDD-Model only uses 5 out of just 50 features per class, while maintaining 97 to 100 % the accuracy on four common benchmark datasets compared to the baseline model with 2048 features.
Imput
Tril slsgger zerecle
Sparse Deciston
Extrackor
Visketen
Feature Activation:
Jow
high
colot: green
Figure 1: Local explanation by SI.DD-Model The two features used for the predicted class, emerged without additional supervision, are aligned with human interpretable attributes and localized (described in App. D) adequately.
INTRODUCTION
Understanding the decision of deep learning model is becoming more and more important. Espe- cially for safety-critical applications such as the medical domain or autonomous driving, it is often either legally (Bibal et al. (2021) by the practitioners required be able to trust the decision and evaluate its reasoning (Molnar 2020). Due to the high dimensionality of images, most previous work on interpretable models for computer vision combines the deep features computed by deep neural network with method that is considered interpretable, such as a prototype based decision tree (Nauta et al. 202I). While approaches for measuring the interpretability without humans exist for conventional machine learning algorithms (Islam et al. 2020), they missing for methods including deep neural networks. this work, we propose a novel sparse and www-dimensional SLDD-Mods which offers measurable aspects of interpretability. The key aspect is heavily reduced number of features, out which only very few are considered per class. Humans can only consider 7 + aspects once Miller 1956) and could therefore follow decision that uses that many features. To

---

Under review as a conference paper at ICLR 2023
be intelligible for all humans, we aim for average of 5 features per class. Having a reduced number of features makes it feasible to investigate every single feature and under- stand its meaning: We are able align several of the learned features with human concepts post-hoc. The combination of reduced features and sparsity therefore increases both global How does the model behave? and local interpretability Why did the model make this decision?. demonstrated in Figure I Our proposed method generates the SLDD-Model by utilizing gim-saga (Wong et al. (022] to com- pute sparse linear classifier for selected features, which we then finetune to the sparse structure. We apply feature selection instead of a transformation to reduce the computational load and preserve the original semantics of the features, which can improve interpretability (Tao al. 2013), especially if more interpretable model like B-cos Networks (Böhle et al. 2022) is used. Additionally, we propose a novel loss function for more diverse features, which is especially relevant when one class depends on very few features, since using more redundant features limits the total information available for the decision.
Our main contributions are as follows:
We present a pipeline that ensures a model with increased global and local interpretability which identifies single class with just few, e.g. 5, features of its low-dimensional representation. We call the resulting model SLDD-Model.
Our novel feature diversity loss ensures diverse features. This increases the accuracy for the extremely sparse case.
We demonstrate the competitive performance of our proposed method on four common benchmark datasets in the domain of fine-grained image classification as well as ImageNet- IK (Russakovsky et al.) 2015). and show that several learned features for algorithmic decision-makee can be directly connected to attributes humans use.
Code will be published upon acceptance.
2 RELATED WORK
2.1 FINE-GRAINED IMAGE CLASSIFICATION
Fine-grained image classification describes the problem of differentiating similar classes from one another. It is more challenging compared to conventional image recognition tasks (Lin al. 2015) since the differences between classes are much smaller. To tackle this difficulty, several adaptions to the common image classification approach have been applied. They usually involve learning more discriminative features by adding a term to the loss function (Chang et al.] 2020 Liang et al. 2020
Zheng et al. 2020), introducing hierarchy the architecture (Chou et al. (0222) or using expensive
expert knowledge (Chen et al.) 2018 Chang et al.) (021). Chang et al. (2020) divide the features into groups, s. every group is assigned to exactly one class. While training, an additional loss increases the activations of features for samples of their assigned class and reduces the overlap of feature maps in each group. Liang et al. (2020) tried to create specifc filters by inducing sparsity in the features. Both (Chang et al., 2020) and (Liang et al., 2020) optimize for class-speciff filters, which are neither suitable for the low dimeennnnio case when the number of classes exceeds the number of features nor interpretable, since it is unclear if the feature is already detecting the class rather than a lower level feature. The Feature Redundancy Loss (Zheng et al.] (020) (FRL) enforces the K most used features to be localized differently by reducing the normalized inner product between their feature maps. This adds a hyperparameter and does not optimize all features at once.
2.2 INTERPRETABLE MACHINE LEARNING
Interpretable machine learning is broad term and can refer to both models that are interpretable by design, and post-hoc methods that try to understand what the model has learned. Furthermore, interpretability can be classified as the interpretability of a single instance (local) or the entire model (global) (Molnar, 2020).
In this work, we present methods making models more interpretable by design but also utilize post-hoc methods to offer local and global interpretability. Common local post-hoe methods are saliency maps like Grad-CAN (Selvaraju et al. (2017) that aim to show what part of the input image is relevant for the prediction. While they can be helpful, they have to be cautiously interpreted, they do not show
2

---

Under review as a conference paper at ICLR 2023
many desired properties would expect from an explanation like shift invariance (Kindermans et al. 2019) or only producing reasonable explanations, when the model is working as intended (Adebayo et al. 2018 Another way of obtaining saliency maps is based on masking the imput image and measuring the impact on the output (Zeiler & Fergus| 2014, Fong & Vedaldi 2017] Jain et al. 2022). As global post-hoe method, Elude (Ramaswamy al. 2022) generates an explanation for model by mimicking its behavior with a sparse model. This model uses additional attributes and main directions of the remaining feature space of the model as input. Instead of explaining a model, we directly train the more interpretable model in this work. Another line of research tries to align learned representations with human understandable concepts from an additional labeled dataset (Kim et al.) 2018; [au et al., 2017 McGrath et al., 2021), increasing the global interpretability of the model. To our best know ledge measuring the interpretability of a deep neural network is an open task, as previous work focuses on measuring the quality of explanations of black boxes (Rokade & Alluri 202I) or on conventional machine learning algorithms (Islam et al., 2020), where increased inter- pretability is measured when model complexity is reduced, e.g. via the number of operations (Yang) et al. 2017; Friedler et al.] 2019; Rüping et al.) 2006) or number of features (Küping et al. 2006))
The sparsity and now-dimensionality of our proposed SLDD-Model is motivated by these findings. Due to the limitations of post-hoc methods in explaining deep neural network, models that are more interpretable by design are becoming more relevant. Nauta al. (2021) and Yang et al. (2022) used deep feature extractor in combination with a prototype based decision tree(s) to achieve state-of-the-ard performance on fine-graing grained image classification while increasing interpretability. However, decision trees struggle to model linear dependencies, and it is hard to globally interpret an ensemble of deep decision trees. Kim et al.] (202T) and [Hoffmann et al.) (202T) additionally indicate gap between the perceived similarities of humans and prototype based models. Concepr bottleneck models (CBM) (Koh et al. 2020) first predict concepts annotated in the dataset and then use simple model to predict the target class from the concepts. CBM-AUC (Sawada & Nakamura (0022) extended CBM by allowing unsupervised concepts to influence the decision. PCBM (Yuksekgonul et al. 2022) created post-hoc CBM using TCAV (Kim et al., 2018) compress sigh-dimensional learned features into a concept bottleneck. Margeloiu et al. (2021) and Elude (Ramaswamy et al. 2022) both suggest that training the CBM end-to-end leads to the encoding of additional information next to the concepts, which reduces the interpretability.. In contrast to CBM, our proposed method does not require additional labels for training and leads to a very sparse decision process. While their features are generally more aligned with the given concepts, they also need to be analyzed thoroughly.
2.2.1
Glm-saga
Wong et al. (2021) developed gim-saga, a method to efficiently fit a heavily regularized sparse layer to the computed features of a backbone feature extractor by combining the path algorithm of
Friedman et al. 2010) with advancements in variance reduced gradient methods by Gazagnadou ct al. (2019). They showed that human understanding is more aligned with the decision process of the sparse model and sparsely shared features can be more easily aligned with human concepts. Additionally, they reached levels of sparsity that network wide sparsity methods do not obtain in the final layer with competitive accuracy (Gale et al., 2019). For precomputed and normalized features, glm-saga computes a series of n sparse linear classifiers
[W b1). (W Sparas b2).
(Wsparse
sparse bb,)],
W where the sparsity of Sparas is decreasing with i. This series is called regularization path, Each of the models minimizes the elastic net loss
L = Liarget + XR(W)
R(W) = (1- a1-a), a) W|| F| + W11,1
(2)
with the initial optimization goal Ciarget, in our case the cross-entropy loss, and regularization strength X, which decreases along the path. The regularization function R(W) with weighting factor a E [o. 1] is known as Elastic Net (Zou& & Hastie, 2005). Gim-saga optimizes the problem iteratively, clipping entries in W with an absolute value below a threshold after each step, to ensure real sparsity. For reference, the pseudocode for glm-suga is included in Appendix C Since their aim is understanding the neural network. they fit the sparse layer to fixed features and do not finetune the features to the sparse layer, which requires different optimization strategies than dense networks (Tessera et al. 202I). In this work, we utilize glm-saga to create a more interpretable model with competitive accuracy by applying it on selected features with higher diversity and finetuning the
3

---

Under review as a conference paper at ICLR 2023
features afterwards. This leads to improved accuracy and enables higher sparsity which, combined with the reduced number of fratures,increase interpretabilityy
METHOD
3.1 PROBLEM FORMULATION
We apply the proposed SLDD-Model to the domain of fine- fine-grained image classification. We consider the problem of classifying an image I 3xxxx of width w and height h into one class c using trainable deep neural network Φ. This neural network extracts the feature maps M E TR was xhar and aggregates them into the feature vector f E R Then it applies the trainable neural network C' to obtain the final output ' as y C(f).
{cI: .2..... Cr.
3.2 SLDD-Model
Train Dense
Feature
Sparsify Layer
Finetune Features
SLDD-Model
Model
Selection
with Laiv
Figure 2: Overview of our proposed pipeline to construct SLDD-Model
We propose flexible, generally applicable method for generating pretable model with no need for additional labels and an adjustable tradeoff between interpretability and accuracy. We make the decision process more interpretable by only using n features with ni ny and using interpretable classifier C. At the core of an interpretable classifier C lies linear layer = W + b with the weight matrix W RR) and bias b E R In order for it to be interpretable, W has to very sparse, meaning the number of non-zero entries Nw has to be very low. Miller 1956) showed that humans can handle 7 + 2 cognitive aspects at once, which constitutes an appropriate upper bound on the average number of relevant features per class Twwe na. In our work we focus on Tww
locally and globally inter-
The pipeline of our approach is presented in Figure 2 and utilizes glm-saga for sparsification and feature selection. We first train a deep neural network with our proposed feature diversity loss Laiv until convergence. Then the features ptrain E T for all ny images in the training set are computed, which are the average pooled feature maps M. Afterwards, the features are selected as described in Section 3.2.2 and glm-saga, presented in Section 2.2.1] is used to calculate the regularization path. Finally, the solution with the desired sparsity is selected from the regularization path and the remaining layers get finetuned with the final layer set to the sparse model, s. t. the features adapt it.
3.2.1 FEATURE DIVERSITY LOss
The goal of the proposed feature diversity loss Ldiv is that every feature captures a different indepen- dent concept. This is achieved by enforeing differently localized features their feature maps. The proposed loss is motivated by the Mutual-Channel Loss (MCL) (Chang et al., 2020) and the Feature Redundancy loss (FRI) (Zheng et al. 2020 In contrast to FRI, we use Choos--hanne--axx. Mooolin (CCMP) (Goodfellow et al., 2013) over all weighted feature maps to optimize all features jointly and reduce the need for the hyperparameter K. MCL also uses CCMP but instead of grouping the channels into class-specific filters, we apply the diversity component to all feature maps M, to aim for shared interpretable features. For notation, wy describes the entry in W that is assigned to the specific feature E 10.1...., ny 1} for the predicted class à arg max(y) (y) and ml, Mis.j.
ft |wal o Ey= exp(m)yy) max f ||wellz
exp(m),)
(3)
hM
hM
Laiv =
max($! S, ..,...,,8
(4)
1=1
4

---

Under review as a conference paper at ICLR 2023
Dataset
CUB-2011
Stanford Cars
FGVC-Aircraft
NABirds
ImageNet-Il 1K
#Classes ne
200
196
100
555
1000
Training Testing 774
5 999
8 144
667
23929
281 167
004
3.333
24
50000
Table 1: Overview of the number of classes, training and testing examples for the used datasets
Equation uses the softmax transform the feature maps M by normalizing their entries m/y over the spatial dimensions and then scales the maps so that they focus on visible and important features by maintaining the relative mean of M while weighting them according the predicted class, t. different to MCL absent features do not have to be localized in small background patches. Equation decreases the loss if the different weighted feature maps S' attend to different locations. The final training loss is then Linal LC Baavv with the weighting factor R+.
3.2.2 FEATURE SELECTION
For selecting the set of features N2- from the initial features Ny s. [Ny-11 n²,, run an adapted version of gim-saga, introduced in Section 2.2.1 until one solution (Wysparae b,) of the regularization path uses feature not already in Ny., which we then add to the set of selected features Ny. and restart the adapted gim-saga. adaptation, we extended the proximal operator of the group version of glm-saga, which operates on wy W which are the entries in W that correspond to an entire feature I. Since || w2rl2 indicates the importance of I, we additionally only keep entries for features that have the maximum norm or are in Ny-, t. exactly one feature is added per iteration.
The resulting proximal operator with A = 7λa and d2 = 7A(1 a) is:
we [. (12--_ (1+ mil2
if 12 > \ A
EWW W ViENs.
Prox ,2 (w,,
otherwise
(5)
The extensions underlined and y is the learning rate of glm-saga.
4 EXPERIMENTS
This section contains experimental results. We validate method using Resnet50 (He et al.
2016), DenseNet121 (Huang et al. 2017) and Inception-v3 (Szegedy et al. 2016) on four common benchmark datasets in the domain of fine-grained image classification. Additionally, the applicability of a SLDD-Model for large scale datasets like ImageNet- IK (Russakovsky et al. 2015). An overview of CUB-2011 (Wah al. 201T). Stanford Cars (Krause et al.] 2013). FGVC- Aircraft Maji et al. 2013), NABirds (Van Horn et al. 2015) and 1K given in Table I Additionally, CUB-20I1 contains labels for the images such attributes (e.g. "red wing") as well as for the classes, which makes it easier to measure the alignment with understandable concepts. After the competitive accuracy and the impact Laiv shown, the interpretability Model is discussed and these attributes are used to show the alignment of the learned features. The implementation details can be found Appendix B.I Finally, the tradcoff between interpretability and accuracy is visualized.
show
the SLDD-
4.1 DIVERSITY METRIC
To assess the impact of Ldiv. we developed measurement for the local diversity of the feature maps M that led the decision, inspired by the diversity component of MCL (Chang et al., 2020), which entails different way of computing the features. For that, we consider the i feature maps Mr & that are weighted the highest for the predicted class è in W. To only compare the localization, softmax is applied to the Mr. yielding Sg. With these distributions Sg. we compute the diversity as
Elar Σ WM max (8))
. .........
diversity @k
(6)
5

---

Under review as a conference paper at ICLR 2023
Method
Additional Supervision required
Features trained end-to-end
Ry
Roe
ImageNet-IK f
CUB-2011
+
+
f
CBM Koh et al. 2020 join
112
112
80. (76.8*)
CBM (Koh 2020 independent
X
112
112
76.0
CBM-AUC (Sawida & Nakamura 2022
256
256
82.3
gli saga Wong et at, 2027]
X
38.0
76.. (78.6+)
SLDD-Model (Ours) with x = -y
76.7
80.3 (86.5*)
SLDD-Model (Ours)
50
72.8
78.3 (84.0%)
Table 2: Comparison with competitors on accuracy percent. For ease comparison, we evaluated
CUB-2011 I on Inception-và v3 (n f 1024) and ImageNet- IK with Resnet50 (n 2048) (*denotes Resnet50 accuracy). The dense Resnet50 achieves 80.9 % on ImageNet- IK. For glm-saga. we
selected the solution with maximum we 5. Arrows indicate generally preferable directions. For CBM joint, Resnet50 results were created as described in Appendix B.1.1
2348
50
2045
10
2005
17.
30
- 2349
-000
Dotac
Spare
Eact
Spane
Fact
Danc
Spase
Find.
Spasc
Flxt.
Danc
Spane
Find.
Spane
Flad.
Desic
Spanic
Fizes
Syenic
Frae
155
755
90.0
50.4
57.5
F8.1
65.2
89.3
MA.5
940
16.5
117
840
914
16.7
91.1
198
16.1
544
160
798
81.7
956
92.1
81.3
91.1
92.
101
119
152
811
15.3
2071
173
131
811
01.0
311
R71
311
153
82.
900
16.2
60.3
908
91.
Table 3: Impact of the loss function on accuracy in percent for Resnet50. Best results per column are in bold.
with diversity@ I 1] measure how different and pronounced the M a are localized. Since we focus on Tewe S 5, we set k = 5. We report the mean diversity @5 5 for all classes that use at least five features. Note that the proposed Laiv is a weighted version of diversity @nf-
4.2 RESULTS
We report the accuracy on the test set for the dense model after training the pretrained model on the training data with Twee ny, for the sparse model with Nue I 5, and for the result of our whole pipeline, the model with finetuned features, obtained by training the sparse model on the training data, and still Towe I 5. Every shown metric is the average over five (four for ImageNet- K) randomly seeded runs. The standard deviations are included in the appendix.
Table the competitive performance of our SL.DD-Model to the dense Resnet50. It is evident that an extreme sparsity of s = 2048 can be obtained in the final layer with just 0.1 to 0.4 percent points less accuracy. Additionally decreasing the number of features by 97.6 %. resulting in just 50 instead of the previous 2048 features, only reduces the accuracy compared to the dense model by 1.3 to 2.7 percent points. Finally, proposed Laiv improves the accuracy for all sparse models. Table shows the general applicability of our method with different backbones. However, we observed some instability and no increased accuracy when finetuning the DenseNet121 with n; - 2048, showing a positive effect of sharing features. Table 2 compares our approach to competitors: Without requiring additional supervision, we achieve competitive performance compared CBM (Koh et al.) 2020)-based methods, while achieving lower dimensionality and higher sparsity. Additionally, we improve the accuracy of glm-saga with heavily reduced n². For ImageNet-1K, we skipped the dense training and directly used the pretrained model. The good scalability of our proposed method to this large dataset with a higher number of classes displayed in Table 5 Table shows the diversity @ 5: With Laiv it is very close to the maximum value of 100 % in the dense case and still heavily increased in the sparse cases. This showcases that Laiv is suitable to ensure a diverse localization and improved interpretability of the used feature maps which is visualized in Figures to 17 Finally, the total number of features that is used by the unrestricted (n² = ny) models Table 3 is reduced from 912 to 719 for the models with Laiv. but still shows a high number class-specinc features. That Laiv leads more shared features supports our motivation of enforcing different features to capture different concepts.
CUB-2011
EGVC BOVC-Alknut
NARinns
Statfodd
x2 Sparse
aa =
n2
vf -
<] t0
50
Brekbsoe
Dense
Fise.
Spani
Fost.
Desse
Spurse
Faxt.
Spane
Fiect.
Donc
Spanic
Firet.
Spase
Fas.
Derac
Spase
Fisci.
Spenic
Ract.
Inoppion-ri
K2.3
1K.5
ML.5
340
78.3
10.9 80LK
874
75.6
72.
73.5
765
91.5
18.9
MLS
96.5
39.4
Reasssil
$4.0
WLS
81.7
$48
91A
96.7
91.1
9L1
94.4
KLB
840
798
KL.
936
92.1
91.1
92,0
Table 4: Accuracy in percent dependent on backbone. Best results per column are in bold.

---

Under review as a conference paper at ICLR 2023
Dense (1² 2048)
Sparse (n)
2048)
Finet. (n) 2048)
Sparse (n² 50)
Finet. (n² = 50)
80.9
62.2
76.7
44.8
72.8
Table 5: Accuracy in percent for Resnet50 on ImageNet- IK using the pretrained dense model.
CUB-2011
PGVC-Aircraf
Sunfond Cars
27.
2018
Cau
Dense
Spune
Finet.
Syume
Finet.
Dense
Sparse
Finet.
Sparse
Finet.
Dense
Syunse
Finet.
Spurie
Finet.
50.2
46.0
43.4
48.0
46.5
46.5
43.8
40.9
45.9
44.0
45.0
41.7
39.1
43.6
43.7
98.9
69.9
71.9
6522
72.6
98.8
85.7
86.6
69.3
73.9
99.2
72.4
74.6
63.7
74.8
Chang et
202O
52.5
50.1
FRI
Zringg dral
2020
51.1
44.1
46.3
48.2
Table 6: Impact of the loss function on diversity @5 in percent for Resnet50. Best results per column are in bold.
4.2.1 COMPARISON WITH OTHER LOss FUNCTIONS
We compare our diversity loss Laiv with the MCL (Chang et al. 2020) and the FRL (Zheng et al. 2020). The used hyperparameters for the loss functions are reported in Appendix 1...1 and we focussed on three datasets to save computational resources. Table 3 shows that our Laiy reaches the highest accuracy across all datasets. Notably, the accuracy reported in (Chang et al. 2020) for the MC-Loss is achieved by a two-layer MI.P plus additional techniques, whereas we only use one layer to ensure linearly separable representations for our SLDD-Model. Although it is expected that applying Laiv has a positive effect on diversity@* @5 due to the similar formulation, we could observe remarkable uplift in diversity @5 (Table 6 compared to MCL and FRL, which also optimize for differently localized features.
4.3 INTERPRETABILITY
In this section, we discuss the interpretability of the proposed SLDD-Model using example models. The interpretability of the proposed SLDD-Model is based upon using very few (Nuee) features from a small pool of n to make a decision. A low ri allows the analysis of the remaining features to try to align them with human understandable concept, which is discussed in Section 4.3.1 Since the sparse linear layer is casily interpretable, the complete model with sufficiently well understood features is both locally and globally interpretable.
For global interpretability. the final layer of the SLDDD-Model can be fully visualized and analyzed. Figure shows W Waprrs This allows the practitioner to verify the global behavior of the model. For example, the attribute aligned with thas-bil-shaape:needle has non-zero weight for all four classes that have the attribute in more than 30 % of examples. The visualization of the classes positively related to specific attribute and features related to a class like Figures 9 to 17 helps to trust the model. Additionally, Wssparse allows for further feature understanding, since it is possible to analyze the similarities between classes that share a feature. If
in the presented model in Section [33.]
50
100
150
parta cader Masa
cole iiteoent
sogost
*LL HHLLH noade ROSae
tea preee caler baxx
has breat conrrarrie
the theest cal pursee
hou So codee parple
1.5
Festures
11.0
0.5
20
0.2
30
0.0
40
Classes
50
Attributes
(a) Exemplary W The alignment of the fea- tures with attributes in CUB-201 is displayed in Figure 3b
(b) Relationship between chosen features and at- tributes (C > 20%) of CUB-200 2011 for the ex- emplary model. Higher values indicate that the feature describes the attribute.
Figure 3: Visualization of the sparse matrix and the feature alignment.
7

---

Under review as a conference paper at ICLR 2023
3.0 Aocece Pacditer
2,5 Mep Metergged
2:41 Orenn Wikier
A321
A363.50
4545
(a) Feature 45 with C = 0.39 for the attribute "has- bill-shapenneedle.. Higher activations are local- ized around the bill, and needle-like bill is visi- ble.
(b) Manually aligned feature of a model trained on FGVC-Aireraft without additional labels: The feature is manually aligned with four engine air- craft shown in Appendix
Figure 4: Example images and localization L, scaled to indicate feature activation, in ascending order for two models. The text between the rows describes the activation value for the image, which drops below 0 due to the normalization of gim-saga, and the class name.
the feature is aligned well, this leads to knowledge discovery.
The local interpretability describes the explanation of single decision made by the model. De- cisions with sparsely connected features are inherently locally interpretable, if the features can be interpreted and localized, as shown in Figure The practitioner understand where and what was found in the image, and due to the full global interpretability also understand the behavior around the current example.
4.3.1 FEATURE ALIGNMENT
In this section, demonstrate how the features of the proposed SLDD-Model can be aligned with interpretable concepts. We describe how one can use additional labels or expert knowledge to interpret the features and demonstrate that several learned sparse features directly connected attributes relevant to humans. Thus, our model learns such abstract concepts directly from the data. Overall, due the very limited number of used features ný, the features can and should be thoroughly analyzed and interpreted to facilitate interpretability. For feature localization, we follow a masking approach similar Fong & Vedaldi (2017), which described in Appendix
Alignment with Additional Data We use the attributes A contained in CUB-2011 to align the learned features with these labels after the finetuning. For each attribute a E and feature j we compute score Caj that corresponds to relative increase of the feature when the attribute is present:
Saj
8aj
ptrain
Σ iEPs
ptrain
Caj max Eetrain :. min Firainy
Pact
Pa-
The set of indices whose images contain the attribute is denoted by Po-. We considered attribute to be present the human annotated it with "probably" or "definitely". Annotations with "guessing" neither included in the positive (pa+) negative (Pa-) examples. For one exemplary model part of the matrix of C' values displayed in Figure It is clear, that some features correspond colors, some specific shapes like "bill-shape:needlee" and other features do correlate with specific attributes. Figure 4a visually validates the connection that was implied in Figure 3b
Manual Alignment To ensure that the entirety of feature understood, or in absence of additional data, the features have to be manually aligned for increased interpretability. This is enabled by the low number of features and sparsity. Some useful aspects for understanding a feature are the localization, extreme examples, feature visualization (Olah et al. 2017) or which classes use that feature. One such alignment for a model trained on FGVC-Airerafi is displayed in Appendix E and Figure 4b Aligning learned features with human understandable concepts is still challenging to multiple aspects and human understandable concepts do not need to be axis aligned (Szegedy et al. 2013). However, the low dimensionality of the remaining features allows for sophisticated analysis
single feature refer
8

---

Under review as a conference paper at ICLR 2023
90
92
Finetune Accuracy
Accuracy
90
85
=.50
88
2040
80
Luy
Stanford Cars
Ldy
2048
75
FGVD-Aircraft
0f 50
84
10a
10a
10a
20
Number of Features oj
Features per Class Dac
(a) Impact of changing ri with Tiwe 5
(b) Impact of changing Thee with rij 50 or 2048
Figure 5: Relationship between Finetune Accuracy and aspects related to interpretability for Resnet50.
of every feature in practice, which could even discover spurious correlations as features as done by the glm-saga publication.
4.4 INTERPRETABILITY TRADEOFF
In this section, we analyze the impact of changing ni and Then on the finetuning accuracy of the model trained with Ldiv, shown in Figure Figure (5a visualizes the impact of nf: With decreasing n the accuracy drops slowly until a dataset-specifie threshold reached, at which a steep decline starts. Additionally, the proposed Laiv works regardless of nz. Figure 5b shows the finetuning accuracy in relation to Nuee The accuracy is rather insensitive to Rawe only decreasing when Theee < 5. which the case for both dimensions and showcases that five features suffice for competitive model even if the features are shared among classes. Figure F demonstrates the tradeoff that our SLDD-Model offers: Both Twe and n can drastically reduced with either a negligible or small impact accuracy adapt the amount of interpretability needed.
5 LIMITATIONS AND FUTURE WORK
As shown in Figure 5 the SLDD-Model cannot get arbitrarily low-dimensional sparse with competitive accuracy via our proposed method. The optimal sparsity and dimensionality for a given problem are hard to predict and might require some experiments to determine the minimum values for competitive accuracy. Aligning all used features with human concepts is still difficult, albeit more feasible than without WIDDD-Modds Future work could use
interpretable feature extractor like B-cos Networks (Böhle et al. 2022) to alleviate that problem. seems promising to apply SLDD-Model other safety-critical domains, such medical, where expert can be utilized to align the features and follow the decision, as it can help bring the required interpretability and irustworthiness the domain. Embodied autonomous agents can also benefit from as the entire decision process can be thoroughly analyzed. While more interpretable models could be used to more deliberately bring harm, they can disclose existing problems with machine learning models and open up the opportunity to build fair and trustworthy models. Finally, sparsity and dimensionality could be part metrics used to quantify the trustworthiness of model.
CONCLUSION
In this work, we proposed the more interpretable sparse now-dimensional decision model (SLDD- Model) allow a human to follow and understand the decision of Deep Neural Network for image classification. Our proposed pipeline constructs SLDD-Model with drastically increased global and local interpretability while still showing competitive accuracy. As demonstrated, practitioner can manually configure the pipeline to set the tradeoff between accuracy and interpretability. Our novel loss increases the feature diversity and showed that identifying class with varied features can improve the accuracy. Finally, our SLDD-Men -odeel offers measurable aspects of interpretability. which allows future work to not just compare itself accuracy but also interpretabilityy

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY
For reproducibility, we uploaded the code for both the feature selection and the feature diversity loss Ldiv supplementary material. Additionally, clearly reported the used hyperparameters and data augmentation in the implementation details, Appendix B.
REFERENCES
Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6541 2017.
Adrien Bibal, Michael Lognoul, Alexandre Streel, and Benoît Frénay. Legal requirements on explainability in machine learning. Artificial Intelligence and Law, 29, 2021. doi: 10. 10.1007/ s10506-0200-99270-4
Moritz Böhle, Mario Fritz, and Bemt Schiele. B-cos networks: Aligmment is all we need for interpretability. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Dongliang Chang. Yifeng Ding, Jiyang Xie, Ayan Kumar Bhunia, Xiaoxu Li, Zhanyu Ma, Ming Wu, Jun Guo, and Yi-Zhe Song. The Devil the Channels: Mutual-Channel Loss for Fine-Grained Image Classification. IEEE Transactions on Image Processing. 29:4683 4695, 2020. ISSN 1057-7149. doi: 10. 21100//i.20220.978338
Dongliang Chang. Kaiyue Pang, Yixiao Zheng, Zhanyu Ma, Yi-Zhe Song, and Jun Guo. Your "Flamingo" is My "Bird": Fine-Grained, or Not. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 00:1 10o:1141-111 (44441114444 11480, 2021. doi: 10. 00.1109/cvpr44447.2221.. 200.:446.67...221111111
Tianshui Chen, Wenxi Wu, Yuefang Gao, Le Dong, Xiaonan Luo, and Liang Lin. Fine-grained repre- sentation learning and recognition by exploiting hierarchical semantic embedding. In Proceedings of the 26th ACM international conference on Multimedia. pp. 2023-2022220222 2031, 2018.
Po- Yung Chou, Cheng-Hung Lin, and Wen-Chung Kao. A Novel Plug-in Module for Fine-Grained Visual Classification. arXiv, 2022.
Logan Engstrom, Andrew Ilyas, Hadi Salman, Shibani Santurkar, and Dimitris Tsipras. Robustness (python library), 2019.
com Ma MadryLab/ robustness
Ruth C Fong and Andrea Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. In Proceedings of the IEEE international conference on computer vision, pp. 3429-3437, 2017.
Sorelle A Friedler, Chitradeep Dutta Roy, Carlos Scheidegger, and Dylan Slack. Assessing the local interpretability of machine learning models. 2019.
Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1):1, 2010.
Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks. arXiv 2019.
preprint arXiv:/902.098:
Nidham Gazagnadou, Robert Gower, and Joseph Salmon. Optimal mini-batch and step sizes for saga. In International conference on machine learning, pp. 2142-2150. PMLR, 2019.
Ian Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning, pp. 1319-1327. PMLR, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks. arXiv preprint MrXiv:2105.02968, 2021.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Sheikh Rabiul Islam, William Eberle, and Sheikh K Ghafoor. Towards quantification of explainability in explainable artificial intelligence methods. In The thirty-third international flairs conference, 2020.
Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Alek- sander Madry. Missingness bias in model debugging. In International Conference on Learning Representations, 2022. URL https://openrevv//
net www.:/ee//ooiiteeooooooo Tett:///ooouum..dteetttttaa Te5ytkqsn1
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Sunnie SY Kim, Nicole Meister, Vikram V Ramaswamy, Ruth Fong, and Olga Russakovsky. Hive: evaluating the human interpretability of visual explanations. arXiv preprint arXiv:2112.03184, 2021.
Picter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T Schütt, Sven Dähne, Dumitru Erhan, and Been Kim. The (un) reliability of saliency methods. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pp. 267-280. Springer, 2019.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pp. 5338-5344 PMLR, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. object representations for fine- fine-grainer -raiied categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), 13), Sydney, Australia, 2013.
Haoyu Liang, Zhihao Ouyang, Yuyuan Zeng, Hang Su, Zihao He, Shu-Tao Xia, Jun Zhu, and Bo Zhang. Training interpretable convolutional neural networks by differentiating class-specific filters. In European Conference on Computer Vision, pp.
Springer, 2020.
Tsung-Yo Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for fine-grained visual recognition. In Proceedings of the IEEE international conference computer vision, pp. 1449-1457, 2015.
S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine- Fine-grained visual classification of aircraft. Technical report, 2013.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04255 2021.
Thomas McGrath, Andrei Kapishníkov. Nenad Tomašev, Adam Pearce, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of Chess Knowledge in AlphaZero. arXiv, 2021.
George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
Christoph Molnar. Interpretable machine learning. Lulu. com, 2020.
Meike Nauta, Ron Bree, and Christin Seifert. Neural prototype trees for interpretable fin-grainee image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19933-14943 2021.
11

---

Under review as a conference paper at ICLR 2023
Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2(11))e7, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 5024-8035. Curran Associates, Inc., 2019. U neur http://ppperrs..mri.ss.ooo
9015 pytorch-taastt. annechaa---ae iewan-iimppattiii style wwwhhhhggghhhhce performacee deepeeeedddee-eeeeeene pdf
library.
Vikram V Ramaswamy, Sunnie S Y Kim, Nicole Meister, Ruth Fong, and Olga Russakovsky. ELUDE: Generating interpretable explanations via decomposition into labelled and unlabelled features. arXiv, 2022. Decomposition in unlabelled / labelled features. Sparse explanation, not prediction as uses ground truth attribute labels for explanation. Does not have to be faithful with actually done prediction.
Pratyush Rokade and BKSP Kumar Raju Alluri. Towards quantification of explainability algorithms. In 2021 The 5th International Conference on Advances in Artificial Intelligence (ICAAI), pp. 31-37, 2021.
Stefan Rüping et al. Learning interpretable models. 2006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.1007/s1 12000//11263--05-08116666
Yoshihide Sawada and Keigo Nakamura. Concept Bottleneck Model With Additional Unsupervised Concepts. IEEE Access, 10:41758-41765 2022. ISSN 2169-3536. doi: 10. 10.1 3167702.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 218-626, 2017.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 1-18262, 2016.
Hong Tao, Chenping Hou, Feiping Nie, Yuanyuan Jiao, and Dongyum Yi. Effective discriminative feature selection with nontrivial solution. IEEE transactions on neural networks and learning systems, 27(4):796-80) 2015.
Kale-ab Tessera, Sara Hooker, and Benjamin Rosman. Keep the gradients flowing: Using gradient flow to study sparse network optimization. arXiv preprint orXhv:2102.01670, 2021.
Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie, Building a bird recognition app and large scale dataset with citizen scientists: The fine print fine-grained dataset collection. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 995-604 2015. doi: 10. 11109//VVR.201..7299866
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd
birds-200-2011 dataset. 2011.
Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pp. 11205-1 1216. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Hongyu Yang, Cynthia Rudin, and Margo Seltzer. Scalable bayesian rule lists. In International conference on machine learning, pp. 3921-3930. PMLR, 2017.
Peiyu Yang, Zeyi Wen, and Ajmal Mian. Multi-grained interpretable network for image recognition. In 26TH International Conference on Pattern Recognition, 2022.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIR*2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2022. UtL.htt.s::/ppeeeeiiii httt:/ooroommmmllaMMMa.....
Matthew Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference computer vision, pp. 818 903. Springer, 2014.
Runkai Zheng, Zhijia Yu, Yinqi Zhang, Chris Ding, Hei Victor Cheng, and Li Liu. Learning Class Unique Features in Fine-Grained Visual Classification. arXiv, 2020.
Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series (Statistical Methodology), 67(2):301-320, 2005, ISSN
1467-9868. doi: 10. 11111//.467799668
13

---

Under review as a conference paper at ICLR 2023
CUII-5011
IGVC-Aiinart
NAllink
Statfed Cars
2145
2048
50
2045
50
2348
50
Cav
Desse
Spaci
Fiact.
Spess
Faxt.
Dessc
Spane
Flast.
Spase
Flatt
Desse
Spane
Fixet.
Spusic
Flac.
Denoc
Spane
Firet.
Spacic
Flast.
$1.5
85.3
73.5
13.4
15.4
914
813
16.1
562
79,5
55.3
17.3
92.6
85.3
91.3
10.
10.3
50.
30.
30.
103
304
163
161
10.3
101
-03
100
19.1
36.3
361
#02
262
24L
20.2
2033
202
203
303
20.
201
362
282
20.0
692
201
#02
203
20.1
#61
100
Table 7: Accuracy in percent dependent on Laiv for Resnet50. Best results per column are bold and + indicates the standard deviation across five runs.
50 Firet
(A) Tlast
50
52
Bacthone
Dinoe
Spuro
Spuris
Spures
Piner
Spani
Thrrz
Spars
First
Squrid
Plast
Derns
Spanz
Flart
Sperio
Prrv
963
76,2
$2.9
15.1
10.1
91.5
$5.2
89-5
23.0
$4.1
72.8
64.6
T1
98.3
$7.5
91.3
$5.8
01.4
Incqpion-r]
281
12.3
25.3
753)
2414
91.3
203
14.
-384
19.3
28.1
28.9
222
$7.5
382
S8L1
339
2413
2012
974
73.
77.3
382
73.1
2811
76,3
2011
91.
282
58.9
382
20.4
Reows50
10.2
1611
91.5
115
333
140
91.1
182
146-
918
003
91.
162
81.7
1811
102
183
102
92m
Table 8: Accuracy in percent dependent backbone. Best results per column are in bold and + indicates the standard deviation across five runs.
A APPENDIX
In this appendix, provide implementation details and standard deviations for the experiments. Additionally. the pseudocode for gim-saga (Wong et al. (027) is shown. Finally, we present the feature visualization technique and more ablations Ldiv-
B DETAILED RESULTS
The full results of Section 4.2 with the standard deviations are presented in Tables 7 to 13 The reported standard deviations are, except for DenseNet121 on NABirds, as mentioned Section generally rather small compared to the differences in means, which supports conclusions. We also show exemplary images with the 5 most important features for the dense and sparse conventional model in Figures 9 to 17 The comparison with the finetuned SL.DD-Model shows improved localization and interpretability of the features. The finetuned -LDDD-Model in Figure TO seems to use features that each individually focus more on chest, lower belly, head. bill or crown, whereas for the dense and sparse models the different features focus on the same regions. This increased diversity @ and with interpretability was also measured in Section 4.2
4.2
B.1 IMPLEMENTATION DETAILS
We use Pytorch (Paszke et al. 2019) to implement our methods and on ImageNet pretrained models as backbone feature extractor. We utilized glm-saga and robustness (Engstrom et al. 2019). The images resized to 448 x 448 (299 299 for Inception-v3, 224 224 for ImageNet-1KK) normalized, randomly borizontally flipped and jitter is applied. The model is finetuned using stochastic gradient descent on the specific dataset for 150 (100 for NABirds) epochs with batch size of 16 (64 for ImageNet-IKG I), starting with 5 10 3 as learning rate for the pretrained layer and 0.01 for the final linear layer. Both get multiplied by 0.4 every 30 epochs. Additionally, we used momentum of 0.9, fg-reuularization of 5. 10 and apply a dropout rate of 0.2 on the features to reduce dependencies.
ImageNet-IKK
2048 n Sparse
n 50 Finet.
Dense
Finet.
Sparse
62.2
76.7
44.8
72.8
80.9
+0.0
+0.0
+0.1
+0.1
Table Accuracy in percent for Resnet50 on ImageNet- KKusing the pretrained dense model.. Best results per column are in bold and + indicates the standard deviation across four runs.
14

---

Under review as a conference paper at ICLR 2023
Algorithm Pseudocode from gim-saga (Wong et al. 2027)
1: Initialize table of scalars df 0 for i [n]]
2: Initialize average gradient table Javg and Joavg 0
3: for minibatch B [n]] do
4: for E do
5: a; = B + 30- 30-8 y/
6: = 2; // calculate new gradient information
7:
9i d: z, // calculate stored gradient information
8:
end for
9:
g [ Σ
iEB
9i
10:
g [R] Σ LEB g
11:
= ΣiEB
12:
96 = . EieBo
13:
8 8 7g 9 gavg)
3o 3. - 790 - 90 g0oog)
15:
B Prox 71111- (
16:
for E do
17:
af a, // update table
18:
9aog = Javg +
(g g)) update average
19:
gaang goorg
B
n [go 96
20:
end for
end for
was set to 0.196 for Resnet50, 0.098 for DenseNet121 and 0.049 for Inception-v3. For the feature selection, we set a = 0.8 and reduce the regularization strength by 90 % as we found sped up the process without decreasing performance.
We use gim-saga to compute the regularization path with a = 0.99 and all other parameters set to default with lookbehind of T 5. From this path, the solution with maximum Rww 10 is selected. Then the non-zere entries with the lowest absolute value get zeroed out until we are left with Twwe as we empirically found that they do not improve test accuracy after finetuning.
This selected solution replaces the final layer of our model. Then we train for 40 epochs, starting with the final learning rate of the initial training multiplied by 100 1oo of that for ImageNet- IK), and decrease it by 60 % every 10 epochs. Dropout on the features was set to 0.1 and momentum was increased to 0.95. Note that, while the increased momentum has been important for the stability of the final training, the hyperparameters were not thoroughly optimized for the sparse case.
B.1.1 COMPETITORS
For creating the accuracy for Resnet50 and CBM (ohh et al. 2020) joint Table 2 we resized the images 448 448 and used batch size of 16. The remaining used hyperparameters were almost identical to the CBM experiments with n33 but we only trained for up to 400 epochs, as 650 led to decreased accuracy (-0.8 percent points). Additionally, the learning rate not decayed, mirroring the published code. The reported accuracy stems from three runs with a standard deviation of 0.7.
For MCL, we used the reported hyperparameters of = 0.005 and À = 10. For finetuning, we assigned every feature every class that was using We optimized the hyperparameters for FRL based on accuracy, leading to K = 10 and X = 0.01.
C Glm-saga
15

---

Under review as a conference paper at ICLR 2023
CUII-5011
IGVC-Aiinart
NAllink:
Stnffed Cars
2145
2048
50
2045
50
2348
50
Cav
Desse
Spacie
Flact.
Spess
Faxt.
Desse
Spane
Flast.
Spass
Flatt
Desse
Spane
Fixet.
Spusi
Flac.
Denoc
Spane
Firet.
Spazic
Flast.
562
460)
<3.4
45.0
46.3
45.8
459
440
41,3
364
34.9
35.1
45.9
413
2111
43.
-3.3
10.3
.0.3
10.3
105
307
104
201
106
163
181
101
-06
163
19.1
306
16.1
#01
#65
2044
203
202
315
#14
508
+13
10.1
2111
612
110
200
830
81.
513
203
Table 10: diversity@5 in percent dependent on Laiv for Resnet50. Best results per column are in bold and + indicates the standard deviation across five runs.
This section includes the Pseudocode for glm-saga (Wonn et al. 2027 in algorithm 1 operator , (B) is defined as:
The proximal
A1
A i otherwise
if
A2
8+X1
(8)
D VISUALIZATION OF FEATURES
For feature visualization, we follow masking approach. We systematically blur, following (Fong & Vedaldi 2017), one patch of size of the image and measure the difference in feature activation between the augmented image and not augmented image. The actual localization map Lp " for that square size is computed by
Pary ReLU(((() - p(ppa))
where Lpzy indicates the image where a p-sized patch starting at position (I*p, y*p) is blurred and the ReLU suppresses parts that increased the feature activation, since blur should not injecting a feature. The final localization map is the combination of different square sizes p E {28, 56, 64, 112, 224) to accommodate for differently sized features:
L
L -=Σ
(10)
max(Lp)
Notably, Lp has to be resized according to the smallest and we only show L' for one feature
E FEATURE ALIGNMENT
In this section, we use the alignment of the shown feature in Figure 4b with four-engine aircraft to exemplary show how one can align feature manually. We first visualize the distribution of the feature over the training data in Figure 6 This indicates more binary attribute and by investigating the images and saliency maps, observed an alignment with four-engine aircraft. We the hypothesis by filtering for the classes "A340-500" and "BAE 146-300", for which the feature corresponds to the four engines. Figure shows that the lowest activating examples of this group not clearly show the four-engines which supports our hypothesis. Note that one feature can correspond to multiple concepts, as another hypothesis is an alignment with the propeller. Whether all correlating concepts have to be understood or how exact the analysis has to be depends on the application.
F ABLATIONS ON FEATURE DIVERSITY Loss
In this section, we present an additional analysis of the factors in Laiv and the impact of B.
F.1 FACTOR IMPORTANCE
We analyzed the impact of the two factors in Equation 3 with for accuracy optimized B, shown in Tables [4] and 15 The label w/o Class-Specify indicates not using the weights of the predicted class
and w/o Rescaling refers to not maintaining their relative mean. We used 3 = 0.001 196 as 1e-5 for w/o Class- Class-Specific, since
2048
the size of the feature maps with 196 wM has and the number
16

---

Under review as a conference paper at ICLR 2023
S0 Fiect.
50 Flas.
-0
Beckbooe
Dense
Spase
Flast
Spuno
Fixe.
Dasic
Spease
Faxtt.
Spani
Donc
Spens
Firet.
Spase
Detse
Spanc
Fiact.
Spenic
Fixt.
DessoNei221
06.5
603
16.3
71.2
42.4
3
$7.6
686
153
47.3
631
662
98.7
47.4
96.3
71.8
(8.1
963
76.9
65.
59.
52.9
15.1
83.3
72.2
380
36
8L.8
41-
410
42.
456
920
763
683
32.0
20.7
103
1.00
1864
-0.
4.84
482
A14
8.11
-03
618
-10
603
43.6
110
103
186
1.06
41.
105
Revad50
963
101
600
46.5
71.
14.4
23.4
692
12.6
-0.
23.3
986
41144
60.3
1403
-13
739
201
98.
-11
80.
-1.
418
76.7
853
540
101
962
330
724
03.5
413
404
7436
Table 11: diversity @5 in percent dependent on backbone. Best results per column are in bold and + indicates the standard deviation across five runs,
CUB-2011
PGVC-Aireaat
Stanford Cars
2018
n] Spunie
2018
n] Spunio
2018
n]
Eaiv
Dhess
Sparie
Fincs.
Fincs.
Diese
Sparie
Fineet
Fines.
Diesse
Sparse
Firet.
Sparie
Finet.
79.5
10.2
MCL
Chang al. 2020
[5.]
20.3
$1.9
202
$5.1
79.4
20.1
$2.8
90.3
20.2
53.4
102
59.0
57.2
20.3
38L1
05.]
91.0
10.2
10.2
92.5
10.3
59.0
90.7
FRI
7heng et al
2020
-0.2
602
20.
789
20.5
3:.1
20.2
68.5
10.4
10.
87.5
202
58.
30.1
10.3
10.2
102
894
102
90.9
Table 12: Accuracy in percent for Resnet50 compared to other loss functions. Best results per column are in bold and + indicates the standard deviation across five runs.
CUB-2011
RGVC-Airctat
Sanford Cars
2048
2048
2048
Law
Desse
Sparie
Sparie
Finct.
Desse
Sparie
Sparic
Finct.
Sparsc
Finet.
Sparie
x
#9.2
90.2
:022
46.0
=1.33
43.4
20.5
48.0
20.3
46.5
+:9.5
46.5
=40.7
43.8
-40.4
40.9
45.9
3110
31.6
44.0
19.3
45.0
440.3
41.7
-40.1
39.1
43.6
20.4
43.7
M6.9
69.9
71.9
652
72.6
98.8
85.7
56.6
69.3
73M
99.3
72,4
74.6
63.7
748
+9.1
-0.5
20.4
114
20.3
+...
11.5
11.
303
11.3
19.1
+2.0
11.5
11.3
2099
MCT.
Chang
2020
FRI
Zheng
2020
Table 13: diversity @5 in percent for Resnet50 compared to other loss functions. Best results per column are in bold and + indicates the standard deviation across five runs.
CUB 2011
FGVC-Airvaft
Stanfond Cars
2048
nj
50 Finet.
2018
2018
50
Doase
Spaase
Sparie
Dense
Sparse
Finet.
Spuaee
Finet.
Dense
Spunc
Finet.
Sparse
Eair
86.6
$4,0
1:02
81.7
10.3
$4.9
91.4
-:0.3
-0.2
91.1
1:011
90.1
.00.2
93.6
1:0.1
93.3
=0.1
91.1
Rescaling
=0.2
86.3
R2.7
+0.5
83.
00.2
90.1
10.3
91.2
89.0
Class Specitic
-0.2
10.2
$2.0
79.0
100
102
83.1
10.4
10.4
10.2
93.2
92.6
102
-0.3
89.1
Table 14: Impact of factors in Laiv on accuracy in percent for Resnet50. Best results per column are in bold and + indicates the standard deviation across five runs.
CUB-2011
PGVVC-Airenft
Stanford Cars
2048
nf
50
2048
500
R?
2018
50
Sparse
Finer.
Spurie
Finct.
Dense
Syurse
Finet.
Spuaie
Dense
Spune
Finct.
Sparsc
Firet.
Caiee
=0.1
98.9
#0.5
60.9
+:0.
71.9
+:14
65.2
+1.3
72.6
+:0.2
98.8
1:133
A5.7
11..4
$6.6
#0.8
69.3
+:13
73.9
+0.11
99.2
+2.0
72.4
1:1.5
74.6
=1.3
63.7
#6.9
74.8
weo Resculing
-0.0
10.9
67.6
12.0
70.5
11.3
73.3
99.7
72.4
112
11.65
74.6
12.1
73.6
10.0
99.7
40.8
100
65.3
-3.6
68.3
42.0
Spass-Sprciffe
46.8
43.7
45.5
39.5
=1.1
Table 15: Impact of factors in Laiv on diversity @5 in percent for Resnet50. Best results per column are in bold and + indicates the standard deviation across five runs.
17

---

Under review as a conference paper at ICLR 2023
10
Least acliveting
-3.10
-2.96
-2.74
-2.39
-1.84
-1.02
A321
757-300
767-300
F/A-18
A330-300
DC-10
6.29
9.87 A340-500
10.94 BAE 146-300
11.80 DHC-8-300
12.75 BAE 146-300
13.86
A360-500
148:300
Figure 6: Top: Distribution of feature activation over the training data. Bottom: Different examples of this distribution with their scaled feature localization. The activation and class name given above the image.
of features of the baseline model nı 2048 as scaling factors in order to be less dependent of model architecture and image size, and 0.1 for w/o Rescaling. Only the combination of both factors leads to an improved accuracy, validating our idea that it is important to only enforce diversity of features that are found the input and used in conjunction.
F.2 Loss WEIGHTING
This section is concerned with the impact of the weighting factor 3 for the feature diversity loss Laiv- For our proposed method, we use 3 = 0.196. Tables [6] and [7 show that Laiv improves the diversity 5 and accuracy across all datasets in the sparse with increasing / up to maximum
18

---

Under review as a conference paper at ICLR 2023
DRA
18 12
Figure 7: The three least activating training examples of classes " "A340-500" and "BAE-16-300" for the given feature.
CUB-2011
FGVC-Aireaat
Stanford Cars
2044
R] Sparse
2048
np. Spanse
Dense
Sparie
Finet.
Finet.
Dense
Sparse
Finet.
Finet.
Dense
Spance
Finet.
Sparse
Finet.
36.6
$1.8
85.3
79.5
N3.4
90.0
88.4
39.4
87.3
85.1
92.2
90.9
92.6
39.3
91.1
+0.4
-0.3
-02
+0.3
+:0.2
+0.3
+0.3
+0.2
-0.4
+0.3
+0.1
-0.2
+0.1
+0.3
#0.1
0.0196
1.0.2
86.4
2.0
82.2
-03
85.3
10.6
79.4
1:0.3
83.2
1:04
90.6
1:03
89.0
10.3
39.7
204
87.4
10.4
85.5
20.1
93.1
200
91.0
1:02
92.6
1:02
39.0
10.2
90.9
+0.1
+0.2
Table 16: Accuracy in percent dependent on 3 for Resnet50, Best results per column are in bold and + indicates the standard deviation across five runs. Our used is underlined.
roughly around B = 1. Setting the value higher leads to Laiv dominating the training. To ensure that the network is still mainly optimized for classification, we choose 0.196, even though some cases could still observe slight gain in accuracy with slight increase of 3. The positive relation between diversity@* @5 and accuracy, visualized in Figure supports our approach enforcing varied features for the extremely sparse case.
CUB-2011
FGVC
Stanford Cars
n2
2018
n
n]
8
Dease
Spars:
Finct.
Sparse
Finct.
Dense
Sparse
Fisct.
Sparse
Finct.
Dense
Sparse
Finct.
Sparsc
Finct.
50.2
46.11
43.4
48.0
46.5
46.5
43.8
411.9
45.9
44.0
45.0l
41.7
39.1
43.6
43.7
:0.2
=0.2
=0.3
#0.5
111.3
+:0.5
+:0.7
::0,4
20.6
30.6
20.2
20.3
20.11
10.55
104
0.00196
49.3
43.8
0.0196
10.6
64.0
10.6
50.8
-0.4
47.8
11.0
50.3
48.6
10.9
10.55
75.4
10.77
54.4
10.6
50.2
11.9
50.2
48.5
11.0
10.7
66.1
10.6
48.7
45.2
104
12.0
46/6
11.3
46.0
739
Table 17: diversity @5 in percent dependent on 3 for Resnet50, Best results per column are bold and + indicates the standard deviation across five runs. Our used /3 is underlined.
19

---

Under review as a conference paper at ICLR 2023
91.0
Finetune Accuracy [%]
90.5
90.0
1001
89.5
B
89.0
88.5
10022
88.0
B 0.196
87.5
50
60
70
80
Finetune diversity@5
Figure 8: Relationship between finetuned diversity @5 and accuracy for varying 3 for Resnet50, portrayed via color, on FGVC-Aircraft, Each dot represents an increase by factor of /10 and the standard deviation is indicated by the shaded area. 1.96 Laiv is not shown, it dominates the training.
C30 maga
222
R-0220
(a) Conventional Dense Model
(b) Sparse with n² 71
(c) Finetuned Model
Figure 9: Feature maps of the top 5 features by magnitude for class Scarlet Tanager on example images. The used weights for the respective features also displayed.
20

---

Under review as a conference paper at ICLR 2023
6-00
R-079
(a) Conventional Dense Model
(b) Sparse with nj n
Finetuned SLDD-Model
Figure 10: Feature maps of the top 5 features by magnitude for class White Crowned Sparrow on example images. The used weights for the respective features are also displayed.
200
4-002
(a) Conventional Dense Model
Sparse with n; m
Finetuned MldDl-Model
Figure 11: Feature maps of the top 5 features by magnitude for class Brown Creeper on example images. The used weights for the respective features are also displayed.
21

---

Under review as a conference paper at ICLR 2023
ClAb mope
(a) Conventional Dense Model
(b) Sparse with ni f
(c) Finetuned SLDD-MMdel
Figure 12: Feature maps of the top 5 features by magnitude for class Cape Glossy Starling on example images. The used weights for the respective features are also displayed.
4.22
Conventional Dense Model
Sparse with n² n
Finetuned SL.DD-Model
Figure 13: Feature maps of the top 5 features by magnitude for class Ford Freestar Minivan on example images. The used weights for the respective features are also displayed.
22

---

Under review as a conference paper at ICLR 2023
(a) Conventional Dense Model
(b) Sparse with nj n
Finetuned SLDD-Model
Figure 14: Feature maps of the top 5 features by magnitude for class Audi S5 Coupe 2012 on example images. The used weights for the respective features are also displayed.
300
Conventional Dense Model
Sparse with n; n
Finetuned MLdDl-Model
Figure 15: Feature maps of the top 5 features by magnitude for class Ford Ranger SubCab201 on example images. The used weights for the respective features are also displayed.
23

---

Under review as a conference paper at ICLR 2023
(a) Conventional Dense Model
(b) Sparse with nj n
(c) Finetuned SLDD-Model
Figure 16: Feature maps of the top 5 features by magnitude for class Tornado on example images. The used weights for the respective features are also displayed.
Conventional Dense Model
Sparse with ni n
Finetuned SldDD-Model
Figure 17: Feature maps of the top 5 features by magnitude for class Fokker 100 on example images. The used weights for the respective features are also displayed.
24


Reviews:
[{"review_id": "qnZpUic0jS", "reviewer": "Reviewer_iicb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a training strategy by applying sparsification to obtain a more interpretable network. The training loss combines the feature diversity loss and sparsification loss used in Glm-saga. The measure of feature diversity is proposed and the paper shows that classes with the diverse feature can improve the accuracy. ", "summary_of_the_review": "This paper targets an interesting problem. However, the contribution of the paper is not clear. Before having a better understanding of the challenges that this paper resolves, I tend to vote for a reject recommendation. ", "strength_and_weaknesses": "\nStrength:\n\nRegularizing the sparse feature for a more interpretable network is a doable method. \n\nWeakness:\n\n1. The paper is not easy to follow with missing details and an unorganized structure. For example, $\\lambda_1, \\lambda_2$ is not explained in Section 3.2.2. In Section 3.2.1, $L_{final}$  is defined as diversity loss plus cross-entropy, while in Section 3.3 $\\mathcal{L}_{target}$ is cross-entropy alone. \n\n2. The main contribution of the paper is not clear. For example, the claimed contribution includes:\n\na). Feature diversity loss: Such loss is widely explored in the existing literature like MCL, FRL, CCMP. It is unclear why the proposed $\\mathcal{L}_{div}$ is better. \n\nb). Feature sparsification loss: It is a direct follow-up of glm-saga. And the sparsification in the final layer leading to better interpretability was also investigated. \n\nc). Measure of interpretability: The measurement in this paper is coarse by counting the dimension of the feature. A low dimensionality does not necessarily lead to better interpretability. For example, one can inject an identity matrix after the final FC layer. The identity matrix is fully sparsed, but meaningless. Therefore, a better measurement would be the one-to-one correspondence between a concept and a single feature vector. Such measurement is already well-explored in Net-dissection (Bau et al., 2017).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GK6xjJ6_7Z", "reviewer": "Reviewer_Vk6D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed model that makes predictions from a sparse set of features. To do so, it employs a novel diversity-based loss and a feature selection process. The model is shown to achieve competitive performance from only very few features (e.g. 5). Empirical analysis on the learned features is able to map them to human-understandable semantic concepts. ", "summary_of_the_review": "Overall, I think this paper is on track for making a good contribution, but the interpretability analyses feel a bit thin. While I would empathize if the authors feel that they may not be necessary because the model is \"self-explaining\", I would push back with [3], which argues to the contrary. \n\n[3] https://aclanthology.org/2020.acl-main.386/", "strength_and_weaknesses": "Strengths: \n\n1. The paper differs from most vision interpretability papers in proposing a self-explaining model rather than focusing on post hoc explanations. \n2. This paper proposed a very sparse model that achieves very good performance, using a novel loss formulation. \n\nWeaknesses: \n\n1. My key concern is the analysis of the learned features. First, the global analysis seems to require dense attribute-level annotations, such as what's found in the CUB dataset. However, these attributes are expensive to collect and not often provided.\n2. A hidden assumption behind the above analysis is that the model does use these semantic attributes for decision making. This seems to be a strong assumption, as various works [e.g. 1, 2] found that neural networks can use various spurious correlations and shortcuts, and furthermore many popular explanation methods are not well-equipped to detect such usage. I would appreciate any discussion on this aspect. \n3. The manual feature alignment on the four-engine attribute seems a bit too ad hoc. There could be many different explanations, such as the color or style of the plane, that result in the heatmap focus. Such statements should be made more quantitative, and even better, falsifiable (e.g. does removing two engines affect the heatmap?). \n4. The description on the diversity loss function moves quite fast, especially for readers not familiar with MCL and FRL. The same goes with the feature selection, which assumes prior knowledge of glm-saga. \n\n[1] https://arxiv.org/abs/2004.07780\n\n[2] https://arxiv.org/abs/2104.14403", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0ILg_c2AuT", "reviewer": "Reviewer_9CdM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to train a sparse neural network that only utilizes a small portion of all possible input features per class. The paper argues that due to a small number of input features, they are more interpretable. Another main highlight is that despite using only 10% of the features, the accuracy of the tested network remains relatively stable (97-100%). The paper shows that the small number of features that the model uses to make decisions can also be easily aligned with human-nameable concepts for additional explainability. ", "summary_of_the_review": "While it was interesting to read, I found myself having critical questions about the main motivations of the paper by the end. The paper hinges on two main things: A network that is \"inherently interpretable\" is better than interpretability tools built on top of traditional networks. And the second is that fewer features lead to better interpretability. As mentioned above in W1, I am not sure about either of those claims and the paper does not show enough proof that it is true. So, despite interesting results (only using a few features gives good accuracy), this is a sub-par paper for me looking at it primarily from an interpretability angle. The paper isn't really written from a feature-compressive perspective, So, I am also unable to comment on that. ", "strength_and_weaknesses": "**Strengths:**\n\n*[S1]:* Surprisingly effective results: The main highlight of the paper for me is the fact that despite using very few features, the proposed method can achieve such good accuracy results. \n\n*[S2]:* Clearly presented: The paper is well-written and easy to follow. The experiments are described clearly and the main techniques used are adequately clear.\n\n\n**Weaknesses:**\n\n*[W1]:* Unclear why the proposed method is better for anything: The paper does a poor job of motivating exactly why a model using few features is more interpretable. I attempt to understand it from several angles, but do not find a satisfactory answer. \n\n- Possibility 1: Since there are few features, they can be visualized easier: I'm not satisfied because now we are talking about the efficacy of visualization techniques, many of which can map complex decisions to simple-to-understand visualizations (Grad-CAM, etc..)\n- Possibility 2: Fewer features can be easily aligned with the human-nameable concept: I do not understand why the analysis in Fig 3b and Fig 4 can also not be done for traditional networks. In the end, we would only be left with features that correlated with the concepts anyways; Why does it matter whether we run them on 5 or 50 features to start with? If more than one feature correlates with a single concept, they can then simply be visualized as a \"group\". \n- Possibility 3: Fewer features can show us what was really important: There are a lot of ways to rank the \"importance\" of features in neural networks. Why isn't ranking the features by feature importance enough? Why do we need to train a network to only make decisions using these features?\n\nFor all of those, there is a direct actionable item for the authors to show; i.e., show us the results of the alternatives and how the proposed method fares better. \n\n*[W2]:* Limited Novelty: The proposed work is a fairly straightforward adoption of glm-saga; While this would absolutely not be a problem on its own, the fact that there is also not sufficient analysis (W1 above) makes it harder to overlook the fact that the paper does not present \"previously unknown\" technique that others might use differently. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject