Enhancing Pre-trained Representation Classifiability can Boost its Interpretability
Authors: Shufan Shen
Zhaobo Qi
Junshu Sun
Qingming Huang
Qi Tian
Shuhui Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Representation interpretability
vision representations
image understanding
Primary area: interpretability and explainable AI
TL;DR: We find a positive correlation between representation interpretability and classifiability.

Abstract: The visual representation of a pre-trained model prioritizes the classifiability on downstream tasks, while the widespread applications for pre-trained visual models have posed new requirements for representation interpretability. However, it remains unclear whether the pre-trained representations can achieve high interpretability and classifiability simultaneously. To answer this question, we quantify the representation interpretability by leveraging its correlation with the ratio of interpretable semantics within the representations. Given the pre-trained representations, only the interpretable semantics can be captured by interpretations, whereas the uninterpretable part leads to information loss. Based on this fact, we propose the Inherent Interpretability Score (IIS) that evaluates the information loss, measures the ratio of interpretable semantics, and quantifies the representation interpretability. In the evaluation of the representation interpretability with different classifiability, we surprisingly discover that the interpretability and classifiability are positively correlated, i.e., representations with higher classifiability provide more interpretable semantics that can be captured in the interpretations. This observation further supports two benefits to the pre-trained representations. First, the classifiability of representations can be further improved by fine-tuning with interpretability maximization. Second, with the classifiability improvement for the representations, we obtain predictions based on their interpretations with less accuracy degradation. The discovered positive correlation and corresponding applications show that practitioners can unify the improvements in interpretability and classifiability for pre-trained vision models. Codes are available at https://github.com/ssfgunner/IIS.

Published as a conference paper at ICLR 2025
ENHANCING PRE-TRAINED REPRESENTATION CLAS- SIFIABILITY CAN BOOST ITS INTERPRETABILITY
Shufan Shen , Zhaobo Junshu Sun 1.2 Qingming Huang! 12.4, Qi Tian", Shihui Wang
1 Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS
2 University of Chinese Academy of Sciences Harbin Institute of Technology, Weihai
4 Peng Cheng Laboratory Huawei Inc.
{ tshenshufan222, sunjunshu21s, vangshuhuij@ict. www.feeict......mmm.mmm cn qi zibbhiit... edu. gmhuang@ucas.cm ac. tian. qi wwwwwwwwww...... com
ABSTRACT
The visual representation of pre-trained model prioritizes the classifiability on downstream tasks, while the widespread applications for pre-trained visual mod- els have posed new requirements for representation interpretability. However, it remains unclear whether the pre-trained representations can achieve high inter- pretability and classifiability simultaneously. answer this question, we quan- tify the representation interpretability by leveraging its correlation with the ratio of interpretable semantics within the representations. Given the pre-trained rep- resentations, only the interpretable semantics can be captured by interpretations, whereas the uninterpretable part leads to information loss. Based on this fact, we propose the Inherent Interpretability Score (IIS) that evaluates the information loss, measures the ratio of interpretable semantics, and quantifies the representa- tion interpretability. In the evaluation of the representation interpretability with different classifiability, we surprisingly discover that the interpretabilily and clas- sifiability are positively correlated, i.e.. representations with higher classifiability provide more interpretable semantics that can be captured in the interpretations. This observation further supports two benefits the pre-trained representations. First, the classifiability of representations can be further improved by fine-tuning with interpretabiliiyy maximization. Second, with the classifiability improvement for the representations, we obtain predictions based on their interpretations with less accuracy degradation. The discovered positive correlation and corresponding applications show that practitioners unify the improvements in interpretability and classifiability for pre-trained vision models. Codes are available at here
INTRODUCTION
With the rapid development of vision pre-training models, their representations have been employed in diverse scenarios (Li et al. 2022a Kirillov et al., 2023 Cui et al. 2024). achieving remarkable performance downstream tasks (Dosovitskiy al. 202 Liu al. 2021 Tong al.) 2022 Wang et af. 2023). However, except for better classifiability. the extensive applications that re- quire reliable predictions underscore the interpretability of vision representations. To this end, anteppetability-oorineeed methods (Koh et al. 2020 Wang et al., 2021 Chen et al., (2244) impose pre-defined semanties on each dimension of the representation space. These semanties serve as the interpretability constraints for further predictions. different from the training strategies that solely prioritize classifiability improvement (Dosovitskiy et al. (022). However, the dre-nef semantics gain representations with interpretability but degraded classihaailiity, indicating an inherent conftict
between interpretability and classifubility (Rudin al. 2022 Zarlenga et al. 2022).
Different from the encepretability-orieiti representations, the other type of methods (Kim et al. 2018 Ghorbani al. [2019] Oikarinen et al. (203)) obtain interpretations by capturing interpretable semantics within the classiiibiliiy-orieneed pre-trained representations. However, in contrast to the pre-defined semantics for the interpretabilit-orienni representations, the semantics of the
"Corresoodingg author.

---

Published as a conference paper at ICLR 2025
classifiabillty-oriinn representations not guaranteed to be interpretable and fully captured dur- ing interpretation. As a result, semantic information loss occurs when only a subset of the orig- inal semantics be interpreted Yüüksekgönül et al. 2023). The interpretability difference be- tween the classffabilit--orieneed representations facilitates us to further explore the interpretability- classifiability conflict: first, whether the reduced interpretability of the classifiabiliy--orieetee repre- sentations is due to their high classifiability; second, whether the improvements in the interpretabil- ity always lead reduced classifiability. Our target answer that can the classifiablity-oriented representations achieve high interpretabilityy and classiftability simultaneously?
In this paper, we present stud- ies on the aforementioned ques- tion and discover that interpretabil- ity and classifiability are not con- flicting but positively correlated for classifabbilty--riiinte represen- tations. To quantify the repre- intro-
Interpretable Semanties
Uninterpeetable Scmanties
Represcntation
Interpretation
(Interpretability)
VIT-L
sentation interpretability, duce the Inherent Interpretability Score (IIS). IIS is defined as the ability of representations to preserve task-relevant semantics (e.g., con- cepts and patterns useful for clas- sification) in interpretations (Fig- ure More interpretable repre- sentations should have less semantic information loss in interpretations. For the representation classifiability, employ prediction accuracy the evaluation metric. With accuracy and our proposed IIS, investigate re-
ViT-B
IIS
Rest52
Res50
Res34
Resi8
Inherent Interpretability
Accuracy
Score (IIS)
(Classifiabilitt)
Figure 1: Classifiabilit--orentee representations have unin- terpretable semantics, causing interpretability reduction. We propose the IIS to quantify representation interpretability with the maintenance of task-relevant semantics in interpreta- tions (left). By comparing the IIS and prediction accuracy of representations from different models, observe positive correlation between interpretability and classifiability (right).
the eeeeprrtabiiity-clasfffabb
lationship of pre-trained representations. Extensive experiments different datasets (Krihevsky
et al. 2009; Wah et al. 20111 Russakovsky et al. 2015) and pre-trained representations (He et al.
2016 Dosovitskiy al. [202]] Liu al. 2021 2022) reveal positive correlation between inter-
pretability and classifibiility rather than contradictive one as discussed in (Mori & Uchihira 2019
Mahinpei et al., [021] Zarlenga et al.) 2022 Dombrowski al., (023).
The outcome of our study benefits two aspects. First, improving interpretability during representa- tion fine-tuning can further promote the classifiability on downstream tasks. Second, representations with higher classifiability possess enhanced interpretability. which also indicates more interpretable task-relevant semantics. Therefore, predictions based on the interpretations become more compa- rable to the original classffabbilty-oriente representations and exceed that of the interpretability- oriented representations (Zarlenga et al. 2022; Yang et al. 2023). The mutual promoting relation between interpretability and classifiability provides new pathways for interpretable representation learning. The contributions of our work are as follows:
We propose a quantitive measure (called IIS) for the representation interpretahility.
We uncover the positive correlation between the interpretability and classifiability of the classifiahilit--orinnte pre-trained representations.
We discover the mutual promoting relation between the interpretability and classifiability of the measifiabiliiy-oriieete representations. Improving the interpretability can further improve the classifiability, and representations with higher classifiability are more inter- pretable to preserve more task-relevant semantics in interpretations.
REPRESENTATION INTERPRETABILITY MEASUREMENT
In this section, we propose IIS to quantify the representation interpretability. which measures the semantic information loss during interpretation by the accuracy retention rate. Given dataset D,
2

---

Published as a conference paper at ICLR 2025
representation
interpretation
acc
prediction acc, with represeniation
acc
prediction acc, with interpretation
visual concepls
soloct a concept library
acc
conaget
concept S
5x100% concepts
zero-BLt
acc
ARA
lextual concepts
Comere
concept
concept 2.
IIS
concept
sandpiper. lips and cars.
projection
sparsification
sparsity ratio (s)
Figure 2: Definition and computation of the IIS. Given pre-trained model and downstream task, we first collect task-relevant concept libraries (left) and interpret model representations by projecting them into the concept space. By interpreting representations with sparse concepts, we can extract their interpretable semantics (middle). The IIS is defined as the representation's 's ability to retain accuracy when predicting solely based interpretations (right).
a pre-trained vision model f R + RD that maps the inputs from R to representation space RD we first interpret the representation space D by correlating with M human-unddesstnddabe concepts which span concept space in IR M (Section 2.1). Then we project the representation to RM as interpretations, which can also serve for making predictions on downstream tasks (Section IIS with the accuracy retention rate between the representation and interpretation predictions measures the semantic information loss from the representation space R.D the constructed space R indicating the representation interpretability (Section 2.3
2.2
2.1 BRIDGING REPRESENTATION SEMANTICS WITH CONCEPTS
The representation space can be interpreted by correlating to human-understandde concepts. To achieve this, we collect a task-relevant concept library with M concepts inspired by previous meth- ods (Kim et al.) 2018 Ghorbani et al. 2019 Oikarinen et al. (023). These concepts span a concept space - which can be employed for interpretations by correlating with the vectors D.
Concept Library. As the fundamental issue of interpretations, how to define the concept library has an important impact on our investigations regarding the seapretaailityyclasiffaa To ensure our investigations are not confined to specific concept library, we perform investigations based on four types of concept libraries (Prototype, Cluster, End2End, Text) respectively. These libraries can be categorized into Visual and Textual 1 based the concept modalities. Note that different concept libraries are utilized separately during investigations without being concatenated.
relationship.
Visual. A visual concept is represented by several image patches (Kim et al. 2018) seg- ments (Ghorbani et al., 2019). Here we take the patch as an example. Given a set of patches, a pre-trained vision model is utilized to extract visual concepts. This model takes each patch input and generates a corresponding feature. Then the patches are aggregated into visual concepts based on their features. Specifically, we build three types of concept libraries according to their aggrega- tion strategies. (i) Prototype: each patch serves as concept without being aggregated. (ii) Cluster: patches are aggregated into concepts with pre-defined clustering methods applied their features. (iii) End2End: patches are aggregated in a learnable manner during end-to-end training.
Textual. A textual concept represented by a single word or phrase. The development of large language models (Brown et al. 2020) enables the automated collection of Text concept library based on target classes. This capability stems from the domain knowledge encapsulated within these models regarding the salient concepts for detecting each class. In light of the success of previous work (Yang et al.] 2023] Oikarinen et al. 2023), we utilize GPT-3 (Brown et al. 2020) to extract textual concepts for each class with pre-defined prompts.
Projecting Relations between Representation Space and Concept Space. Given a pre-trained vision model f R R.D. an image classification dataset and a concept library C {cI, C2, constructed based on D, can denote each concept c; with vector c, is a visual concept comprised of n image patches/segments {Ii,..., kx)), its concept vector C2 E IRD is obtained by averaging the representations of these patches/segments,
RD
1
C
f(zz).
(1)
n1
3

---

Published as a conference paper at ICLR 2025
If G is textual concept, the corresponding vector C; is obtained in a learnable manner. A vision- language model [Radford et al. 2021 fwi R x C - IR is utilized to generate the soft label y = foi (zz c;) E R for concept c; image E D. The vector c; is obtained through the minimization of the discrepancy between y and the inner product of c; and the image representation (()
C; arg min Eg (v fx))
The concatenated concept vectors C =
[ci, c..] E DxM can project representations from RD to RRd empowering the interpretation of the representations in R with M concepts.
2.2 INTERPRETING REPRESENTATIONS WITH SPARSE CONCEPTS
With the concatenated concept vectors C, the pre-trained representation x f(x) E R of image r can be projected into the constructed concept space R.M.
x -x(( C
where the i-th dimension of the projection result x RM can be regarded the contribution score of the concept & to the original representation x. However, these contribution values can not be em- ployed as human-undee unduuunnnnddsstdabbll interpretations due to the human cognitive limitations in processing a large amount of concepts (Ramaswamy et al. 2023). concept library typically involves a large number of concepts encompass task-related semantics comprehensively (Oikarinen et al., 2023 Yang et al. 2023). Therefore, incorporating the contribution of each dimension in Ia for inter- pretation would entail an overwhelming number of concepts, which counter prouter-prouuctvvlly becomes inconprehensible for humans. To address this problem, we utilize a sparsification mechanism (Shen al. 2024) to control the number of concepts that contribute the interpretations.
Given a sparsity ratio E [0, 1]. the sparsification objective is to zero out x 100% elements in the projection result x E TRM.. Based on Equation 3 the larger absolute values in x indicate the greater contribution of the concepts and their stronger correlations with image semantics. Therefore, we zero out elements and sparsify the concepts based on the element values in x Let xy denote the element with the x Mftth smallest absolute value of x For the i-th element xf in x the sparsification can be formulated as,
x; = gx()); ; max 1x
(4)
where xC.* RM denotes the interpretation of representation x. This sparsification function re- moves concepts in ascending order based on their similarity to the image representations and ensures that only [(1 8) x M] concepts serve the interpretation.
2.3 INHERENT INTERPRETABILITY SCORE
The semantics of the pre-trained representations decoded with various concepts during inter- pretation, where more interpretable representations empower less semantic information loss (Sarkar et al.) 2022: Yüksekgönül et al. 2023). Only the interpretable semantics captured by the in- terpretations. As a result, interpretations with certain information loss become less discriminative and classifiable downstream tasks compared the original representations. In light of this, formulate the evaluation of the representation interpretability as the accuracy comparison between predictions based on the original representations and their corresponding interpretations.
To evaluate the accuracy of predictions based on the interpretations, we further train a linear classifier Jeie that takes interpretations as inputs for prediction.
(xCe) = W C, +bb
(5)
where W [W1, W2 wx] MNN and b E V are trainable parameters.
comparison between the prediction accuracy of the original representations and their correspond- ing interpretations, we propose Accuracy Retention Rate (ARR), i.e., the ratio of prediction accuracy based on interpretations to that on representations,
gc, h D) Aco(f.gan gels o o gc. D) geis o go ge1s ARR(F,
(6)
Acc h D)
4

---

Published as a conference paper at ICLR 2025
where h denotes linear classification head. Acc(f. h. D) means the accuracy dataset D, achieved by a model with a pre-trained backbone f : R RD for representation extraction and linear classification head h RR RN for classification.
Note that ARR is still not sufficient for the mcasurement of the representation interpretability. Dur- adjust the sparsity ratio through the proposed sparsity mechanism to
ing the interpretation, control the amount of concepts to satisfy the rule of human-comprehensibiityy However, determin- ing the value of the sparsity ratio s poses another challenge, where the optimal value depends on both the downstream task and subjective human judgment. To minimize these dependencies and ensure the broad applicability of the evaluation, we consider the values of E [0, and measure the total ARR across these values as the interpretability measurement. Formally, the interpretability metric IIS is the area under the curve of sparsity ratio and ARR,
IIS(S, [ So1s 9o gc, h D)ds.
ARR(J,90
In practice, given a pre-trained model f, a dataset D and a concept library C, we select multiple sparsity ratios and train the corresponding Ic1s for each ratio to estimate IIS.
Prototype
Cluster
118
I1S
0.6
0.700
0.725
0,750
0,773
0,800
0.25
0,850
0.855
0.700
0,725
0.750
0.775 0.300
0,825
0,850
0.755
Accuracy
Accuracy
End2End
Text
119
0.50
115
0.95
0.70
0,700
0.725
0.750
0,715 0,800
0,525
0,830
0.855
6,700
0,725
0,580
0.775
0.800
0,625 0,850 0,875
Accuracy
Accuracy
RetNet:
RecNet: 50
ResNett 152
VIF-L
Swin
ConvNeXt-T
Com-NeXe
R4
ResNee- 101
VI-BB
Swin-T
Swin B
ConvNeXi-S
Com-NeXtL
Figure 3: The relationship between IIS and the prediction accuracy of pre-trained representations on ImageNet. We provide experiments with four types of concept libraries.
ITTERRRETABILLTYY-CLASSSS
RELATIONSHIP INVESTIGATION
First, we provide the experiment settings in Section 3.1 Then, we investigate the interpretability- classifiability relationship by comparing the IIS and prediction accuracy of pre-trained representa- tions (Section 3.2p. Finally, we analyze the relationships between classifiability and key factors of interpretability, namely, sparsity and ARR (Section 333
3.1 EXPERIMENT SETTINGS
Pre-trained Representations. We quantify the representation interpretability of well-known pre- trained models including ResNet- 183.Nett1//34/0/111111
(He (016)). ViT-B/I.-16 (Dosovitskiy et al., 2021), ConvNcXt-T/S/BAl (Liu et al. 2022), and Twin-T/S/B (Liu et al. 2021). The weights of
these pre-trained models are provided by Torchvision (Paszke et al. 2019). 9). In addition to image
representations, also conduct experiments video representations Carreira & Zisserman 2017
Feichtenhofer et al. 2019 Tong et al. 2022 Wang et al. 2023 Li et al. 2023b a) in Appendix A.4
Datasets. We compute the IIS of representations
ImageNet1 (Russakovsky et al., 2015), CUB-
200 Wah et al. 201T). CIFAR-10 (Krizhevsky et al. 2009) and CIFAR- 100 (Knzhevsky et al.]
2009 These datasets diverse tasks. CIFAR- 10/100 and ImageNet1 K are widely used for
5

---

Published as a conference paper at ICLR 2025
CUR-200
CIFARIO
C0
0.85
0.94
0.90
0.92
0.89
0.75
0.90
0.70
0.70
0.75
0.75
0.85
0.70
0.75
Accuracy
Accuracy
Accuracy
Ro-
ReaNes550
Nas-110
VIT-B
VII4
Swin-T
Swin
Figure 4: The relationship between the accuracy and IIS on three datasets (CUB-200, CIFAR-10, and CIFAR-100). The IIS is computed based on the textual concept library.
general image classification tasks. CUB-200 is specifically designed for fine-grained bird-species classification. Their sizes vary greatly, with CUB-200 having 5900 training samples, CIFAR datasets 50,000 each, and ImageNetIK K having 1-2 million training images.
Concept Library Construction. For visual concepts. we aggregate image patches or segments into 200 concepts using three strategies illustrated in Section 2.1 on ImageNetIK. For textual concepts, we follow previous work (Oikarinen
al.] 2023) that uses GPT-3 (Brown et al. 2020 to extract
concepts with prompts. The number of concepts is as follows: 143 for CIFAR-10, 10, 892 for CIFAR- 100, 370 for CUB-200, and 4751 for ImageNetIK. We use CLIP (ViT-B-16) (Radford et al. 2021) to generate soft concept labels for all datasets.
More implementation details are presented Appendix A.I
3.2 INTERPRETABILIY AND CLASSIFIABILITY ARE POSITIVELY CORRELATED
In this section, explore the interpretability- intrretabblttyccllasffabbiii relationship of pre-trained representa- tions. Empirical analysis across representations from different models is first conducted, followed by concentrating on representations from the same model with different training iterations.
Analysis across Representations from Different Pre-trained Models. With representations of different pre-trained models, we investigate their seaprretaabbityylllsffabbbii relationship with dif- ferent concept libraries (Figure 3 and datasets (Figure 4. A positive correlation between classifia- bility and interpretability can be observed for all types concept libraries and datasets, especially when comparing representations from models sharing the same architecture but different numbers of parameters (points linked by dotted lines, such as ViT-B and ViT-L). These results indicate that the interpretubility of pre-truined representations increases along with their classifiability improve- ment. We observe the presence of this phenomenon in the pre-trained video representations as well (Appendix[A.A) A.4). Additionally, we notice that the representation interpretability is also influenced by the model architecture, as evidenced by the non-strictly positive relationship between IIS and accu- racy across representations from different architectures (e.g., ViT-B and Swin-T). This observation reveals the potential of IIS evaluating the interpretability of general model architectures.
Analysis along the Pre-training Process. To further investigate this positive correlation between interpretability and classifiability, we analyze the IIS evolution during the pre-training process of representations. Figure the changes in the IIS on different prediction accuracy and training epochs. In the initial phases of training, the representations achieve high IIS because both interpreta- tions and representations have similarly low accuracy. This phenomenon does not affect the applica- tion of IIS in measuring the interpretability of pre-trained representations. However, as the number of training iterations increases, the IIS gradually increases with improved accuracy. Therefore, the representation interpretability is primarily enhanced in the later stage of the training process.
3.3 FACTOR ANALYSIS FOR INTERPRETABILITY
To provide an in-depth analysis of the interpretability, we decouple IIS into two key factors, i... sparsity and ARR, investigating their mutual relationship and the relationship with classifiability.
First, we present the sparsity-ARR curve of pre-trained representations from different models on ImageNet in Figure 6, using four types of concept libraries. Considering representations from the same model, with the sparsity ratio s increasing from to 1, the interpretations rely on sparser con- cepts, and the corresponding values of ARR decrease. This indicates that sparser concepts lead to

---

Published as a conference paper at ICLR 2025
0.75
0.70
ARR (%)
ARR (%)
19
T M Manape
Nes
VIT-B
0.60
0
10
20
30
50
60
70
Accuracy
Epochs
Egochs
(a)
(b)
(c)
Figure 5: The evolution of the IIS with varying accuracy (a) and epochs during the pre-training process of representations from ResNet-50 (b) and ViT-B (c) on ImageNet.
more information loss in the interpretations of the same representation. Considering the compari- son of different models, the representations show different abilities to support interpretations given different sparsity ratios. Representations that can better retain classifiablity (higher ARR) in inter- pretations under different sparsity ratios gain larger areas under the curve. This further validates IIS the quantification metric of representation interpretabilityy
We further analyze the factor-level seeeppetabliitycclaaffiiaiiiii relationship in Figure 7 ARR in- creases simultaneously with the improvement of the classifiability of the representation across dif- ferent sparsity ratios. This indicates that as the representation classifiability improves, it can be interpreted by sparser concepts with less task-relevant semantic loss (higher ARR). Based on these relationships, we can obtain completely interpretable predictions using concepts, with higher accu- -riientt
racy compared interpretability oriented methods. This can be further verified in Section 4.2
Prototype
Cluster
End2End
Text
80
00
80
ARST (15)
(55)
050
60
ReaNes-88
RodNen 18
RceNesi8
RosNer 18
40
ReeNes-34
10
Bes
40
RodNas-44
BesNet-3 34
BusNei500
20
RecNes1011
20
Henhet-1li
20
RacNe-101
20
RaNNe-101
RegNie155 152
Bechet-15 152
ReaNen 152
BesNee152
Sparsity Ratio (4)
Sparsity Ratio (x)
Sparstit Ratio (x)
Sparsity Ratio (x)
Figure 6: The sparsity-ARR curve of representations from multiple models with four types con- cept libraries. All representations are evaluated ImageNet.
ImageNet
CUB-200
CIFAR-10 10
CIFAR-100
0.99
ARR
ARR
ARR
ARR
0.0
Accuracy
Accuracy
Accuracy
Accuracy
Figure 7: Relationship of the sparsity, accuracy, and ARR on different datasets. Larger point sizes indicate larger sparsity (i.e., interpreting representations with sparser concepts). Experiments are conducted with text concept libraries specific the downstream dataset.
4 APPLICATIONS OF INVESTIGATION RESULTS
The investigation results of Section 3 guidd the application studies in two aspects. First, inspired by the positive correlation between interpretability and classifiability in Section 3.2 we can improve the classifiability of representations through interpretability maximization. Second, offer inter- pretable predictions with higher accuracy based on the observation in Section 33. that improving representation classifiablity results in improved ARR across different sparsity ratios.
7

---

Published as a conference paper at ICLR 2025
Table 1: The classification accuracy of widely-used pre-trained models and their accuracy after fine- tuning with IIS maximization.
Acc. (%)
ARR (%)
75
Model
Type
@cc@1
Acc@5
ResNet-50
origin
75.66
92.67
74
ours
77.36
93.53
Ori-Ace.
96
I1S-Acc.
Swin-T
origin 81.46
95.78
ARR
ours
81.80
95.86
95
-onvNext-T
origin 82.42
96.18
Epochs
ours
82.63
96.35
Figure 8: Prediction accuracy with repre- sentations in the original space (Ori) and sparse subspace (IIS) during the fine-tuning of ResNet-50. The ARR also provided.
origin 80.58
95.15
VIT-B
ours
81.07
95.33
Table 2:
IIS maximization on ImageNet for 50
Table 3: IIS maximization ImageNet for 100
epochs. The sparsity ratio is set 0.1.
epochs. The dimension M is set to 200.
M
100
200
300
600
Sparsity Ratio (s)
0.1
0.5
0.9
Acc@1 1
76.25
76.72
76.62
76.53
Acc@
76.62 92.98
77.07
76.95
76.67
Acc@5
93.08
93.20
93.12
93.16
Acc@
93.43
93.28
93.13
4.1
IMPROVING CLASSIFIABILITY BY INTERPRETABILITY MAXIMIZATION
Based on the interpretability (IIS) enhancement brought by classifiability improvement, this section, on the other hand, investigates whether enhancing IIS for representations would result in improved classifiability. Since the pre-trained model f changes for every optimization step, computing IIS each step can be consumin. To expedite the optimization process, we employ a simplified ver- sion of IIS, i.e., replacing the concatenated concept vectors C with learnable matrix C₁ E IR DxM and fixed sparsity ratio s. Maximizing the simplified IIS can improve the original IIS (Table A18 indicating that the simplified IIS mization. With the simplified IIS, we fine-tune the model f with the following objective:
viable alternative to the original IIS during the opti-
L(y, (h f())) + Cly, (gals (C f(z))),
(8)
where L denotes the cross- coss-ey loss function. The first term corresponds to the original predic- tion head, and the second term corresponds the with simplified IIS constraints. The simplified IIS is enhanced by maximizing accuracy with sparsity constraints in a learnable subspace. Table I presents the results ImageNet. By incorporating the IIS maximization objective into the optimiza- tion, we observe a classifiability improvement for multiple widely used pre-trained representations. This underscores the mutual promoting relationship between interpretability and classifiability.
To better understand the classifiability improvement with IIS maximization, Figure g depicts the accuracy evolution for classification heads corresponding to the first and second terms Equation R during the fine-tuning process. The evolution of ARR is also presented to analyze the interrelation between the original prediction head and the head with IIS constraints. With an increase in the num- ber of training iterations, we observe consistent improvement in the accuracy of both classification heads and their resulting ARR. This phenomenon can be explained by the regularization effect duced by the second terms in Equation g which encourages task-relevan semantics within IR to be less complex and fit the low-dimension sparsified space in R.M
We further investigate the effect of hyperparameters in the simplified IIS. Ablation studies on the concept space dimension M for C and the sparsity ratio s are provided in Table 2 and Table respectively. For the concept space dimension M. observe that the model achieves optimal performance when M is set to 200. For the sparsity ratio s, the optimal performance is achieved at small setting (s 0.1), while any higher ratios would result in performance degradation.
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparisons between interpretability- oriented methods and our methods on CUB-200 *We re-implement ECBM on our concept library.
Table 5: Accuracy of interpretable predic-
tions (Ace Inte), accuracy of pre-trained rep- resentations (Ace Ori) and ARR ImageNet.
Method
#Concept
Acc@
Method
Acc_Int Inte
Ace Ori
ARR
Ante-Hoc
312
64.17
SENN
58.55
CBM
112
75.90
Ante-Hoc
65.09
CEM
112
79.60
LaBo
83.97
Proto2proto
2000
79.89
ProtopNet
2000
80.80
Ours (Swin- T)
78.72
81.46
96.63
ECBM
112
81.20
Ours (Swin-S)
81.09
83.14
97.53
ECBM
247
83.21
Ours (Swin-B)
81.88
83.61
97.93
Ours (VTT-L)
123
81.30::0.08
(VTT-B) 78.50
80.58
97.42
Ours (VIT-L)
247
83.50:0.115
Ours (ViT-L)
86.85
87.99
98.70
4.2
PROVIDING INTERPRETABLE PREDICTIONS WITH HIGH ACCURACY
Section 3.3 shows that representations with higher accuracy can be interpreted by sparser concepts with less semantic loss. In light of this, we offer interpretable predictions based on J=is in Equa- tion with higher accuracy that is close to the original representations. Table 4 presents com- parisons between predictions based on the interpretations of classifabiiity-oriette representations and the interpretability-orieee Our interpretations achieve comparable or even better accuracy to anlepretability-oriinnte ods 81.29% vs. 81.20%) with a similar number of concepts M (123 vs. 112). As M increases from 123 to 247, we observe significant accuracy improvement over the existing best result (83.50% vs. 83.21%). Note that we only take the classification head as the trainable module, contrast to interpretability-orieni methods training the entire model. This demonstrates the advantage of low training costs using classificabillty-orinni
methods that train representations with interpretability constraints.
meth-
representations to produce interpretable predictions.
Furthermore, report the accuracy comparison ImageNet in Table 5 Compared with exist- ing unterpretabiliiy-orieeeee
methods, the interpretations of classifiability representations surpass them by 2.88% (86.85% 83.97%). Moreover, compare the accuracy of interpreta- tions (Ace Inte) and their original representations (Acc Ori). Taking ViT-B and ViT-L for examples, representations with higher accuracy (87.99% ARR (98.70) 78% vs. 97. 97.42%). thereby yielding interpretable predictions with significantly improved accuracy (86.85% vs. 78.50%). This observation is consistent with conclusions in Section 3.3 By leveraging the enhanced classifiability of pre-trained representations, we can employ their inter- pretations for predictions with performance closer to the original representations.
80.58%) derive interpretations with higher
Prediction: downcher Confiderce:
Prediction: black grosse
Confidence: 1,000
orndpiper
a grored
black ruff around the reck +4452
black planage
small stocky duarcind 10042
not bnoee
+8.07
motted brown plamage 18
loud coow
170
flekk 6.42
Block
+7.63
25
20
Concept
Crnsqe
Figure 9: Visualizations of two correct samples of interpretable predictions
The visualization examples for interpretable predictions are shown in Figure and more visualiza- tions are available in Appendix A.3 These examples show the most important concepts with high absolute contribution value. For the image Figure 9 recognizing this image as a "dowitcher" involves identifying "a sandpiper" the most important concept.
Additionally, we show the impact sparsity on the interpretations in Figure I0 by providing an overview of the significant concept-class relationships. Given low sparsity ratios, these relation- ships are low informative. Specifically, a single class is correlated to concepts with comparable contribution values, and similarly, concept correlates to classes with comparable contributions. As a result, the concepts are indistinguishable to classes, and vice versa, failing to provide clear knowledge for humans about the crucial concept in predictions and classification rules learned by

---

Published as a conference paper at ICLR 2025
Top Concepts Class Baseball
Top-10 Classes of Concept chang ng table
Senectherd
peaher
polet shon
baby
Hooo444 gino
Browng Iaaudall
Ioding CAA
boad wid
recyting
Sparsity
Sparsity
Sparsity
Sparsity
Figure 10: Visualization of the average contribution of concepts to classes, showcasing how the sparsity ratio affects the quality of interpretations through the entropy of concept contributions.
the models. By increasing the sparsity ratio, the model tends to correlate class smaller number of key concepts and provide clearer insights into its decision-making process.
5 RELATED WORK
Post-hoc Interpretation Methods. Post-hoc methods have been widely studied for interpreting model predictions by mapping the model's representation human-understandable domains (e.g.. image pixels. words, etc.). To interpret representations, previous work attempts to decouple rep- resentations into human uammaannuddrttaddabbll elements such as concept vectors (Ghorbani et al. 2019 Oikarinen et al.) 2023), explanation graph (Zhang et al. [0177] 2018) or decision tree (Zhang et al,, 2019 Bai et al. 2019). Inspired by methods that correlate representations with concepts, we extract interpretable semantics embedded within representations for their interpretability measurement.
Interpretability Quantification. Interpretability describes the extent to which humans are capable of understanding (Longo et al. [2020). As a subjective metric, interpretability can be impacted by many factors, including causality (Holzinger et al., 2019), faithfulness (Khakzar et al.) 2022), and sparsity (Oikarinen et al. (202)). Existing metrics are based on heuristic approaches that compare the relevance attributed to the different features with the expectation of an observer (Bau et al. (017), a domain expert (Neves et 2021), or calculate the homogeneity score between interpretations and ground-truth labels (Zarlenga et al. 2022). Different from the above metrics designed for model interpretations, we focus pre-trained representations and quantify their interpretability by their accuracy retention ability when predicting solely based on interpretable semantics.
The conflicting relationship between interpretability and classifiability has been observed in interpretability-riieee
Relation.
methods and studied for years (Baryan- nis al. 2019 Rudin et al. 2022). It says that models imposed with stronger interpretabiliity
constraints or inductive bias demonstrate weaker classification accuracy (Mori & Uchihira 2019
Mahinpei et al.) 2021; Zarlenga et al.] 2022; Dombrowski et al.] (023)). Existing work tries to overcome this issue by the design of interpretable architectures (Zarlenga et al., 2022] or feature engineering (Gossiewska et al., (02T)) and make slow progress. contrast to previous investigations regarding interpretabillty-orienn
methods, we focus classifiabillty representations and indicate that classifiability and interpretability are not conflicting but positively correlated.
CONCLUSION
We explored the possibility of achieving high interpretability and classifiability simultaneously by identifying interpretable semantics within pre-trained representations. First, we proposed the IIS to measure the representation interpretability by the maintenance of telk-relevant semantics in interpre- tations. Then, we investigated the tations. Extensive experiments on multiple datasets indicate that classifiability and interpretability positively correlated. By enhancing representation interpretabilityy improve the classifiabil- ity of pre-trained representations. By improving representation classifiability, we offer interpretable predictions based on interpretations with more comparable accuracy the original representations. For future work, we will provide a theoretical analysis of the interpretability sesprrttabiiyycllasffbbblil classifiability re relation- ship and take more factors (e.g., trustworthiness, structural entropy of representations) refine IIS, and seek to develop an interpretable training paradigm for llnnualea foundation models.
-igetttabiilyyycasiiiiiill relationship of pre-trained represen-
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide comprehensive details on datasets, pre-trained models, and concept library construction in Section 3.1 Details of training strategies and IIS com- puting can be found in Appendix A.I Source code has been submitted as supplementary materials.
ACKNOWLEDGEMENT
This work was supported in part by the National Key R&D Program of China under Grant YFC250055,,, in part by the National Natural Science Foundation of China: 62236008, 62022083, U21B2038, 62306092 and 62441232, and in part by the Fundamental Research Funds for the Central Universities. The authors would like thank Zhengqi Pei, Yue Wu, and the anonymous reviewers for their constructive suggestions that improved this manuscript.
REFERENCES
Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süssttrunk. Slic superpixels compared to state-of-the-an superpixel methods. IEEE transactions on pattern analysis and machine intelligence, 34(11):2274-2282, 2012.
Jiawang Bai, Yiming Li, Jiawei Li, Yong Jiang, and Shutao Xia. Rectified decision trees: Towards interpretability, compression and empirical soundness. ArXiv preprint, abs/1903.05965, 2019.
URL
org. abs/ 1933. 05965
George Baryannis, Samir Dani, and Grigoris Antoniou. Predicting supply chain risks using machine learning: The trade-off between performance and interpretability. Future Generation Computer Systems, 101:993-1004, 2019.
David Bau, Bolei Zhou, Adity Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 3319-3327. IEEE Computer Society, 2017. doi: 10.1109/CVVPR.207.33. URL https: //dos doi. org. ///g/11..0 100.11109//PP. 2017. 354]
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th Inter- national Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pp. 813-824. PMLR, 2021. URL http:://roceedings mlr. wttppp/eee////eettaaeeeee
html
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http::::pprcccc.gggss.. new.ips...ni...n................ ctt.ccc/ppppaa wttp://ppe////////a/////// wttp:///9.....1..eeaaaaaa1114110 Abst:eataattttttttt. html
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In 2017 IEEE Conference Computer Vision and Patlern Recognition, CVPR 2017, Honoludu, HI, USA, July 21-26, 2017, pp. 4724 4733. IEEE Computer Society, 2017. doi: 10.
1109/CVPR.2017.555 URL httph:tpp....m //do:i /oi. org /10 111.111111/10 CVtp/ 5022
Guikun Chen, Xia Li, Yi Yang. and Wenguan Wang. Neural clustering based visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 5714-5725, 2024.
11

---

Published as a conference paper at ICLR 2025
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, al. A survey multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958-979, 2024.
Mischa Dombrowski, Hadrien Reynaud, Johanna P Müller, Matthew Baugh, and Bernhard Kainz. Pay attention: Accuracy versus interpretability trade-off in fine-tuned diffusion models. ArXiv preprint, abs/2303. 17908, 2023. httpp:s:s///axxii org. abs 2303. 2bb.//33.3.790
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL ptttttss://openeeviiie Id=YicbFdNTTy
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In 2021 IEEE/CVF International Con- ference on Computer Vision, ICCV Montreal, QC, Canada, October 10-17, 2021, pp. 6804- 6815. IEEE, 2021. doi: (0.11099CCC449922.0000 100675. 1109/ ICCV48922 2021 00675
URL http:::tt///..... /ooi org/
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In 2019 IEEE/CVF International Conference Computer Vision, ICCV Seoul, Korea (South), October 27 November 2, 2019, pp. 6201-6210. IEEE, 2019, doi: 109/109/ 2019.00630. URL https: :/ org/ 10. 11111101CCCCC 2019. 00630
Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting clip's image representation via text-based decomposition. arXiv preprint arXiv:2310.05916, 2023.
Amirata Ghorbani, James Wexler, James Y. Zou, and Been Kim. Towards automatic concept- based explanations. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d' d'Alché Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp- 9273-9282, 2019. URL
neur ienurrips..... ccc/paper//219/haatt Abstract html
Alicja Gosiewska, Anna Kozak, and Przemystaw Biccek. Simpler is better:
Lifting interpretability- performance trade-off via automated feature engineering. Decision Support Systems, 150: 10:1113. 113556, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1109/CVPR.2016
htRL:tts::/ http:://oooo.oooo
1wt..//1../..... 2016.
Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal, and Heimo Müller. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 14) 2019.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL net/ d
Glenn Jocher. Yolov5 by ultralytics.
2020.
12

---

Published as a conference paper at ICLR 2025
Ashkan Khakzar, Pedram Khorsandi, Rozhin Nobahari, and Nassir Navab. Do explanations plain? model knows best. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 10234 10243. IEEE, 2022. doi: 10. 101.0/CVR52288.2222222222 2022.01000
htI.htts::/
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learming. ICML 2018, Stockhoimsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2673- 2682. PMLR, 2018.
mlr www.seeeseap.... kiml 8d. html
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 3992 4003. IEEE, 2023, doi: 10. (0.1109/CCCV5100022220 00371. utttttts:://ooi org 10.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5338-5348. PMLR, 2020. WRRLittpp//proceeiing
mlr. pwess/vviss wttsss///19//oohhaa.mmm html
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 12888- 12900. PMLR, 2022a. ml r. www.prrsss........ vl 62/ 22n html
URL
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Limin Wang, and Yu Qiao. Uni- formerv2: Spatiotemporal learning by arming image vits with video uniformer. ArXiv preprint, abs/2211 1,0952, 2022b. URL htttt::t////arttt
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Limin Wang. and Yu Qiao. Uniformerv2: Unlocking the potential of image vits for video understanding. In /EEE/CVF International Con- ference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 1632-1643. IEFE, 2023a. doi: 10.1 119//CCV510700.222..0
URL phttps://do. org 1109/
ICCV51070.22 2023. 00157
Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 2023b.
Tang Li, Mengmeng Ma and Xi Peng. Deal: Disentangle and localize concept-level explanations for vlms. In European Conference Computer Vision, pp. 383-401. Springer, 2025.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference Computer Vision. ICCV 2021, Montreal, QC, Canada. October 10- 17, 2021, pp. 9002- IEEE, 2021. doi: 10. 100100//C4449922220202 1.0996. URL https: 7/doi. maciioorg/100 / 1109 TCCC/9CCVV2422 2021 60986
13

---

Published as a conference paper at ICLR 2025
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 1199666-19966 IEEE, 2022. doi: 10. (199CCVPR52688202222.1111 URL https: /doi orgg 1ttp://111..111//2//2222
2022.
01167
Luca Longo, Randy Goebel, Freddy Lecue, Peter Kieseberg, and Andreas Holzinger. Explainable artificial intelligence: Concepts, applications, research challenges and visions. In International cross-domain conference for machine learning and knowledge extraction, pp- 1-16. Springer, 2020.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. ArXiv preprint, abs/2106. 13314, 2021. URL https://arxiz: wwtp:////g//aaaa//////oo
Toshiki Mori and Naoshi Uchihira. Balancing the trade-off between accuracy and interpretability in software defect prediction. Empirical Software Engineering, 24:779-822 825, 2019.
Ines Neves, Duarte Folgado, Sara Santos, Marilia Barandas, Andrea Campagner, Luca Ronzio, Federico Cabitza, and Hugo Gamboa. Interpretable heartbeat classification using local model- agnostic explanations on ecgs. Computers in Biology and Medicine, 133: 13::30433,,2 2021.
Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bot- tleneck models. In The Eleventh International Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpt:t:////ppceeeettss net net/pdf? het/pdffid-Fllqee AMNBBA
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library.
Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 8024 8035, 2019. URL http::sss/ppooceiiggg...so neur. CC/ http://cc.ccpppee//////aaaa/aa/ lbdcca28ffee 7 wtwedfeee9922222222 100.2228880.111110..000012222222
html
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Ma- mir.
chine Learning Research, pp. 8748 8763. PMLR, 2021. URL html
sxess/v133/radfoore
Vikram V. Ramaswamy, Sunnie S. Y. Kim, Ruth Fong, and Olga Russakovsky. Overlooked factors in concept-basee based explanations: Dataset choice, concept learnability, and human capability. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. (00032-10999 IEEE, 2023. doi: 10. 10.1109/9VPS552222000 01052. URL httpt::p////m ptstps://dof /oi. wttp:////...1.////.2.........22. 2222222922222222222 01052
Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. In- terpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 11::211-252, 2015.
Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N. Balasubramanian. A frame- work for learning ante-hoc explainable models via concepts. In IEFF/CVF Conference on
14

---

Published as a conference paper at ICLR 2025
Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA. June 18-24, 2022, pp. 000076--1288 IEEE, 2022. doi: 10.1009/VVPS22688222222 202.5220688.22.000000 URL https: //doi. 1099/VV5522228 2022. 01004
Shufan Shen, Junshu Sun, Xiangyang Ji, Qingming Huang, and Shuhui Wang. Expand- ing sparse tuning for low memory usage. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp. 76616-76642. Curran Associates, Inc.,
2024. URL https://proceddngg.... neurips cC/ cc/paper httpa//wwswa..ppppeeeaaa/aaa///r File/
files/paper/2024/ oop.aa/330000aaaaaa-----o-eeeeee www.eereooocooee...e.....ee pdf
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 804-814, 2022.
K Soomro. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012.
Divyansh Srivastava, Ge Yan, and Tsui- Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance. arXiv preprint arXiv:2408.01432, 2024.
Zhan Tong, Yibing Song. Jue Wang. and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training- In Sanmi Koyejo, S. Mo- Cho, and A. Oh (eds.), Advances Neural Information Processing Systems 35: Annual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 9, 2022, 2022. URL 2tt.:222222211222226662222222222
hamed, A. Agarwal, Danielle Belgrave,
wnps..i..ipp.............. cc/pppercpppppr.ppppppppppppp http://sesssppppe/////////aaaa html]
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham dataset, a large collection of multi-source dermatoscopie images of common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-ucs/ birds-200-2011 dataset. 2011.
Jiaqi Wang, Huafeng Liu, Xinyue Wang, and Liping Jing. Interpretable image recognition by con- structing transparent embedding space. In 2021 IEEF/CVF International Conference on Com- pufer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 875-884. IEEE, 2021. doi: 10.1 D0.109/CCV44922.222...00 100000. URL BRLhttps: / /oi. org / 10. 101./000.1111//144222222 2021. 00093
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae V2: scaling video masked autoencoders with dual masking. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 1449-144560) IEEE, 2023. doi: 10. W099///VR22299.2222..2 https: /oi. .rgg 10. . 10t/C/9222/2922....// 20t.2012.111111 01398.
URI.
Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy- basedd concept bottleneck models. In The Twelfth International Conference on Learning Representations, 2023.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3090-3100, 2023.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 19187-19197 IEEE, 2023. doi: 10. 1090CCVP5522992222.0111 URL https: /ttp:////.oo.oo.//.oooo// 1t1p:////111111/////22233333 2023.
0 0
15

---

Published as a conference paper at ICLR 2025
Mert Yüksekgönül, Maggie Wang. and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
nASAZBCEyow
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frédéric Precioso, Stefano Melacci, Adrian Weller, Pietro Lió, and Mateja Jamnik. Concept embedding models: Beyond the eccuracy-xpllainabiiiy trade-off. In Sanmi Koyejo. S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November De- cember 9, 2022, 2022. URL / . cc/ wwwpppprrcppppppp.ppppppp..pp..p www.eessppppe.
html
Quanshi Zhang, Ruiming Cao, Ying Nian Wu, and Song-Chun Zhu. Growing interpretable part graphs on convnets via multi-shot learning. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 2898-2906. AAAI Press, 2017. URL http://aaes
org/ocs/ weocs/indee.. index BBB//AAAI/AAAA /tt.A//AAA1////pr///aa/aa.
14909
Quanshi Zhang. Ruiming Cao, Feng Shi, Ying Nian Wu, and Song-Chun Zhu. Interpreting CNN knowledge via an explanatory graph. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, Febru- ary 2-7, 2018, pp. 4463-4463 AAAI Press, 2018. URL https: htt::://www... www. aaai wwwaaa..rgooooooooe ocs/
index. Index.php/Bae phppppppppAaa//// AAAT /AAATI httpi//ppppppp/// /eer/vvieee 17354
Quanshi Zhang, Yu Yang, Haotian Ma, and Ying Nian Wu. Interpreting cnns via decision trees. In IEEE Conference on Computer Vision and Pattern Recogni- fion, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6261-6278 Com-
puter Vision Foundation IEFE, 2019. doi: 10.1109/CVPR.C0 /0//CVVRR2019900642
URI.
httpp://opemaccess.cc thecvf. thecvt..teeee..... com/ wttp/otttttttteeeeeeeeee CVPR RttCcC/2PPPP___Eee2 //09/htmm/2hang
Interpreting CNNNss via Decision Trees CVPR 2019 paper. html
Chenyang Zhao, Kun Wang, Xingyu Zeng, Rui Zhao, and Antoni Chan. Gradient-based visual explanation for transformer-based clip. In International Conference on Machine Learning, pp- 61072-61091. PMLR, 2024.
16

---

Published as a conference paper at ICLR 2025
Table A6: Configurations of Selected Pre-trained Models.
Model
Interpolation
Resize
Input Size
Pretrain
ResNet- 18
Bilinear
256x256
224x224
ImageNer1 K V1
ResNet-34
Bilinear
256x256
224x224
ImageNet1 K_V1
ResNet-50
Bilinear
256x256
224x224
ImageNet1 K_V1
ResNet-101
Bilinear
256x256
224x224
K_a
ResNet-152
Bilinear
256x256
224x224
Mn.geNetlK.Vi
ViT-B-16
Bilinear
256x256
224x224
ImageNet1IKVI
ViT-L-16
Bilinear
512x512
512x512
ImageNet SWAG S E2E
Swin-T
Bieubic
246x246
224x224
ImageNer1 V1
Swin-S
Bicubic
246x246
224x224
ImageNetI K_V1
Swin-B
Bicubic
238x238
224
ImageNetIJKVV
ConvNeXt-T
Bilinear
236x236
224
ImageNetl
ConvNeXI-S
Bilinear
230x230
224x224
ImageNetIK.V
ConvNeXt-B
Bilinear
232x232
224x224
ImageNet1K.V
ConyNeXt-L
Bilinear
232x232
224x224
ImageNetIK K.V1
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 CONCEPT LIBRARY CONSTRUCTION
Visual Concepts. Given image classification dataset with N classes, randomly sample 10 images for each class. For these images, we extract segments or patches from them to build visual concepts. For patches,
resize each image to dimensions of 224 224. Then, split each sized image into 56 x 56 patches. For segments, We apply the SLIC (Achanta et al. 2012) algorithm extract segments unsupervvseelly. We perform the segmentation process three times, resulting in 10, 20, and 30 segments respectively, to capture segments at different scales. To avoid redundancy, eliminate segments with significant overlap at the same scale. During the experiment, we ob- served high redundancy among these segments/patches. Therefore, we randomly select H 20 segments/patches for each class. These selected patches or segments are then aggregated to form M visual concepts. Here we take the patch as an example for simplicity.
For Prototype concept library, we randomly select
M patches for each class.
For Cluster concept library, we aggregate all patches into A visual concepts by clustering based on their latent vectors using the K-Means algorithm.
The End2End concept library is specific to pre-trained model f. We design a learnable method aggregate the patches into concepts based on their latent vectors generated by f. The learning objective is to obtain concepts that maximize the classification accuracy built upon f. Specifically, for the concatenation of latent vectors of patches CP E H we assign these patches to M visual concepts
min (ly.))/Cyy.ess S01s (C, Q.9eca
(A9)
where f-1. R R depicts linear classification layer, L means the cross-entropy loss func- tion, Q lq 92. qNd E {0, 1 is a learnable transportation matrix responsible for assigning patches visual concepts. Each column q. E {0. 1] is one-hot vector that assigns with patch one of the concepts. practice, the discrete constraints of Q are enforced through Gumbel-Softmax with Straight Estimator (Jang et al. 2017).
Textual Concepts. Following previous work (Oikarinen et al. 2023), the textual concepts are auto- matically extracted for each class. The prompts are follows:
List the most important features for recognizing something as a (class):
List the things most commonly seen around a (class):
Give superclasses for the word (class):
17

---

Published as a conference paper at ICLR 2025
Table A7: Sparsity Selection Strategies for different concept libraries.
Dataset
Concept Library
Concept Num.
Selected Sparsity Ratios
Prototype
200
(0, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.98)
Cluster
200
(0, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 0.98
ImageNet
End2End
200
(0, 0.10 0.3, 0.5, 0.7, 0.9, 0.95, 0.98)
Text
4751
{0, 0.9, 0.99, 0.995, 0.997, 0.999}
CUB-200
Text
370
{0, 0.5, 0.7, 0.9, 0.99, 0.995)
CIFAR-10
Text
143
0.5 0.7, 0.9, 0.95, 0.97]
CIFAR-100
Text
892
0.5, 0.7, 0.9, 0.95, 0.97]
Moreover, the extracted concepts filtered. First, concepts with lot of words removed to make them easy to visualize and understand. Second, concepts with large similarities with target classes are removed prevent trivial explanations. Third, the retention of duplicate concepts (cosine similarity greater than 0.9) is limited to only one instance. Finally, concepts not presented in the training images (cosine similarity less than 0.4) are removed.
The number of concepts for different datasets is as follows: 143 for CIFAR-I0, 892 for CIFAR- 100, 370 for CUB-200, and 4751 for ImageNetIK. We use CLIP (ViT-B-16) (Radford et al. 2021 to generate soft concept labels for all datasets.
Specifically, we use the same configurations as Torchvision for the selected pre-trained models pre- sented in Table A6 All models are are-trained on ImageNet, except for ViT-L, which is pre-trained using SWAG (Singh al. 2022 and fine-tuned on ImageNet.
A.1.2 SPARSITY RATIO SELECTION
Practically, compute IIS by selecting series of sparsity ratios The selection strategy of corre- sponds to the concept library. It should be noted that we avoid selecting an excessively high sparsity ratio resulting in extremely sparse interpretations that single concept). The specific selection strategies are shown in Table A7
be confusing to humans (e.g.,
A.1.3 TRAINING STRATEGIES
During the training process in Section 2.2 the pre-trained model f is frozen. For Prototype and Cluster concept libraries, only the classifier Ja1s trainable. For End2End concept library, the projection from representation space to concept space and classifier Je1s are trainable. For Text concept library, the training process is divided into two stages. In the first stage, the concept vectors C are obtained through Equation 2
In the second stage, only the classifier Je1s is trainable.
We present the hyperparameters during training in Table A8 For each model, train three times using multiple learning rates of {0.1, 0.01, 0.001} and select the result with the highest accuracy, as different models may perform better with different learning rates. For visual concepts (Prototype, Cluster, and End2End), we utilize the Adam optimizer and the exponential scheduler with decay rate 0.99. For textual concepts (Text), we utilize the SGD optimizer without schedulers.
A.1.4 IIS MAXIMIZATION
For the fine-tuning process with IIS maximization as the objective, we utilize the sparsity ratio s = 0.1 and vector number M 200 to compute the simplified IIS. The pre-trained models undergo fine-tuning for 200 epochs (with the first 20 epochs warmup), utilizing batch size of 128. The finetuning process incorporates the AdamW optimizer with betas set to (0.9, 0.999), a momentum of 0.9, cosine decay learning rate scheduler, an initial learning rate of 3e 4, and weight decay of 0.3. Additional techniques such as label smoothing (0.1) and cutmix (0.2) are also employed. Beyond these, no further techniques are applied.
A.2 ADDITIONAL EXPERIMENTAL RESULTS
18

---

Published as a conference paper at ICLR 2025
Table A8: Hyperparameters for CBM Training
Dataset
Concept Libraries
Learning Rate
Epoch
Optimizer
Scheduler
Batch Size
Prototype
{0.1, 0.01, 0.001]
30
Adam
EXP
1024
Cluster
(0.1, 0.01, 0.001
30
Adam
EXP
1024
ImageNet
End2End
10.1,0.01, 0.001
30
Adam
EXP
1024
Text
{0.1, 0.01, 0.001
30
SGD
NAN
256
CUB-200
Text
{0.1, 0,01, 0.001
100
SGD
NAN
256
CIFAR-10
Text
(0.1, 0.01, 0.001
100
SGD
NAN
256
CIFAR-100
Text
(0.1, 0.01, 0.001
100
SGD
NAN
256
Table A9: IIS-accuracy relationship on CUB-200 dataset with different sparsification mechanisms.
Model
Acc.
Origin
Descending
Clustering
HardThres
ResNet-18
63.39
0.695
0.284
0.664
0.769
RscNet-34
64.22
0.719
0.298
0.675
0.774
RscNet-50
67.01
0.766
0.340
0.721
0.809
ResNet-101
67.13
0.776
0.352
0.732
0.816
RseNet-152
68.00
0.780
0.354
0.742
0.828
A.2.1 INFLUENCE OF SPARSIFICATION MECHANISMS
In this section, we investigate the impact of different concept sparsification mechanisms on IIS. In addition to the strategy in Equation 4 we implemented three alternative approaches to investigate relationship between interpretability and classifiability.
(i) Descending. We remove concepts in descending order based on their similarity to image repre- sentations. Specifically, given the similarity between image representations and concepts x E RM in Equation we first invert it and then apply Equation 4 for sparsification. (ii) Clustering. We reduce the number of concepts by clustering rather than directly eliminating concepts. Specifically, given sparsity ratio s, cluster M concepts to [(1 s) x M] clusters with KMeans. Image representations are projected into these clusters for IIS computation. (iii) HardThres. We replace the soft threshold in Equation with the hard threshold (i.e., directly masking concepts with smaller value than the threshold).
As Table A9 shows, representations with higher classifiability still demonstrate higher IIS under different sparsification mechanisms. This phenomenon further demonstrates that the positive cor- relation between interpretability and classifiability is inherent characteristic of pre-trained repre- sentations, rather than a consequence of specific sparsification methods.
A.2.2 INFLUENCE OF TEXTUAL CONCEPT VECTOR ACQUISITION
2 LF-CBM (Oikarinen et al.) 2023) has proposed the cos-cubed loss to learn concept vectors. For the similarities, the ob- jective of both loss functions is to map set of concept vectors [c1, C2.... Car] in the repre- sentation space to textual concepts [c1, C2.. cyr]. For the dissimilarities, given the soft-label ya [V vf2 .... vCM] of image and the inner products between concept vectors and image representations qx = [f()) C1, fx))
Besides our methods to acquire textual concept vectors in Equation
f(x) cas)- The cos-cubed loss normalizes vectors ya, q. and utilizes the inner product of their cubic powers similarity measure. In contrast,
directly employ the mean square error based on the normalized vectors.
Additionally, conduct experiments with cos-cubed loss as Table A10 shown. Replacing Equa- tion 2 with cos-cubed loss does not impact the positive correlation between IIS and classifiability. which verifies the generality of conclusion.
A.2.3 EXPERIMENTS ADDITIONAL DATASETS & MORE COMPARISONS
19

---

Published as a conference paper at ICLR 2025
Table A10: IIS-accuracy relationship on CUB with the cos-cubed loss for concept vector acquisition.
Metrie
ResNet-18
ResNet-34
ResNet-50
ResNet-101
ResNet- 152
IIS 0.756 63.39
0.774 64.22
0.789 67.01
0.805 0.802 67.13
Acc@1
68.00
Table A11: IIS of ResNet- 18/50/152 on different datasets. Experiments are conducted with textual con- cept libraries from LaBo.
Table A12: Performance comparisons on different datasets. Experiments are conducted with textual concept libraries from LaBo.
Table A13:
Performance comparisons with LF-CBM different representations.
Model
ImageNet CUB
LF-CBM (ViT-B)
77.94
68.97
Model
UCF101 DTD HAM
Model UCFI01 DTD HAM
LF-CBM (VIT-L)
85.52
82.83
R8
0.949
0.934 0.912
50
0,967
0,940 0.914
LaBo 97.68
76.86 81.40
Ours (ViT-B)
78.50
69.14
R5sNet-152
0.976
0.957 0.915
Ours 97.79
77.22 81.90
Ours (ViT-L)
86.85
83.51
haapprettabiilyyCaassftaaabbiiii Relationship on Additional Datasets. To further illustrate the gen- erality of conclusions in Section 3.2 conduct experimental results addition datasets in- cluding UCF-101 (Soomro) 2012), DTD (Cimpoi et al. (014) and HAM10000 (Tschandl et al.) 2018). In Table A11 further verify the positive relationship between interpretability and classi- fiability on three datasets, indicating the strong generality of our conclusions.
More Performance Comparisons. Performance comparisons with LaBo (Yang et al. 2023)
3 additional datasets (Soomro) 2012, Cimpoi et al. (2014; Tschandl et al. 2018) are provided in Table A12 Our interpretable predictions demonstrate superior performance compared to LaBo. Furthermore, Table A13 presents the comparisons with LF-CBM (Oikarinen et al. 2023). a frame- work for providing interpretable predictions based on arbitrary vision representations. Our method outperforms LF-CBM using different pre- pre-trained representations.
Effects of Vision- LanguaLa Pre-training on Classifiability and Interpretability. In Table A15 we compute the IIS of ViT-B and CLIP-ViT-BB on the CUB-200 dataset. The experimental results remain consistent with our investigations Section 3.2] i.e., representations with higher classifiabil- ity exhibit enhanced interpretability. Representations pre-trained with large-scale datasets have stronger generalization ability than representations of the standard ViT, thereby re- sulting in improved classifiability on this dataset and subsequently enhancing their interpretability.
A.2.4 RELATIONSHIP BETWEEN IIS AND SEGMENTATION PERFORMANCE
The classification task serves as a fundamental component of vision pre-training- Enhanced clas- sifiability in representations often leads improved performance on downstream vision tasks (Liu al., 2021). For example, therefore, mentation performance. Given the pre-trained representations of ResNet50, we perform two sets of comparative experiments.
can derive positive correlation between IIS and seg-
(i) We directly fine-tune the representations (Origin). (ii) We first adjust the representations through IIS maximization Equation and then fine-tune them on segmentation tasks (Ours). As Table A14 shows, compared original represen- tations, enhancing the IIS of representations leads to an improvement in their classifiability (Acc.). The improvement leads to an increase the segmentation performance (mloU).
segmentation tasks
A.2.5 CONCEPT ACTIVATION VISUALIZATIONS
We visualize the activation locations of each concept through Grad-CAM and compare them with the input images. As Figure shows the activation regions of the concept are related the concept
20

---

Published as a conference paper at ICLR 2025
Input
Exercise
Input
A living
Aisles books
Input
Land vehicle
Snow
Input
Mountain
Bicycles
Input
Instrumentalists
bruss instrument
Input
Sand
A
Input
Icons
Clickable buttons
Input
Crispy crust
Roll shape
Figure All: Concept activation visualizations with pre-trained representations of VIT-L.
Table A14: Relation between Table A15: Comparisons Table A16:
Performance comparisons with DEAL on ImageNet- 1k.
IIS, accuracy on ImageNet-Ik and mloU on ADE20K.
between standard ViT and
CLIP-ViT on CUB.
Model
Acc.
mloU
Model
IIS
Acc@1
Model CLIP-RNSO CLIP-VIT-B/22
Origin 0.953 75.66
40.60
VIT-B
0.764
DEAL
70.0
70.8
Ours 0.958
77.36
42.35
CIT--VIT-B 0,808
80.6
Ours
70.8
73.0
semantic, thereby qualitatively supporting the trustworthiness of leveraging powerful pre-trained representations (e.g., ViT-L) for interpretable predictions.
A.2.6 SPARSITY-ARR CURVES
In this section, show the sparsity-ARR curves of more pre-trained representations in Figure A13 to better support our conclusions in Section 3.3 We observe a similar phenomenon to Figure demonstrating that for representations with fixed accuracy, the information loss increases when pro- jecting pre-trained representations into more human-understandable patterns.
A.2.7 ABLATION STUDIES OF VISUAL CONCEPTS
We conducted verification experiments to confirm the consistency of our findings from Figure 4 in the main text. These experiments encompassed different types of visual elements, including patches and segments. In Table A17 we present the relationship between representation accuracy and IIS computed using the End2End concept library aggregated from patches and segments respectively. The experimental results show the same phenomenon as Figure the main text. Moreover, we observe that segments achieve higher IIS than patches.
21

---

Published as a conference paper at ICLR 2025
Prototype
Cluster
End2End
Text
100
100
80
80
80
AB0 (80)
60
ARR (50
60
AED (50)
60
A0B (5)
40
40
40
20
W-F-B-16
20
VIT-3-16
VIT-B-16
20:
VI7--116
VIT-1-16
VIT-L-15
151-1-16
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.990
0.995
1.000
Spansity Ratio (x)
Sparsity Ratin 6x)
Sparsity Ratin (x)
Sparsity Ratio
Prukolype
Cluster
End2End
Text.
80
80
60
ARN (5)
AND (8)
60
ATR (%)
60
ARR
40
40
20
Swin 5
20
Soite S
Swis-T
20
Swin-S
Swin:T T
20
Swin-S
T
0
Swin-8
SWin H
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
0.990
0.995
1,000
Sparsity Notto
Sparsity Ratio
Spursity Ratio (x)
Spansity Ratio [x]
Prototype
Cluster
End2End
Text
80
80
100
60
60
80
80
ABD (%5
60
ADR (50
60
AIR
40
ARR
40
Com-NNNi-Tiy
CowsNeN0-Tig
40
Comsect Thy
20
CunyNeXt-Smna Stmalh
CorvNiett-lae
20
ComNeNil-Bae
20
20
CossNeCh Seall
Can-NeNi-Bac
ConvNeXt Large
Large
Large
ComoNeXi-l onge
0.0
0.5
1.0
0.0
0.5
1.0
0.5
1.0
1,000
Sparsity Hatio 1x)
Sparsity Ratio (s)
Sparsity (x)
Sparsity Ratio ())
Figure A12: More experiments about the sparsity-ARR curve of different pre-trained represtttios on ImageNet with four types of concept libraries.
Table A17: Ablation studies of different visual elements (Patches and Segments)
Visual Elements
ResNet-18
ResNet-34
ResNet-50
ResNet-101
ResNet- 152
Patches Segments
0.6833 0.6879
0.7021 0.7043
0.7210 0.7311
0.7231 0.7386
0.7654
0.7694
A.2.8 RELATIONSHIP BETWEEN SIMPLIFIED IIS AND ORIGINAL IIS
In Equation 8 maximize simplified version of IIS improve pre-trained representations. Ex- perimental results in Table A18 show that during the IIS maximization process, the representation accuracy and original IIS increase with the simplified version of IIS. This indicates that the maxi- mization of the simplified IIS can improve the original IIS.
Table A18: The accuracy (Acc.), simplified IIS, and original IIS of ResNet-50 finetuned on Ima- geNet with the IIS (Simplified) maximization as training objective.
Epoch
0
40
200
Acc.
0.7566
0.7644
0.7736
IIS (Simplified)
0.9654
0.9776
0.9805
IIS (Original)
0.9533
0.9573
0.9583
22

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL VISUALIZATION OF INTERPRETABLE PREDICTIONS
We provide more visualizations of interpretable predictions based on textual (Figure A14) and vi- sual (Figure A13) concepts on ImageNet. These were created using the same procedure Figure The concepts that have large contributions are relevant to the class. Additionally, Figure show- cases the model editing with human guidance by zeroing out the contribution of incorrect concepts such concept bottleneck models (Zarlenga et al. 2022 Oikarinen et al. 2023).
A.4 ADDITIONAL. RESULTS ON VIDEO PRE-TRAINED MODELS
Building upon the observed positive correlation between interpretability and classifiability of im- age representations, we investigate whether this phenomenon extends to video representations on the Kinetics-400 dataset (Carreira & Zisserman 2017). For video concept, utilize the YOLOv5 (Jocher, 2020) pre-trained on COCO (Lin et al. 2014) to extract visual-gra -rounddng ob- jects. These objects are aggregated as concepts with the strategy of the End2End concept library. By computing the IIS and prediction accuracy of representations from video pre-trained models (Car-
reira Zisserman] 2017 Feichtenhofer al. 2019 Bertasius et al., 2021; Fan al. 2021 Tong al.) [2222] Wang al. 2023; Li et al. 2023b) 2022b), observe similar phenomenon that
observed in image representations (i.e., the positive correlation between IIS and accuracy).
Stowiaa
115
0.775
0.750
Unlornee
0.750
0.775
0.800
0.825
0.850
0.875
Accuracy
Figure A13: The relationship between accuracy and IIS for video pre-trained representations on the Kinetics-400 dataset. The IIS is computed based on the End2End concept library.
A.5 DISCUSSION
A.5.1 RELATIONSHIP BETWEEN IIS AND INTERPRETATION QUALITY METRICS.
IIS measures the representation interpretability through the accuracy comparison between represen- tations and their corresponding interpretations. The formulation of IIS is similar the faithfulness
Sperio
Poctetiv ProcCvend Bustinor Derfdaec
ventove
peodle
H+411
Poctriie Auamsiee Kotgl- Coulamec
Prodite Cal Eolleee
123
Wiilgees Stuk
11.44
Coorge
Figure A14: More visualizations of our interpretable predictions based on textual concepts.
23

---

Published as a conference paper at ICLR 2025
GT: Entlebucher Sennenhund
Prototype
Cluster
End2End
1.8516
2.6715
2.9798
0.6958
0.7819
1.5526
Figure A15: Visualizations of our interpretable predictions based on visual concepts.
label: washing machine
prediction: dishwasher
prediction: washing machine
appliance
+36.29
dryer
+35.34
dryar
+18.82
uppli lippiance
+31.38
oven
+14.78
Wrong concepts
wusher
+16.18
dishes
-14.36
detergent
+11.75
freezer
+13.14
dirty laundry
+9.13
25
Concept contribations
Concept contributions
Figure A16: An example for model editing with human guidance. By zeroing out the contribution of incorrect concepts (e.g., "a oven"", the prediction changes from "dishwasher" to "washing machine".
metric. This metric measures the predictive capacity of interpretations evaluate how interpreta- tions can meaningful and faithfully explain the model's predictions (Sarkar et al. 2022) However, the IIS and faithfulness metric are essentially different follows:
Different purposes: IIS is proposed quantify the interpretability of pre-trained represen- tations (comparing different representations), while the faithfulness metric is designed to measure the quality interpretations (comparing different interpretation methods).
Different definitions: the definition of IIS encompasses interpretation accuracy, representa- tion accuracy, and interpretation sparsity, whereas the faithfulness metric solely focuses interpretation accuracy.
A.5.2 RELATIONSHIP BETWEEN IIS AND CONCEPT BOTTLENECK MODELS.
The prediction based on interpretations of IIS shares the same formulation as concept bottleneck models (CBMs) (Koh et 2020 Zarlenga et al. 2022 Oikarinen al. 2023 Yan et 2023). which first identify fine-grained grained concepts for an input image and then make predictions based on identified concepts. We draw inspiration from concept bottleneck models in constructing concept libraries and the emphasis on interpretation sparsity. However, IIS primarily focuses on the inter- pretability measurement of pre-trained representations, while CBMs concentrate on improving the performance of concept- based predictions by designing new architecture (Zarlenga et al. (2022) and loss functions (Xu et al., 2023). Despite different focuses, our investigation contributes to the CBM community by revealing their strong compatibility with the development of pre-trained representa- tions. This is because higher accuracy representations possess more interpretable semantics, which can be effectively captured by CBMs built upon them.
A.5.3 COMPARISONS BETWEEN IIS AND VLG-CBM
VLG-CBM (rriaastaaa al. 2024 also obtains accuracies of concept-based predictions with dif- ferent sparsity levels. These accuracies are averaged to evaluate the ability of CBMs to provide accurate predictions with different numbers of concepts.
24

---

Published as a conference paper at ICLR 2025
The sole similarity between IIS and this metric is assessing the accuracies of varying sparsity levels. The dissimilarities include three aspects. (i) Objective. The objective IIS is to quantify the interpretability of arbitrary vision pre- pre-trained representations. The average accuracy in VLG-CBM serves as a metric for evaluating the performance of CBMs, a specific type of interpretable model. (ii) Sparsification. Our sparsification mechanism prediction during forward propagation. VLG-CBM adjusts the number of concepts by manipulating the sparsity regularization strengths in the loss function, which requires complex strategies including gradually reducing the regularization strengths during training and post- post-training weight pruning. (iil) Metric Definition. IIS defined as the area under the curve formed by the sparsity and accuracy. The metric in VLG-CBM is solely defined based on the average accuracy at different sparsity levels.
accurately control the number of concepts for
A.5.4 DISCUSSIONS WITH DEAL AND OTHER SIMILAR METHODS
Recently, DEAL LL al. 2025) has observed that improving interpretability have positive impact on classification performance. For the performance, as Table A16 shows, our method out- performs DEAL ImageNet with the same pre-trained representations. Moreover, our methods are different from DEAL in three aspects.
(i) Training strategy. DEAL is specifically designed for the CLIP architecture and employs trastive learning methods to fine-tune the entire model. Our method can be applied to arbitrary architectures and only fine-tune the visual encoder. (ii) Prediction mechanism. DEAL predicts based on the average embedding similarity between the image and concepts within the category. Our method inputs the image representations -iteeabbiityycllaaffaaiiityy relationship. DEAL solely observed qualitative impact of inter- pretability on classifiability CLIP. In comparison, we find the mutually promoting relationship between both factors on a wide range of models in a quantitative manner.
prediction head without the text encoder. (ii)
Additionally, apart from DEAL, other approaches (Zhao et al., 2024] Gandelsman al., 2023) have also demonstrated an enhancement in classifiability through the improvement of interpretability. Compared with these methods, our method may have similar motivations exploring the relation- ship between interpretability and classifiability.
However, our contributions are different from these methods. (i) Methods. In contrast to these papers that focus designing interpretation methods, we propose "a general metric" of representation in- terpretability to "quantitatively investigate" the interpretability-caan -laaiititccassiiaabbittt relationship. (i) Inves- rigation results. While previous approaches have observed that enhancing interpretability improves classifiability -itpreebbbbity-ccaaaiaaaiiltyl we have discovered "a mutually promoting relation- Nevertheless, incorporating interpretability enhancements from these methods into IIS raises an interesting direction for further explorations.
ship" p
A.5.5 LIMITATION
Despite revealing the mutual-promotte relationship between representation interpretability and ac- curacy, investigations of the underlying mechanism leading to this relationship have not been con- ducted. In addition, we fine-tune pre-trained representations by maximizing just a simplified version of IIS to improve their accuracy. For a deeper understanding of the consistency between inter- pretability and accuracy, we anticipate future work to investigate the underlying mechanisms of our observations, and better integrate IIS into the training process of mr-ttrined models.
25


Reviews:
[{"review_id": "O75aspHysD", "reviewer": "Reviewer_xfXM", "summary": "This paper proposes the Inherent Interpretability Score (IIS) as a novel quantitative metric for evaluating the interpretability of the pre-trained representation. The authors demonstrate the positive correlation between the interpretability and classifiability of the classifiability-oriented pre-trained representations through extensive experiments.", "questions": "1. How do you deal with the linear weights in the additional linear classifier, as shown in Eq. (5)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors proposed a novel evaluation metric to quantitatively evaluate the pre-trained representation's interpretability. The proposed IIS can not only be used in the research area related to post-hoc explainable approaches but also improve model classification performance and interoperability. \n2. The paper is written and well-organized. It is easy to follow the authors' ideas and understand their approaches. The authors clearly show their motivation and ideas in Figure 1 and Figure 2. The notations and experimental results are clear and easy to read.\n3. The authors have conducted extensive experiments across various datasets with different pre-trained models to show the effectiveness of IIS. The experimental results support the authors' claims from several different aspects, for example, the positive correlation, factor analysis,  and applications of IIS.", "weaknesses": "1. Besides extensive experimental results, the authors should have provided clear theoretical analyses of the proposed method.\n2. The proposed method IIS relies on pre-defined or generated concepts, which are usually hard to get and not applicable in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QAXeMLkTvz", "reviewer": "Reviewer_Hosp", "summary": "This paper addresses the critical question of whether high interpretability and classifiability can be achieved simultaneously in pre-trained vision models. To this end, the authors propose the Inherent Interpretability Score (IIS), a novel metric designed to quantitatively evaluate the interpretability of pre-trained representations. Through extensive experimentation, the authors find a positive correlation between interpretability and classifiability. Moreover, they observe that maximizing interpretability can further enhance model performance, which, in turn, improves the performance of interpretability-based models.", "questions": "My questions are listed in the Weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* This paper utilizes concept ablation method for evaluation the interpretability of the representations, which has also been adopted for evaluating the post-hoc interpretation methods [1].\n* The authors utilize both visual and textual concepts for IIS.\n* The paper is well-written and easy to follow.\n* The authors provide code for reproducibility check.", "weaknesses": "1. **Completeness of Evaluation:** Huang et al. [2] have discussed the trustworthiness in concept bottleneck models (CBMs), noting that several concepts can be activated in irrelevant regions with regards to the input image, leading to a loss of model interpretability. However, the authors have overlooked this issue.\n2. **Method:** In the IIS evaluation, in what order are concepts removed? How would the evaluation results differ if removing began with the most important concepts for the category versus the least important?\n3. **Experiments:** If the visual model’s features have already been aligned with language, as in CLIP [3], would the interpretability be significantly higher compared to a standard ViT?\n\n> [1] Hooker, Sara, et al. \"A benchmark for interpretability methods in deep neural networks.\" Advances in neural information processing systems 32 (2019).\n> \n> [2] Huang, Qihan, et al. \"On the Concept Trustworthiness in Concept Bottleneck Models.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.\n> \n> [3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CscHyIzseC", "reviewer": "Reviewer_AruE", "summary": "This paper proposes the IIS metric to measure the interpretability of representations from image encoders and demonstrates a positive correlation between the interpretability and classifiability of the classifiability-based pre-trained representations.", "questions": "Please see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper quantifies the interpretability of representations and discovers an interesting and significant positive correlation between the interpretability and classifiability of representations. The results have been demonstrated on multiple datasets and architectures.\n2. The observation is used in designing IIS maximization loss, which results in improved classifiability. \n3. The paper is well-written, making it easy to follow the main ideas and claims.", "weaknesses": "1. The proposed IIS score is similar to the “average accuracy obtained at different number of effective concepts” proposed in [1]. Particularly, [1] trains the final linear classifier with elastic-net regularization in GLM-SAGA and tunes regularization strength for controlling sparsity (number of effective concepts). The average accuracy is obtained by averaging accuracy at different sparsity levels. The similarities between the metric proposed in [1] and IIS should be discussed.\n2. Limited dataset results: To demonstrate the applicability of this method in wide scenarios, the results should be provided on additional datasets, including Actions: UCF-101, Textures: DTD, and Skin tumors: HAM10000 following [2].\n3. Table 5 lacks comprehensive comparisons to methods [3, 4]. Particularly, [3] can work for any backbone network architecture, including ViT and Swin.\n\n[1] Srivastava, Divyansh, Ge Yan, and Tsui-Wei Weng. VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance. arXiv preprint arXiv:2408.01432 (2024).\n\n[2] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023.\n\n[3] Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, ICLR 2023\n\n[4] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vfy5245sHy", "reviewer": "Reviewer_d5fZ", "summary": "This paper introduces the Inherent Interpretability Score (IIS) to quantify interpretability within pre-trained visual models, establishing a positive correlation between classifiability and interpretability. The study shows that interpretability-oriented fine-tuning can further improve both interpretability and classifiability in vision models. Extensive evaluations across various models and datasets support these claims, positioning IIS as a metric for balancing classifiability and interpretability without sacrificing accuracy.", "questions": "1. While the paper uses segmentation as visual concepts, could the authors provide insights on whether IIS correlates with improvements in segmentation tasks?\n2. In Table 4, compared with ECBM, your method improves the accuracy from 81.20% to 81.29% appears minimal. Could the authors provide a comparison between ECBM and your method when both use approximately 250 concepts?\n3. Instead of concatenating both concepts together, could authors provide a comparison between using textual concepts only and visual concepts only? \n4. While DEAL [1] also claimed that improving interpretability can have positive impact on classification performance too, could author provide a comparison results between DEAL and your method with regard to the classification accuracy improvement.\n\n[1] DEAL: Disentangle and Localize Concept-level Explanations for VLMs http://arxiv.org/abs/2407.14412", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an innovative metric IIS, which provides a practical way to assess interpretability through classifiability and shows that pre-trained representations can achieve high interpretability and classifiability simultaneously.\n\n2. Comprehensive experiments conducted have confirmed the positive correlations between interpretability and classifiability. In the meantime, authors have done thoroughly comparison experiments, showing the broad application direction in classification task.", "weaknesses": "1. The scope of this paper is limited to classifiability-oriented representations, where the fact that the representation can be applied to segmentation task is ignored.\n\n2. There is limited discussion about the selection of textual concepts. The prompts used to generate concepts for each class appear to be basic, without addressing whether overlapping concepts might influence the results.\n\n3. Compared to ECBM, proposed method achieves minimal improvement from 81.20% to 81.29%. Further comments will be listed below in Question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KQXENkrkhb", "reviewer": "Reviewer_511b", "summary": "The authors propose to quantify the interpretability of representations using the correlation with the ration of interpretable semantics. They propose the Inherent Interpretability Score to measure the interpretability and information loss of representations by checking how much a representation can retain the original accuracy after a projection into a sparse interpretable space and a sparsification process. Through this evaluation, they also analyze the correlation between classifiability and interpretability.", "questions": "I included some questions in the weaknesses section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "→ Proposition of an interpretability metric: I believe this is an important contribution to evaluate the interpretability of a model's representation. We know that given the subjectivity of interpretability, it is difficult to quantify this property. The authors present a consistent way to measure it using libraries of concepts and the projection of representations into concept space.\n\n→ Testing of different concepts and validation through a variety of experiments: I appreciate the number of validation experiments, including 4 different concept libraries (visual and textual) and experiments to evaluate: accuracy vs. interpretability for different architectures, evolution through training epochs, sparsity ratio and representation dimension.\n\n→ Two applications: The authors also provide two cases where their study can be applied, improving classifiability by maximizing interpretability and providing interpretable predictions with high accuracy.", "weaknesses": "→ I understand the positive correlation between interpretability and classifiability, but is it not too strong to assume that higher classifiability implies more interpretability in this part?:  “We discover the mutual promoting relation between the interpretability and classifiability\nof the classifiability-oriented representations, indicating that improving the interpretability can further improve the classifiability, and representations with higher classifiability are more interpretable to preserve more task-relevant semantics in interpretations.”\n\nAs I understand it: There is a loss of information caused by the sparcity, which reduces the original accuracy.\n    \n→  I am not sure about the process of increasing sparsity. Could some of the zeroed concepts be some kind of sub-concept that gets diluted in the high dimensionality of the concept library? Or even the errors from obtaining the concept library? It would be interesting to see other ways to reduce the sparsity of these concepts.\n\n→  In Figure 5, you can see that the model is interpretable at the beginning of training and then it loses interpretability. Why is the model interpretable when it knows nothing? The network was initialized with some pre-trained weights? Because if the weights are random, it makes no sense to be interpretable at the beginning. Do you know why this happens?\n\n→  I think it would be important to include the standard deviation in Tables 1— 5.\n\n→  I have not seen much difference in increasing sparsity to justify this sentence: “This indicates that the model is highly sensitive to the representation sparsity and should not employ excessively large sparsity constraints during fine-tuning.”\n\nAnd another question about sparsity: if you are not able to provide a sparser representation, you are limiting the interpretability, right? Because 200 concepts with 0.1 sparsity is a lot to interpret I imagine.\n\n→  I think Figure 1 would be more interesting if it were made with real data and real experiments (even with simplified results).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DTGkCwLArt", "reviewer": "Reviewer_wUp3", "summary": "The paper proposed a novel representation interpretability metric called IIS. A series of experiments are conducted which shows positive correlation between IIS and classifierbility of the representation. This observation is applied in improving classifierbility of representation and reduce accuracy loss for interpretation.", "questions": "1. How is Eq. (2) derived? It looks like the cos-cubed loss in [1]. More discussion on the details is needed. \n2. In sparsification in Eq.(4), besides zeroing out elements, the remaining elements are also adjusted by their distance to threshold. Is there any justification to this sparsification? How will the IIS change for different sparsification method (e.g. only masking out small absolute value elements without changing others).\n3. In Table 4, how is the accuracy for your method calculated? What is the sparity ratio $s$ used here?\n\n[1] Oikarinen, Tuomas, et al. \"Label-free concept bottleneck models.\" arXiv preprint arXiv:2304.06129 (2023).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed IIS metric provides an interesting way to measure representation interpretability. \n2. The observation on relation between interpretability and classifiability is novel and backed by extensive experiments.", "weaknesses": "1. Though the idea of IIS intuitively makes sense, more theoretical discussion on this metric is needed. For example, suppose there is a random representation where the prediction accuracy no better than random guess. In this case, the representation will have low classifiability while high IIS (ARR = 1 as the accuracy will not further decrease after sparsification).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)