Enhancing Pre-trained Representation Classifiability can Boost its Interpretability

Authors: Shufan Shen, Zhaobo Qi, Junshu Sun, Qingming Huang, Qi Tian, Shuhui Wang
Primary area: interpretability and explainable AI
Keywords: representation interpretability, vision representations, image understanding

Abstract:
The visual representation of a pre-trained model prioritizes the classifiability on downstream tasks, while the widespread applications for pre-trained visual models have posed new requirements for representation interpretability. However, it remains unclear whether the pre-trained representations can achieve high interpretability and classifiability simultaneously. To answer this question, we quantify the representation interpretability by leveraging its correlation with the ratio of interpretable semantics within the representations. Given the pre-trained representations, only the interpretable semantics can be captured by interpretations, whereas the uninterpretable part leads to information loss. Based on this fact, we propose the Inherent Interpretability Score (IIS) that evaluates the information loss, measures the ratio of interpretable semantics, and quantifies the representation interpretability. In the evaluation of the representation interpretability with different classifiability, we surprisingly discover that the interpretability and classifiability are positively correlated, i.e., representations with higher classifiability provide more interpretable semantics that can be captured in the interpretations. This observation further supports two benefits to the pre-trained representations. First, the classifiability of representations can be further improved by fine-tuning with interpretability maximization. Second, with the classifiability improvement for the representations, we obtain predictions based on their interpretations with less accuracy degradation. The discovered positive correlation and corresponding applications show that practitioners can unify the improvements in interpretability and classifiability for pre-trained vision models. Codes are available at https://github.com/ssfgunner/IIS.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730392817026

thread_type:
official_review

<thread_object>
creator: reviewer_wup3

<content>
summary:
The paper proposed a novel representation interpretability metric called IIS. A series of experiments are conducted which shows positive correlation between IIS and classifierbility of the representation. This observation is applied in improving classifierbility of representation and reduce accuracy loss for interpretation.

strengths:
1. The proposed IIS metric provides an interesting way to measure representation interpretability. 
2. The observation on relation between interpretability and classifiability is novel and backed by extensive experiments.

weaknesses:
1. Though the idea of IIS intuitively makes sense, more theoretical discussion on this metric is needed. For example, suppose there is a random representation where the prediction accuracy no better than random guess. In this case, the representation will have low classifiability while high IIS (ARR = 1 as the accuracy will not further decrease after sparsification).

questions:
1. How is Eq. (2) derived? It looks like the cos-cubed loss in [1]. More discussion on the details is needed. 
2. In sparsification in Eq.(4), besides zeroing out elements, the remaining elements are also adjusted by their distance to threshold. Is there any justification to this sparsification? How will the IIS change for different sparsification method (e.g. only masking out small absolute value elements without changing others).
3. In Table 4, how is the accuracy for your method calculated? What is the sparity ratio $s$ used here?

[1] Oikarinen, Tuomas, et al. "Label-free concept bottleneck models." arXiv preprint arXiv:2304.06129 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730454994791

thread_type:
official_review

<thread_object>
creator: reviewer_511b

<content>
summary:
The authors propose to quantify the interpretability of representations using the correlation with the ration of interpretable semantics. They propose the Inherent Interpretability Score to measure the interpretability and information loss of representations by checking how much a representation can retain the original accuracy after a projection into a sparse interpretable space and a sparsification process. Through this evaluation, they also analyze the correlation between classifiability and interpretability.

strengths:
→ Proposition of an interpretability metric: I believe this is an important contribution to evaluate the interpretability of a model's representation. We know that given the subjectivity of interpretability, it is difficult to quantify this property. The authors present a consistent way to measure it using libraries of concepts and the projection of representations into concept space.

→ Testing of different concepts and validation through a variety of experiments: I appreciate the number of validation experiments, including 4 different concept libraries (visual and textual) and experiments to evaluate: accuracy vs. interpretability for different architectures, evolution through training epochs, sparsity ratio and representation dimension.

→ Two applications: The authors also provide two cases where their study can be applied, improving classifiability by maximizing interpretability and providing interpretable predictions with high accuracy.

weaknesses:
→ I understand the positive correlation between interpretability and classifiability, but is it not too strong to assume that higher classifiability implies more interpretability in this part?:  “We discover the mutual promoting relation between the interpretability and classifiability
of the classifiability-oriented representations, indicating that improving the interpretability can further improve the classifiability, and representations with higher classifiability are more interpretable to preserve more task-relevant semantics in interpretations.”

As I understand it: There is a loss of information caused by the sparcity, which reduces the original accuracy.
    
→  I am not sure about the process of increasing sparsity. Could some of the zeroed concepts be some kind of sub-concept that gets diluted in the high dimensionality of the concept library? Or even the errors from obtaining the concept library? It would be interesting to see other ways to reduce the sparsity of these concepts.

→  In Figure 5, you can see that the model is interpretable at the beginning of training and then it loses interpretability. Why is the model interpretable when it knows nothing? The network was initialized with some pre-trained weights? Because if the weights are random, it makes no sense to be interpretable at the beginning. Do you know why this happens?

→  I think it would be important to include the standard deviation in Tables 1— 5.

→  I have not seen much difference in increasing sparsity to justify this sentence: “This indicates that the model is highly sensitive to the representation sparsity and should not employ excessively large sparsity constraints during fine-tuning.”

And another question about sparsity: if you are not able to provide a sparser representation, you are limiting the interpretability, right? Because 200 concepts with 0.1 sparsity is a lot to interpret I imagine.

→  I think Figure 1 would be more interesting if it were made with real data and real experiments (even with simplified results).

questions:
I included some questions in the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730496988940

thread_type:
official_review

<thread_object>
creator: reviewer_d5fz

<content>
summary:
This paper introduces the Inherent Interpretability Score (IIS) to quantify interpretability within pre-trained visual models, establishing a positive correlation between classifiability and interpretability. The study shows that interpretability-oriented fine-tuning can further improve both interpretability and classifiability in vision models. Extensive evaluations across various models and datasets support these claims, positioning IIS as a metric for balancing classifiability and interpretability without sacrificing accuracy.

strengths:
1. This paper proposes an innovative metric IIS, which provides a practical way to assess interpretability through classifiability and shows that pre-trained representations can achieve high interpretability and classifiability simultaneously.

2. Comprehensive experiments conducted have confirmed the positive correlations between interpretability and classifiability. In the meantime, authors have done thoroughly comparison experiments, showing the broad application direction in classification task.

weaknesses:
1. The scope of this paper is limited to classifiability-oriented representations, where the fact that the representation can be applied to segmentation task is ignored.

2. There is limited discussion about the selection of textual concepts. The prompts used to generate concepts for each class appear to be basic, without addressing whether overlapping concepts might influence the results.

3. Compared to ECBM, proposed method achieves minimal improvement from 81.20% to 81.29%. Further comments will be listed below in Question section.

questions:
1. While the paper uses segmentation as visual concepts, could the authors provide insights on whether IIS correlates with improvements in segmentation tasks?
2. In Table 4, compared with ECBM, your method improves the accuracy from 81.20% to 81.29% appears minimal. Could the authors provide a comparison between ECBM and your method when both use approximately 250 concepts?
3. Instead of concatenating both concepts together, could authors provide a comparison between using textual concepts only and visual concepts only? 
4. While DEAL [1] also claimed that improving interpretability can have positive impact on classification performance too, could author provide a comparison results between DEAL and your method with regard to the classification accuracy improvement.

[1] DEAL: Disentangle and Localize Concept-level Explanations for VLMs http://arxiv.org/abs/2407.14412
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701717839

thread_type:
official_review

<thread_object>
creator: reviewer_arue

<content>
summary:
This paper proposes the IIS metric to measure the interpretability of representations from image encoders and demonstrates a positive correlation between the interpretability and classifiability of the classifiability-based pre-trained representations.

strengths:
1. This paper quantifies the interpretability of representations and discovers an interesting and significant positive correlation between the interpretability and classifiability of representations. The results have been demonstrated on multiple datasets and architectures.
2. The observation is used in designing IIS maximization loss, which results in improved classifiability. 
3. The paper is well-written, making it easy to follow the main ideas and claims.

weaknesses:
1. The proposed IIS score is similar to the “average accuracy obtained at different number of effective concepts” proposed in [1]. Particularly, [1] trains the final linear classifier with elastic-net regularization in GLM-SAGA and tunes regularization strength for controlling sparsity (number of effective concepts). The average accuracy is obtained by averaging accuracy at different sparsity levels. The similarities between the metric proposed in [1] and IIS should be discussed.
2. Limited dataset results: To demonstrate the applicability of this method in wide scenarios, the results should be provided on additional datasets, including Actions: UCF-101, Textures: DTD, and Skin tumors: HAM10000 following [2].
3. Table 5 lacks comprehensive comparisons to methods [3, 4]. Particularly, [3] can work for any backbone network architecture, including ViT and Swin.

[1] Srivastava, Divyansh, Ge Yan, and Tsui-Wei Weng. VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance. arXiv preprint arXiv:2408.01432 (2024).

[2] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023.

[3] Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, ICLR 2023

[4] An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, 2023.

questions:
Please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730805104462

thread_type:
official_review

<thread_object>
creator: reviewer_hosp

<content>
summary:
This paper addresses the critical question of whether high interpretability and classifiability can be achieved simultaneously in pre-trained vision models. To this end, the authors propose the Inherent Interpretability Score (IIS), a novel metric designed to quantitatively evaluate the interpretability of pre-trained representations. Through extensive experimentation, the authors find a positive correlation between interpretability and classifiability. Moreover, they observe that maximizing interpretability can further enhance model performance, which, in turn, improves the performance of interpretability-based models.

strengths:
* This paper utilizes concept ablation method for evaluation the interpretability of the representations, which has also been adopted for evaluating the post-hoc interpretation methods [1].
* The authors utilize both visual and textual concepts for IIS.
* The paper is well-written and easy to follow.
* The authors provide code for reproducibility check.

weaknesses:
1. **Completeness of Evaluation:** Huang et al. [2] have discussed the trustworthiness in concept bottleneck models (CBMs), noting that several concepts can be activated in irrelevant regions with regards to the input image, leading to a loss of model interpretability. However, the authors have overlooked this issue.
2. **Method:** In the IIS evaluation, in what order are concepts removed? How would the evaluation results differ if removing began with the most important concepts for the category versus the least important?
3. **Experiments:** If the visual model’s features have already been aligned with language, as in CLIP [3], would the interpretability be significantly higher compared to a standard ViT?

> [1] Hooker, Sara, et al. "A benchmark for interpretability methods in deep neural networks." Advances in neural information processing systems 32 (2019).
> 
> [2] Huang, Qihan, et al. "On the Concept Trustworthiness in Concept Bottleneck Models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 19. 2024.
> 
> [3] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

questions:
My questions are listed in the Weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731038923289

thread_type:
official_review

<thread_object>
creator: reviewer_xfxm

<content>
summary:
This paper proposes the Inherent Interpretability Score (IIS) as a novel quantitative metric for evaluating the interpretability of the pre-trained representation. The authors demonstrate the positive correlation between the interpretability and classifiability of the classifiability-oriented pre-trained representations through extensive experiments.

strengths:
1. The authors proposed a novel evaluation metric to quantitatively evaluate the pre-trained representation's interpretability. The proposed IIS can not only be used in the research area related to post-hoc explainable approaches but also improve model classification performance and interoperability. 
2. The paper is written and well-organized. It is easy to follow the authors' ideas and understand their approaches. The authors clearly show their motivation and ideas in Figure 1 and Figure 2. The notations and experimental results are clear and easy to read.
3. The authors have conducted extensive experiments across various datasets with different pre-trained models to show the effectiveness of IIS. The experimental results support the authors' claims from several different aspects, for example, the positive correlation, factor analysis,  and applications of IIS.

weaknesses:
1. Besides extensive experimental results, the authors should have provided clear theoretical analyses of the proposed method.
2. The proposed method IIS relies on pre-defined or generated concepts, which are usually hard to get and not applicable in real-world scenarios.

questions:
1. How do you deal with the linear weights in the additional linear classifier, as shown in Eq. (5)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732104298575

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their diligent efforts and valuable suggestions, which have greatly contributed to improving the quality of our manuscript.

**Summary of strengths**:

We sincerely appreciate that you find our method:

- novel and interesting (reviewers xfXM, AruE, d5fZ, wUp3, and 7e64);

- provide extensive evaluations and convincing results (reviewers xfXM, Hosp, AruE, d5fZ, 511b, and wUp3);

- show broad application direction (reviewers xfXM, d5fZ, and 511b);

- well-organized and well-written (reviewers xfXM, Hosp, and AruE);

- provide a quantitive evaluation for interpretability (reviewers xfXM, Hosp, AruE, d5fZ, 511b, and wUp3);

- help improve classifiability (reviewers xfXM, AruE, and 511b).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the tradeoffs between interpretability and classifiability, finding empirically a strong correlation. For this, they propose a new evaluation metric that is usable for both explainability and improving classification performance. The reviewers unanimously agree that this is a good paper and should be accepted. The main weakness is the limited theoretical explanation for this phenomenon, but all the reviewers agree that the experimental results are convincing.