Escaping Low-Rank Traps: Interpretable Visual Concept Learning via Implicit Vector Quantization
Authors: Shujian Gao
Yuan Wang
Chenglong Ma
Xin Gao
Jiangtao Yan
Junzhi Ning
Cheng Tang
Changkai Ji
Huihui Xu
Wei Li
Ziyan Huang
Jiashi Lin
Ming Hu
Jiyao Liu
Wenhao Tang
Ye Du
Tianbin Li
Jin Ye
Junjun He
Venue: ICLR 2026 Poster (2026)
Keywords: Concept Bottleneck Models; Visual Concept Learning; Vision-Language Models; Representational Collapse; Interpretability; Cross-modality Alignment
Primary area: interpretability and explainable AI
TL;DR: We utilize implicit vector quantisation to address the Representational Collapse issue, while delivering better visual concept learning.

Abstract: Concept Bottleneck Models (CBMs) achieve interpretability by interposing a human-understandable concept layer between perception and label prediction. We first identify that the condition of \textit{many-to-many} mapping is necessary for robust CBMs, a prerequisite that has been largely overlooked in previous approaches. While several recent methods have attempted to establish this relationship, we observe that they suffer from the fundamental issue of \textit{representation collapse}, where visual patch features degenerate into a low-rank subspace during training, severely degrading the quality of learned concept activation vectors, thus hindering both model interpretability and downstream performance. To address these issues, we propose Implicit Vector Quantization (IVQ), a lightweight regularizer that maintains high-rank, diverse representations throughout training. Rather than imposing a hard bottleneck via direct quantization, IVQ learns a codebook prior that anchors semantic information in visual features, allowing it to act as a proxy objective. To further exploit these high-rank concept-aware features, we propose Magnet Attention, which dynamically aggregates patch-level features into visual concept prototypes, explicitly modeling the many-to-many vision–concept correspondence. Extensive experimental results show that our approach effectively prevents representational collapse and achieves state-of-the-art performance on diverse benchmarks. Our experiments further probe the low-rank phenomenon in representational collapse, finding that IVQ mitigates the information bottleneck and yields cross-modal representations with clearer, more interpretable consistency. Code is available at \url{https://github.com/Daryl-GSJ/IVQ-CBM}.

Published as a conference paper at ICLR 2026
ESCAPING LOW-RANK TRAPS: INTERPRETABLE VI- SUAL CONCEPT LEARNING VIA IMPLICIT VECTOR QUANTIZATION
Shujian G Yuan Wang', Chenglong Ma Xin Gao Jiangtao Yan², Junzhi Ning',
Cheng Tang?, Changkai Ji Huihui Xu², Wei LP, Ziyan Huang", Jiashi Lin', Ming Hu?,
Jiyao Liu?, Wenhao Tang', Ye Du², Tianbin Jin Ye', Junjun He
Fudan University
SShangaai AI Laboratory
*Zhejiang University
ABSTRACT
Concept Bottleneck Models (CBMs) achieve interpretability by interposing a human-underssanddbll concept layer between perception and label prediction. We first identify that the condition of many-to-many mapping is necessary for robust CBMs, prerequisite that has been largely overlooked in previous approaches. While several recent methods have attempted to establish this relationship, we observe that they suffer from the fundamental issue of representation collupse, where visual patch features degenerate into a low-rank subspace during training, severely degrading the quality of learned concept activation vectors, thus hinder- ing both model interpretability and downstream performance. To address these issues, propose Implicit Vector Quantization (IVQ), lightweight regularizer that maintains high-rank, diverse representations throughout training. Rather than imposing hard bottleneck via direct quantization, IVQ learns codebook prior that anchors semantic information in visual features, allowing it to act as proxy objective. To further exploit these high-rank concept-aware features, we propose Magnet Attention, which dynamically aggregates patch-level features into visual concept prototypes, explicitly modeling the many-to-many lisiee-concept cor- respondence. Extensive experimental results show that our approach effectively prevents representational collapse and achieves state-of-the-arr performance on diverse benchmarks. Our experiments further probe the low-rank phenomenon in representational collapse, finding that IVQ mitigates the information bottleneck and yields cross-modal representations with clearer, more interpretable consis- tency. Code is available at
INTRODUCTION
Explainable Artificial Intelligence (xAl) aims to embed neural networks with buman-interppetabbee and interactive reasoning processes, thereby opening the black box of end-to-end prediction systems. Among the ante-hoc xAI methods, the Concept Bottleneck Model (CBM) (Koh et al., 2020) is prominent approach that pipelines predictions through an intermediate concept layer. This layer, situated between perceptual encoder and final task head, forces the model to operate in two distinct stages: First, a perception stage maps inputs set of predefined semantic concepts (e.g., shupe of beak or spatial exient of a lesion). Second, reasoning stage uses only these concept activation vectors (CAVs) to make the final decision.
In the two-stage learning, perception and modeling of CAVs in the initial stage is foundational. The core of this stage lies in constructing a cross-modal patch-concen alignment process, which compels
"This work was completed during my internship at Shanghai Al Lab. Corresponding author.

---

Published as a conference paper at ICLR 2026
(a) Comparsion of Previous Method
(b) The Problem: Representational Collapse
0.6 Micro aneuryams
neovosculorization
0.4
240
0.2 Intraretinal Hemorrhages
0.31
Feanuee Gank
120
Neavascul- ariza fion
200
0.2
m
Petch Concepts
Concept Patches
Bank Collapna Region
NO
(Tain)
Previous Methods
20
[Whole Picnure Finherdling)
Ree/ee backbone
(c) Baseline Rank Feature Comparisons
[CLS]
Our Methed
240
120
100
20
Vision Transformer
Ours
Expli:D
[CS]
Myp
20-
DOT
C Patch Emhedding]]
[Human Concept]
102
Epoch
Figure I: (a) illustration of many- Io-many cross-modal alignment in CBMs. (b) Comparison feature rank dynamics during training on the training and validation sets, with and without IVQ. (c) Feature rank dynamic comparisons with previous baselines.
the model to learn the mapping and disentanglement of raw, high-dimensional visual embeddings into a set of structured visual representation vectors corresponding to human-defined concepts. As illustrated in Figure 1a, an intrinsic many-to-many correspondence exists between local visual fea- tures and high level semantic concepts within the cross-modal concept alignment. The relationship two-fold: an individual image patch may map to multiple concepts, while concurrently, the visual representation of single concept is distributed across several distinct image patches. However, pre- vious methods such as (Yang et al., 2023; Yuksekgonul al., 2023; Oikarinen et al., 2023; Sheth & Kahou, 2023) employ visual encoder to extract a single global embedding for concept alignment. Due to the limited expressiveness of a single visual feature, these approaches fail explicitly model the complex many-to-many mapping between local features and high-level concepts.
However, when we attempt explicitly construct this relationship. a critical phenomenon termed representational collapse emerges, severely impeding the process. As presented in Figure 1b, we tracked the rank the patch feature matrix throughout the training; the rank undergoes sharp decline during the initial epochs both training and validation sets, eventually bottoming out at rank 70, a drastic reduction from the potential full rank of 196. This behavior is not an anomaly; shown in Figure 1c, concurrent works such ExplicD (Gao al., 2024) and MVP-CBM (Wang et al., 2025) exhibit a similar pattern, suggesting that representational collapse obstacle this domain. Meanwhile, its essence is symptom of more fundamental problem: the collapse of feature diversity. This collapse is particularly damaging for CBMs, as a degenerated representation space lacks the expressive capacity encode diverse of concepts (Sansone al., 2025). When visual feature embeddings become highly similar and informationally redundant, the learning process itself is confounded (Jing et al., 2022), directly inhibiting the formation of high- quality, disentangled visual features, which are the foundation of the CBM.
fundamental
Hence, we introduce Implicit Vector Quantization (IVQ), a novel regularizer that repurposes the VQ objective as loss term without quantizing the forward pass. The loss effectively forms semantic bridge, compelling each patch feature to align with the nearest learned codebook prototype. Collec- tively, these prototypes act distinct anchors that prevent the feature distribution from collapsing into a degenerate subspace. As depicted Figure 1b and c, this directly counters representation col- lapse by maintaining stable and elevated feature rank throughout training. Furthermore, building upon concept- aware, well-structured visual features, we introduce a Magnet Attention mechanism, which dynamically aggregates the diverse patch-level features into holistic, semantically coher-
2

---

Published as a conference paper at ICLR 2026
ent visual representation prototype for each pre-defined textual concept, effectively modeling the many-to-many mapping.
This work makes three key contributions:
We identify that the key to CBMs lies in modeling the many-to- relationship between concepts and patches. Furthermore, we identify and analyze representational collapse, a key challenge in training modern CBMs that hinders the establishment of CAVs.
We propose IVQ, a novel regularization method that preserves feature diversity and pre- vents representational collapse without creating an information bottleneck. To exploit ob- tained rich representations, we introduce the Magnet Attention mechanism to effectively aggregate the regularized patch features into semantically meaningful concept prototypes.
Extensive experiments on eight benchmarks demonstrate that our IVQ-CBM consistently outperforms eight strong baselines, achieving state-of-the-art accuracy and learning better interpretable representations consistent with textual concepts.
RELATED WORK
Concept Bottleneck Model. It achieves interpretabiiit--y--eisi by introducing human-defined concept layer that bridges raw visual features and human-undenn cumannunderstandablle semantics, providing a foundational explanation for the model's final decision (Koh et al., 2020). A key challenge lies in achieving high-quality cross-modal alignment, i.e., constructing a precise, fine-grained mapping between visual features and textual concepts. However, many popular methods (Yuksekgonul al., 2023; Oikarinen et al., 2023) rely on a single, holistic visual feature, such as the [CLS] token from CLIP-style models or global image embedding from visual foundation models (Kim et al., 2023).
These approaches operate on the premise that global feature vector encapsulates all necessary visual attributes (Zhang et al., 2014; Raghu al., 2022). This assumption is untenable in com- plex visual scenes, especially for medical images characterized by subtle, intricate, and fragmented lesions (Chen et al., 2021).
To establish more fine-grained, many-to-many mapping, recent works have begun to leverage patch-level embeddings from Vision Transformers (ViT') (Dosovitskiy et al., 2021), which offer a powerful prior for perception and alignment. Several works have ventured in this direction, em- ploying techniques such as Optimal Transport (Xie et al., 2025), or Pooss-Attetiion and dynamic pooling to learn visual concepts (Wang et al., 2025; Gao et al., 2024), yet critical issue arises: the visual features extracted from the encoder suffer from representational collapse, a phenomenon where the feature vectors degenerate into low-dimensional subspace, becoming informationally redundant and lacking diversity.
Representation Regularization. Low-rank issue has been extensively studied in the self-supervised learning (SSL) literature. For instance, Barlow Twins (Zbontar et af., 2021) mitigates collapse by minimizing the redundancy between feature dimensions via cross-correlation matrix. Other tech- niques, such as spectral regularization (Yoshida & Miyato, 2017), constrain the spectral norm of weight matrices to improve generalization. A more recent work DINOv3 (Siméoni al., 2025) in- troduces gram regularization loss to prevent the model's output from collapsing to trivial solution.
Nevertheless, critical drawback exists in directly applying these techniques to CBMs: they are not tailored for the specific demands of the CBM task, nor are they designed for cross-modal, many- to-many alignment. Specifically, the core objective of these regularization methods is to indiscrimi- nately maximize feature diversity or reduce redundancy (Huang et al., 2017; Gao & Pu, 2025). For CBMs, however, the goal not arbitrary diversity, but rather meaningful, structured diversity that aligns with human-defined concepts(Gao et al., 2026a). Indiscriminate decorrelation spectral regularization cannot guarantee that the learned feature diversity has any correspondence with the predefined semantic concepts. Consequently, these methods may amplify trivial visual details that are useless or even detrimental to the final task, thereby interfering with the formation of high-qualit, concept vectors (as demonstrate in Section 4.2).
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
3.1 PRELIMINARIES AND OVERALL FRAMEWORK
Problem Formulation. Consider dataset of triplets D - {( Ci, w)) where x; E X' is an input image, Yi E y' denotes the corresponding class label, and 6; is a set of textual descriptions defining the concepts associated with class y/- In contrast to standard black-box models that directly learn mapping 24 + Yi, the CBM pipeline is formulated as a two-stage process: 24 - C + 9i- First, the alignment stage produces CAV, which denote as activation scores for K concepts for a given image 24. It is generated by aligning the learned Visual Concepts, M E KxD with their corresponding textual concept embeddings,
RK This vector contains the
RKXD
Specifically, the activation score for the k-th concept is computed as the dot product between its visual and textual representations:
(v:)& = M Tk: 4k E K].
The CAVs then serve as the exclusive input for the second-stage classification task. Given the straightforward mechanism of the second stage, achieving precise and efficient image-concept align- ment in the first stage is of paramount importance.
Visual Concept Learning. To acquire fine-grained visual features, we leverage a pre-trained CLIP VIT encoder obtain a structured visual representation from the input image x₁:
Z 7cls 21, Z2. zr.]
Ey(x;) E (R++))x
(2)
..... where Zels is the class token embedding and Zp [2 ......., Zp RLXD represents the matrix of
patch token embeddings. This patch-level representation is crucial, as single textual concept often corresponds information spanning multiple patches, while single patch may contain details relevant to multiple concepts.
This complex, many-to-many mapping necessitates a sophisticated mechanism to bridge the gap between local features and high-level concepts. Therefore, our objective is to aggregate the patch features Z, into K meaningful Visual Concepts, denoted by the matrix M E IR K These learned Visual Concepts are subsequently used for the final concept-text alignment in our problem formulation. The following sections detail the two core components of our method designed to achieve this: IVQ for representation regularization and the Magnet Attention mechanism for concept aggregation.
described
3.2 IMPLICIT VECTOR QUANTIZATION
To mitigate the aforementioned issues, we propose novel IVQ mechanism. contrast to conven- tional VQ approaches, which first identify the nearest codebook vector over a distance metric and subsequently propagate the resulting quantized features through the forward pass, our IVQ method strategically discards the quantized features. Instead, it exclusively leverages the commitment and codebook losses as regularization terms during backpropagation. This process compels each visual patch embedding to align more closely with its nearest codebook pector-whiee semantically cor- responds to a textual concept, as will verified in our Section 4.2. This, in turn, distills the core conceptual information while simultancously regularizing the representation space and enhancing the diversity visual features, ultimately benefiting the subsequent aggregation of visual concepts.
Given the encoded visual patch features Zp {2 critical challenge is to prevent representa- tion collapse and enhance feature diversity. We maintain a small, learnable codebook Cuy E ADD, where M equals the number of textual concepts K (discussed in Section 4.2). For each patch feature z, E Zp. we find the nearest codebook vector e E Cog via an a argmin operation over the Euclidean distance:
ky argmin||z, Cı 11.
The quantized representation Z Cky- However, discard this quantized output Z, and do not use it in the subsequent forward pass. Instead, we compute the VQ loss, which consists of a codebook loss and commitment loss, to update the feature encoder and Cug
4

---

Published as a conference paper at ICLR 2026
M
Color
(ig Align
Text Encoder Ey
Texture
Shape
Diagnosis Concept
Textual Emhedding
Textual Concepts J
CLS
Visual Concepts M
Bgye)
Visual
Tey(3)
Encoder E
AStEaCE
Magnet
Quantization
N
Image
Patch Embedding
Codebook Cog
Figure 2: A pipeline of the proposed IVQ-CBM method which optimizes training from three dimensions: classification accuracy, concept aligmment, well representation diversity and quality. An many-to-many mapping between visual embeddings and textual concepts establish foundation of our model. IVQ fur- ther regularizes the representation space and helps to maintain and distill the core information for cach patch throughout training. Building upon obtained high-rank concept-awar features, magnet aggregates visual con- cepts from embeddings. regarded input for concept alignment and final classification.
Live = ss (p)) 2g
+B 1 sg (2
(4)
Codebook Loss
Commiument Loss
where sg(.) denotes the stop-gradient operator and 8 is the commitment hyperparameter, By backpropagating Livo. we compel the patch features in Z. to align with a learned prototype without suffering from the information bottleneck of hard quantization. This process regularizes the repre- sentation space, encouraging more structured and diverse feature distribution, which is crucial for the subsequent aggregation stage.
3.3 MAGNET CONCEPT AGGREGATION
With regularized, high-rank feature space established by IVQ, now introduce mechanism to bridge the gap between low-level visual features and high-level semantic concepts. simple spatial pooling of all patch features would lose critical fine-grained information (Wang al., 2025). Therefore, we propose the Magnet Attention mechanism, a differentiable soft-clustering module designed to aggregate the L patch features into K semantically meaningful Visual Concepts.
To achieve this, we first introduce a set of learnable concept queries, denoted as Q E KDD, where K is the number of concepts. Each query vector dı E Q acts as learnable center-point to attract patch features related to specific concept. For the input patch features Zp {z, ) we compute similarity score between each patch feature z, and each concept query qи- Following common practice, we use negative squared Fuclidean distance as the similarity metric (Van Den Oord et al., 2017b).
This similarity is then converted into a soft assignment matrix A LRLK via softmax function over the concepts:
exp(-
1xll
Ajk
(5)
exp( z2 q
where A₁k represents the soft-assignment weight of the j-th patch to the k-th concept query. These weights form an attention map over the patches for each concept. The final Visual Concepts. de- noted by the matrix M E R x. are then computed as weighted average of the patch features:
M A Zp-
The resulting matrix M contains K rich visual concept prototypes, each summarizing the relevant spatial information from the image corresponding to a distinct semantic concept. This aggregated representation M is then used for the final alignment with the textual concept embeddings 7.
5

---

Published as a conference paper at ICLR 2026
3.4 TRAINING OBJECTIVES
We train our model end-to-end with a composite loss that simultaneously optimizes for task accu- racy, concept alignment, and representation quality.
Classification Loss. The primary objective is to ensure the model accurately performs the final classification task. Let hets denote the final classification head. It takes the vector of concept activa- tion scores E IR* as input to produce the final class logits alignment between the k-th visual concept and the k-th textual con- cept. The classification loss Leks is the standard cross-entropy between the predicted logits and the ground-truth class label y/:
(v:) As defined previously,
each score (v:): represents
Lela (Ps.
Concept Alignment Loss. To ensure the model's interpretability, we explicitly supervise the con- cept bottleneck. The concept activation scores V; E IR serve the logits for concept prediction. We supervise these logits against the ground-truth concept labels e; E {0, 1] which is a multi-hot vector indicating the presence of each of the K concepts for the sample 24. The concept alignment loss Leoncept is the binary cross-entropy (BCE) loss between the predicted concept scores and the ground-truth labels:
Leuncept (v:u((v;.. Ci)-
(8)
This loss is crucial as forces the model to learn a set of visually grounded and semantically mean- ingful concepts.
IVQ Regularization Loss. As detailed previously, the Livo encourages the patch-level features form more structured and diverse representation space without introducing hard information bottleneck. This improves the robustness of the feature extractor and aids the Magnet Attention mechanism in forming higher-quality Visual Concepts.
Overall Training Objective. The final training objective combines these three components into single multi-task loss function. The total loss formulated as weighted sum:
L. = feets + Lecnceet + Livo-
4 EXPERIMENTS AND DISCUSSIONS
Core Questions. We structure our analysis around a series of research questions (RQs) to thoroughly investigate the properties and efficacy of our proposed method. Our goal is to dissect its underlying mechanisms, justify its design choices, and demonstrate its advantages in terms of performance and interpretability.
RQ1: Impact on Representation Collapse. How does implicit quantization affect the rank of the learned representations? Figure. and 6.
RQ2:
Efficacy of Implicit Quantization. Is implicit quantization a genuinely effective technique? How does it conceptually and empirically differ from explicit quantization? Section. 4.2
RQ3: Implicit Quantization as a Regularizer. Our method can be interpreted as a form of repre- sentation space regularization. How does its performance compare against other established regularization techniques? Why does our proposed implicit quantization, which leverages a codebook. outperform traditional regularization techniques on CBM? Section. 4.2
RQ4: Optimal Codebook Configuration. The core component of our method is the codebook, which dynamically updated throughout training. What is the optimal relationship between the code- book size, M. and the number of textual concept vectors, K"? Section. 4.2
RQ5: Interpretability as Visual Dictionary. Beyond performance, how does the codebook con- tribute to model interpretability? Section. 4.2
Baselines, Benchmarks, and Metrics. To comprehensively evaluate the robustness and general- ization of our method, we conduct experiments on a diverse suite of public benchmarks spanning two distinct domains. First, for medical imaging, utilize datasets covering dermoscopy (Codella

---

Published as a conference paper at ICLR 2026
et al., 2019), histopathology (Kather et al., 2018), fundus photography (Porwal et al., 2018; yiwe- ichen04, 2021), ultrasound (Al-Dhabyani et al., 2020), chest X-ray (Zawacki et al., 2019; Johnson et al., 2019), and mammography (Cui et al., 2021). Second, to demonstrate the broad generaliz- ability of IVQ-CBM to natural images, we extend our evaluation to five standard computer vision benchmarks: CIFAR-10 10 and CIFAR-100 (Krizhevsky, 2009) for general object classification, CUB- 200-2011 (Wah et al., 201 1) for fine-grained identification, Places365 (Zhou et al., 2017) for scene recognition, and ImageNet-IK IK (Russakovsky al., 2015) for large-scale classification.
To demonstrate its effectiveness, we benchmark our approach against eight recent and popular CBM methods: LaBo (Yang et al., 2023), PCBM (Yuksekgonul et al., 2023), COOP-CBM (Sheth & Ka- hou, 2023), LF-CBM (Oikarinen et al., 2023), Explicd (Gao et al., 2024), MVP-CBM (Wang et al.,
2025), CLEAR (Dikter et al., 2024), and DOT-CBM (Xie et al., 2025). Besides, following (Gao et al., 2024; Wang et al., 2025), we also compare black-box models (He et al., 2015; Dosovitskiy et al,, 2021) and multi-modal backbones (Radford et al., 2021; Wang al., 2022; Zhang et al., 2025).
To account for the class imbalance prevalent in these benchmarks, we employ both Accuracy (ACC) and Balanced Macro Average Accuracy (BMAC) for a comprehensive evaluation. For implementa- tion details, please refer to Appendix D.
Table I: Overall Performance comparison (ACC %) on multiple medical and general benchmark datasets. evaluate our model against various zero-shot, black-box, and explainable models. Best results are highlighted in bold. Gains (A) are compared within the Explainable methods group.
Medical Datasets
General Benchmarks
Model
ISIC NCT
IDRID BUSI Retina SIM
Cardio
CMMD C-10
C-100
CUB Ple365
ImgNet
Zero-shot
CLIP (ICML 2021]
29.88
26.67
29.84
43.85
18.33
41.01
54.81
27.10
MedCLIP (EMNLP 2022)
13.68
9.57
16.86
44
17.50
42.34
45.94
27.10
RiomedCT.IP (NRIM Al 2025)
40.51
29.36
53.33
39.88
$2,90
Black-box
ResNet50 (CVPR 2016)
83.53
92.12
54.38
76.41
78.33
87.97
81.51
74.59
VTT Base [ICLR 2023]
90.01
93.25
58.07
80.25
83,33
83.56
79.60
74.77
Explainable
LaBo [CVPR 2023]
79.20
91.73
50.77
84.01
72.60
74.13
73.48
70.78
80.23
60.17
69.88
39.67
68.04
PCBM [ICLR 2023]
85.91
91.77
54.36
84.64
66.67
80.02
76.43
69.24
84.61
63.22
72.36
41.13
70.14
COOP-CEM (Nips 2023]
93.43
89.61
85.00
80.57
77.28
85.17
73.06
42.19
71.23
LF-CBM [ICLR 2023]
83.55
87.92
60.59
76.77
73.33
77.62
75.95
74.75
82.94
61.89
71.42
40.33
69.46
Explied [MICCAI 2024]
88.72
95.29
63.26
87.17
83.33
85,70
78.95
76.74
86,03
64.91
74.08
43.07
71.93
MVP-CRM [UCAI 2025]
87.72
97.90
65.38
89.74
85.00
84.89
80.29
75.84
86.72
65.48
74.63
43.81
72.29
CLEAR (WACY 2025]
86.25
89.05
56.54
83.85
72.89
78.26
78.22
73.35
83.77
62.83
71.84
40.92
68.81
DOT-CBM (CVPR 2025)
86.55
90.15
58.45
85.23
74.59
79.52
77.15
71.15
84.38
63.45
72.29
41.76
69.31
IVQ-CBM (Ours)
99.90
67.35
93.59
88.33
88.03
82.01
79.25
88.14
67.12
45.54
+2.00
+1.89
+1.97
+1.42
+1.88
+1.73
4.1 BASIC EXPERIMENTS
Benchmark Comparison. As demonstrated in our experiments in Table 1 and Table 2, our method consistently outperforms all baselines across the full range of medical datasets, highlighting its ro- bust performance under diverse diagnostic conditions. Notably, our approach excels over traditional black-box models, maintaining high diagnostic accuracy while providing interpretability. This is a significant contribution to resolving the long-standing trade-off between performance and inter- pretability in prior works (Zarlenga et al., 2022).
To further validate the scalability and generalization of our framework, we extended our evaluation to widely used general-domain benchmarks. As shown in Table and Table 2, IVQ-CBM demon- strates superior performance across all five datasets, spanning from fine-grained classification tasks such as CUB to large-scale challenges like ImageNet. Noteworthy improvements are observed over the strongest baselines, including +1.64% on CIFAR-100 and +1.13% on ImageNet, confirming that our method's s benefits scale effectively to complex, high-dimensional visual distributions.
Additionally, to probe the underlying mechanisms behind these performance improvements, alyzed the feature rank dynamics during training. Figure 4 and Figure 5 present the rank evolution across medical and general benchmarks, respectively. The results reveal fundamental distinc-
7

---

Published as a conference paper at ICLR 2026
Table Overall Performance comparison (BMAC %) multiple medical and general benchmark datasets. We evaluate our model against various zero-shot, black- box, and explainable models. Best results are highlighted in bold. Gains (A) are compared within the Explainable methods group.
Medical Datasets
General Benchmarks
Model
ISIC NCT
IDRID BUSI
Retina
SIIM
Cardio
CMMD
C-10 C-100
CUB Plc365
ImgNet
Zero-shot
CLIP LCML 2021]
21.32
26.71
25.92
37.85
25.00
22.15
49.12
34,00
MedCLIP (EMNLP 2022]
12.05
43.19
25.42
46.66
30.72
RiomedCIJP (NEIM AT 2025]
24.47
40.39
30.31
36.45
38.33
20.92
46.70
50.00
Black-box
ResNet50 [CVPR 2016]
76.53
91.34
55.88
75.84
77.50
76.21
81.51
60.46
VIT Base [CLR 2021]
84.14
92.73
54.70
83.63
76.66
74.15
79.61
60.35
Explainable
(CVPR 2023]
91.62
73.83
72.16
73.73
65.59
79.15
59.95
39.41
PCBM 2023]
81.76
91.18
58.85
88.91
75.68
76.41
66.68
72.19
40.99
COOF-CBM [Nips 2023]
79.25
93.78
50:41
89.73
80.83
79.81
80.58
64.25
84.99
64.03
72.87
42.01
71.09
LF-CBM [ICLR 2023]
78.46
87.77
56.97
75.77
70.83
71.59
75.95
60.18
82.78
61.73
71.22
40.15
69.21
Explied (MICCAI 2024]
82.42
94.73
63.61
87.37
81.67
78.90
78.94
60.48
85.88
64.78
73.91
42.89
71.77
MVP-CRM (UCAI 2025]
80.35
97.89
57.78
91.45
83.33
78.95
80.29
56.15
86.54
30
74.45
43.66
72.15
CLEAR (WACY 2025]
81.11
86.86
58.33
83.33
70.56
70.12
78.18
61.23
83.59
62.61
71.69
40.75
68.66
DOT-CIM (CVPR 2025)
91.51
84.
71.88
76.48
77.10
62,98
84.19
72.11
41.53
69.17
IVQ-CBM (Ours)
86.22
99.88
73.06
95.38
85.83
81,91
82.01
69.70
87.91
66.88
75.68
45,32
73.23
+ +3.50
+1.99
+9.45
+3.93
+2.50
+2.10
+1.43
+3.02
+1.37
+1.58
+1.23
+1.66
+1.08
tion between our method and prior approaches. While competing methods, particularly DOT-CBM (Xie et al., 2025) and MVP-CBM, exhibit varying levels of representational collapse, our method consistently maintains high and stable feature rank across all datasets. Notably, the collapse is exacerbated on more complex datasets, such as ImageNet, as shown in Figure 5, where the feature ranks of baseline models dramatically decline. This provides compelling evidence that IVQ acts as a strong regularizer, effectively preventing the deterioration of the feature space. By preserving high-rank and diverse representations, this structural advantage directly correlates with the superior and more stable downstream performance observed in Table 2.
Ablation Study. Our ablation study (Table 3) confirms that both the IVQ and Magnet modules are integral to performance. IVQ provides the most significant boost, especially on the BMAC metric for imbalanced datasets like IDRID (+11.81). Removing the Magnet module and reverting to a standard [CLS] token baseline leads to notable performance drop. This indicates that relying on a single global feature vector an oversimplification of the image's content. While this approach may suffice for coarse geassegrained classification, it is inadequate for complex scenes requiring fine- grained analysis, such as localizing small or scattered targets-a common challenge in medical imaging (Chen et al., 2021). Furthermore, representation collapse(Figure 6). Models without IVQ suffer from rank collapse (dashed lines), whereas our method maintains high, stable feature rank (solid lines). This shows IVQ acts powerful regularizer, ensuring diverse and robust feature learning.
find that IVQ's mechanism is linked preventing
4.2 EXTENSIVE EXPERIMENTS
Vector Quantization versus Implicit Vector Quantization. As previously established, inher- ent many-to-many correspondence exists between visual patches and textual concepts, with each patch often encoding multi-faceted semantic information. Standard VQ conflicts with this principle (Van Den Oord et al., 2017b). Its use of a hard argmin operation maps each patch to a single, nearest codebook vector, which forces the collapse of a patch's 's rich information into a discrete rep- resentation. Consequently, only these quantized features are passed forward to the magnet module, discarding other relevant visual attributes and violating the many-to- the critical information bottleneck issue. In contrast, our IVQ is designed to resolve this issue. It retains the quantization objective solely as a regularizer, encouraging each raw patch feature to align with its nearest prototype. Crucially, this allows the original, high-fidelity feature vector to be used in the forward pass for the magnet module. This process distills the core conceptual information from the patch, enabling more effective concept alignment. Results summarized in Table 4, revealing that IVQ substantially outperforms the explicit quantization across all six datasets, on both Classification and Concept BMAC metrics.
relationship. leading to
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation experiments showing both ACC and BMAC metrics for each experimental setup. The final rows display performance gains (+A in red) our full model over second- booodd-eett configuration.
Components
Datasets
Metric
IVQ
Magnet
ISIC
NCT
IDRID
BUSI
Retina
SIM
Cardio
CMMD
ACC
80.88
95.23
57.14
88.45
75.00
85.72
80.21
74.59
BMAC
94,92
45.27
69.16
80.21
52.55
ACC
89.42
96.23
65.38
92.31
83.33
86.47
81.11
77.63
BMAC
82.85
95.61
61.25
91.15
80.00
80.49
81.11
66.81
ACC
90.11
99.90
67.35
93.59
88.33
88.03
82.01
79.25
(+. 0.69)
(+.) 3.67)
(+.) 1.97)
(+A 1.28)
(+.b 5.00)
(+A 1.56)
(+AA 0.90)
(+. 1.62)
86.22
99.88
73.06
95.38
85,83
81,91
82.01
69.70
BMAC
1.38)
(+1 4.27)
11.81)
(+A 4.23)
1.42)
(+A 0.90)
2.89)
Table 4: Ablation study of different representation quantization methods. We compare the performance various medical datasets in terms of Classification BMAC and Concept BMAC. The improvement of our Implicit Quantization (IVQ) over the Explicit baseline is highlighted in parentheses (+A). Our method sistently outperforms baseline, particularly in preventing representation collapse.
Explicit Quantization
Implicit Quantization (Ours)
Dataset
Cls. BMAC
Con. BMAC
Cls. BMAC
Con. BMAC
ISIC
51.47
22.22
86.22 (+34.75)
83.88 (+61.66)
Cardio
60.61
53.30
82.01 (+21.40)
80.25 (+26.95)
BUST
57.28
37.48
95.38 10)
71.02 (433.54)
SIIM
50.00
50.00
81.91 (+31.91)
80.15 (+30.15)
CMMD
50.35
50.00
69.70 (+19.35)
66.66 (+16.66)
IDRID
23.75
50.85
73.06 (+49.31)
59.48 (+8.63)
Table 5: Performance comparison of model and other representation regularization techniques across datasets. Gains (+A) losses (-A) are shown relative to best-performing alternative technique.
SIIM
ISIC
BUSI
Technique
ACC
BMAC
ACC
BMAC
ACC
BMAC
Barlow Twins (Zhontar et al., 2021)
86.39
79.17
87.62
81.16
91.02
88.81
Spectral Regularization (Yoshida & Miyato, 2017)
85.64
80.84
90.21
84.74
92.30
91.98
Gram Loss (Siméoni et al., 2025)
84.56
74.07
76.64
59.83
84.61
82.68
88.03
81.91
90.11
86.22
93.59
95.38
IVQ (Ours)
(+1.64)
(+1.07)
(-0.10)
(+1.48)
(+1.29)
(+3,40)
Representation Regularization Techniques. Several representation regularization techniques have been proposed to mitigate feature rank collapse, particularly in SSL. We benchmark our pro- posed IVQ against three prominent methods: explicit de-correlation via Barlow Twins (Zbontar et al., 2021), Spectral regularization (Yoshida & Miyato, 2017), and Gram Loss (Siméoni et al., 2025). As shown in Table 5, IVQ consistently outperforms these general-purpor regularization techniques across most metrics, yielding significant gains over the strongest baseline.
A critical question raises: is higher feature rank directly correlated with superior CBM perfor- mance? An analysis of feature rank dynamics, presented in Figure 7, reveals more nuanced re- lationship. While methods like Barlow Twins and Spectral regularization often maintain higher feature rank than IVQ, particularly on the ISIC and BUSI datasets, this elevated rank does not trans- late to better downstream performance, suggesting a form of over-regularization. We attribute this to key distinction: unlike general-purposs purpose methods that indiscriminately maximize feature diver- sity, IVQ tailored to foster the meaningful, structured diversity required by CBMs by aligning features with set of learnable prototypes.
Analysis of Codebook Size The IVQ codebook dynamically aligns patch features with a set of learnalle prototypes, which ensures feature diversity while distilling core visual information and

---

Published as a conference paper at ICLR 2026
age
SIM
ISC
BUSI
Cantle
(a) Impact on BMAC
(b) Impact on Accuracy
1.00
0.95
95
Accuracy (%)
90
BMAC
0.85
85
0.80
80
0.70
20x
10x
5x
sizem1
75
20x
10h
5x
1x
sizem1
Codebook Size Multiplier
Codebook Size Multiplier
Figure 3: Performance (ACC BMAC) across various codebook sizes. The size is defined multiple (x) the number textual concepts,
avoiding the bottleneck of conventional VQ. This raises key question regarding the optimal size of the codebook, M, in relation to the number of textual concepts, K. To investigate this, we conduct an ablation study, setting the codebook size to various multiples of the concept count (i.e., M = aK for a E {20, 10, 5, 1}) as well as a baseline with a single shared prototype (M = =).
As illustrated in Figure 3, the model's performance is robust across range of oversized codebooks (20x, 10x, and 5x), but we observe distinct peak both average ACC and BMAC when the codebook size is set equal to the number of textual concepts (M K). This setting achieves the highest average BMAC and Accuracy across all datasets. We hypothesize that setting M = K encourages one-to-one mapping between the learned visual prototypes and the predefined textual concepts, fostering a more structured and semantically aligned representation space. It provides strong evidence for the critical role of our IVQ bridge between visual patches and textual concepts, acting as a system of visual semantic anchors. These results suggest that aligning the codebook's capacity with the task's conceptual granularity is the optimal strategy.
Visualization of CodeBook in IVQ The IVQ continuously distills visual information for each concept during training. To qualitatively evaluate the knowledge captured within the final codebook, visualize the index maps of our learned prototypes on a representative mammogram from the CMMD dataset, as shown in Figure 8a.
The resulting mapping demonstrates highly logical process that mirrors clinical reasoning. The prototype for Mass Marqin (light green) precisely delineates the lesion's 's contour, while those for Mass Shape (cyan) and Calcificoi Calcification Features (blue) correspond its internal char- acteristics. Crucially, the prototype for Associated Features (dark blue) extends its focus beyond the lesion's border, probing the surrounding parenchyma for signs of structural distortion key indicator of malignancy. This structured, multi-faceted assessment validates that our codebook has learned semantically meaningful and clinically relevant concept representations (For more de- tailed visual analysis, please refer to the appendix F).
CONCLUSION
In this work, we propose IVQ-CBM, which explicitly models the many-to-many relationship while addressing representation collapse. Our method features two key components. IVQ, which uses learnalle codebook prior to anchor visual patches, and Magnet Attention, which aggregates these patches into semantically coherent visual concepts aligned with textual definitions. Extensive ex- periments demonstrate that IVQ-CBM achieves superior performance over baselines without sacri- ficing interpretability, with ablation studies validating each component's contribution. Our analysis reveals that IVQ cireumvents the hard information bottleneck of direct VQ and, unlike general regu- larization methods, fosters a meaningful feature diversity that is more effective than indiscriminately maximizing mathematical objective. Visualizations confirm that our approach yields high-quality, interpretable visual concept representations that are consistent with their textual concepts, resulting in more faithful and robust form of interpretability for CBMs.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we release the source code in the supplementary materials. All datasets used in experiments either publicly available; the implementation details, including model chitectures,
and optimization settings, are described section 4 and Appendix D. For baselines, we rely on publicly released implementations and adapt them with the same prepro- cessing pipeline as described in section 4.1. We hope these materials enable the community to faithfully reproduce our results and extend our approach.
REFERENCES
Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. Dataset of breast ultra- sound images. Data in brief. 28: 104863, 2020.
Ivan Butakov, Alexander Tolmachev, Sofia Malanchuk, Anna Neopryatnaya, Alexey Frolov, and Kirill Andreev. Information bottleneck analysis of deep neural networks via lossy compression, 2024. URL https:://// arxiv. org abs 22005.35..00 08013.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Lu, Alan I. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.043000 2021.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive eegguage-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 191922829, 2023.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03363 2019.
Chunyan Cui, Li Li, Hongmin Cai, Zhihao Fan, Ling Zhang, Tingting Dan, Jiao Li, and Jinghua Wang. The chinese mammography database (cmmd): An online mammography database with biopsy confirmed types for machine diagnosis of breast The Cancer Imaging Archive, 1, 2021.
Maor Dikter, Tsachi Blau, and Chaim Baskin. Conceptual learning via embedding approximations for reinforcing interpretability and transparency, 2024. URL htttss://axxii...rr abs/ 2406. 08840.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x 16 words: Transformers for image recognition at /asg/bb///// abs/ /2/100100 20100.11111111111111000111111110 11929.
scale, 2021. URL https:://// //aariivvvrrr www.vrrgaaae
Christopher Fifty, Ronald Guenther Junkins, Dennis Duan, Aniketh Iyengar, Jerry Weihong Liu, Ehsan Amid, Sebastian Thrun, and Christopher Re. Restructuring vector quantization with the rotation trick. In The Thirteenth International Conference on Learning Representations, 2025.
Shujian Gao, Yuan Wang, Jiangtao Yan, Zuxuan Wu, and Yu-Gang Jiang. Thinking with deltas: In- centivizing reinforcement learning via differential visual reasoning policy, 2026a. URL https: //arxiv. 22500//.../////....a/oo 06801.
Shujian Gao, Yuan Wang, and Zekuan Yu. Barl: Bilateral alignment in representation and label
spaces for semi-supervised volumetric medical image segmentation, 2026b. URL https:// arxiv. org/ abs 2210. 221.1.11.............11111111111 16863.
Xin Gao and Jian Pu. Deep incomplete multi-view learning via cyclic permutation of vaes, 2025. URL https:///////m. / //axxvvvv.../////////aa///aaatvv org/ abs/2502 11037.
Yunhe Gao, Difei Gu, Mu Zhou, and Dimitris Metaxas. Aligning human knowledge with visual con- cepts towards explainable medical image classification, 2024. URL https: : /ttwxiiivvv.o///////... abs/2406. 05596.
11

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition, 2015. URL https: arxiv. org/ abs //1//11..110 03385.
Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks, 2017. URL https: : / wttaaaiivvv.ro../eaaaaa aab////////// / 11709.06799..
Mingkai Jia, Wei Yin, Xiaotao Hu, Jiaxin Guo, Xiaoyang Guo, Qian Zhang, Xiao-Xiao Long, and Ping Tan. Mgvq: Could vae beat vae? generalizable tokenizer with multi multi-group group quantization. arXiv preprint arXiv:2507.077 0779777799700 2025.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning, 2022. URL httpttpps//aaxi...ooto
/2110. 09348.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew Lun- gren, Chih-ying Deng, Roger G Mark, and Steven Horn. Mimic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. (No Title), 2018.
Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bot- tleneck models, 2023. URL https: http/aaavvvv..////////.//..aa org/ abs abwaaaass000// /2006.366.00 01574.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models, 2020. URL 04612.
org wwtpw///rrgbbbb/////////
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogerio Feris, Piotr Indyk, and Dina Katabi. Addressing feature suppression in unsupervised visual representations. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1411-1420,
2023. doi: 10.1109/WACV56688. 200.0/55668822020..000 2013.00146.
Yangyang Li, Daqing Liu, Wu Liu, Allen Hc, Xinchen Liu, Yongdong Zhang, and Guoqing Jin.
Omniprism: Learning disentangled visual concept for image generation, 2024. URL https:
//arxiv.org/////o..o.l....... abs/ 2212.122244
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv:1711.5101, 2017.
arXiv preprint
Tuomas Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottle- neck models, 2023. URL httts:://axiiv org/ abaaab/// /2004. 00...6622 0219.
Konstantinos P. Panousis, Dino Ienco, and Diego Marcos. Coarse-to-fine concept bottleneck models, 2024. URL
2231 0 02116.
Namuk Park and Songkuk Kim. How do vision transformers work?, 2022. URL https:
//aaxiv.org/aale http://bb/2222..66002
Prasanna Porwal, Samiksha Pachade, Ravi Kamble, Manesh Kokare, Girish Deshmukh, Vivek Sa- hasrabuddhe, and Fabrice Meriaudeau. Indian diabetic retinopathy image dataset (idrid), 2018.
URL https::///// //dx
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
https::/azze org/ abss 21..2.110000000 00020.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks?, 2022. URL https: //axxiv.
worg/abs/// /ttp///b///888/ 2200. 08810.
12

---

Published as a conference paper at ICLR 2026
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelit images with vq-vae-2. Advances in Neural Information Processing Systems, 32, 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. Imagenet large scale visual recognition challenge, 2015. URL ptttpp://arxi..ooo abs/1409. 0575.
Emanuele Sansone, Tim Lebailly, and Tinne Tuytelarrs. Collapse-proof non-contrastive self- supervised learning, 2025. wwtptttss//tattteeeeeeeee wwwp:iii.orrraaaooooo /2tes/241000 11.40.04400 04959.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https::: /tt.:pp:///peeeeiiiiieee /ppppeeeeeeieeeeeeeeeee net cec/ffrunme wewoormmmiio wit.uummiiimMM--- A1w.ad- 4vmsMcY.
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based lo- calization. International Journal of Computer Vision, 128(2):336-35, October 2019. ISSN 1573-1405. doi: 10.1007/11123--19-00222 -11263-019-01228.
URL http: /xx doi orr.orrg////. 1000//0///00///
Ivaxi Sheth and Samira Ebrahimi Kahou, Auxiliary losses for learning generalizable concept-based models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023, URL
nttps::/ /LEEp:://cpeneeviww wttp://wwwneee///oniiieeeeee
Oriane Siméoni, Huy V. Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Marc Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang. Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi, Jamie Tolan, John Brandt, Camille Couprie. Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. Dinov3, 2025. URL abss/5008 11.8.10100 10104.
Divyansh Srivastava, Ge Yan, and Tsui-Wei Weng. Vlg-cbm: Training concept bottleneck models with dision-language guidance, 2025, URL https: /awxiv.vvvvv/////////a/////aa org/abs///////// /t/la/bs/2/400 2408 ..0000.000 01432.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017a.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017b.
R. Vershynin. high--Dimensioal Probability: An Introduction with Applications in Data Sci-
ence. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018. ISBN 9781 108415194. URL https: https://books. J- J-VjSWEACAAJ.
/books.....////////o wwwglo.oooooooooooooooooo co. co. http:leocccpp/pooekeooeooo jp/booksiid
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The Caltech- UCSD 2irds-200-2011 Dataset, July 2011.
Chunjiang Wang, Kun Zhang, Yandong Liu, Zhiyang He, Xiaodong Tao, and S Kevin Zhou. Myp- cbm: Multi-layer visual preference-enhanced concept bottleneck model for explainable medical image classification. arXiv preprint arXiv:2506. 12568, 2025.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text, 2022. URL https: / http///wwwww...g/////////e/////e h2tp:////////..... 0.1 63.
Yan Xie, Zequn Zeng, Hao Zhang, Yucheng Ding, Yi Wang, Zhengjue Wang, Bo Chen, and Hong- wei Liu. Discovering fine-grained visual-concept relations by disentangled optimal transport concept bottleneck models. In Proceedings of the Computer Vision and Pattern Recognition Con- ference, pp. 331199-30209, 2025.
13

---

Published as a conference paper at ICLR 2026
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition, 2023.
URL https:://axii..att.://// orgg abs/2308. 03685.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable im- age classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19187-19197, June 2023.
yiweichen04. retina dataset. https: https::/github 2021.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning, 2017. URL
10941.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang. Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision, 2022. URL https:
//axxiv.org/abs..../.. /1///.11.1111 11418.
Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models, 2023. URL, https://arxiv 2ttp://xxww...r//a///2222222 15480.
Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Gian- nini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, and Mateja Jamnik. Concept embedding models: Beyond the accuracy-explainabiity tradc-off, 2022. URL https:://axiii org.aiii..r/.oooeooooooooo abs//bb2/9900 22050 09056.
Anna Zawacki, Carol Wu, George Shih, Julia Elliott, Mikhail Fomitchev, Mohannad Hussain, Lahasna Phil Culliton, and Shunxing Bao. Siim-acr pneumothorax segmentation. https: / //k //kkagllleccc com/ compet t tions/ / si weliiim-rrrrpe wwwppeeemooooooooeeeeeeeeeeeeeee 2019. Kaggle.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction, 2021. URL https://///// org/ abs/aass/1000 03. 03230.
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine-grained category detection, 2014. URL https: http://wwxvv.gg///./.../ooo
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon. Biomedelip: mul- timodal biomedical foundation model pretrained from fifteen million scientific image-text pairs, 2025. URL https: : wwwaaiiivvooo.......aoaaa..aa org abs 23033 00915.
Rongzhen Zhao, Vivienne Huiling Wang, Juho Kannala, and Joni Pajarinen. Vector-quantizn vision foundation models for object-centric learning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 5422-5430. 2025.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2017.
Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. Advances in Neural Information Processing Systems, 37:12612- 12635, 2024.
Yongxin Zhu, Bocheng Li, Yifei Xin, Zhihua Xia, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22968 22968-22977 2025.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
In this section, we present additional implementation details, experiment results, theoretical analysis, pseudo code, and supplements. The content structure is outlined as follows:
Section Remaining Related Work
Section C- Theoretical Analysis
Section C.I Motivation:
The Fundamental Limitations of Single Global Feature Section C.2 Proof of Intrinsic Many-to-Many Coupling
Section C.3 Theoretical Analysis: Optimization Dynamics and Rank Preservation
Section C.4 Theoretical Analysis for Gradient Convergence
Section D
Implementation Details
Section E Feature Rank Dynamics
Section F- Visualization of Codebook IVQ
Section Visualization of Codebook in VQ
Section H Visualization Analysis With Baselines
Section I Results with Other Multi-Modal Backbone
Section J Statement on the Use of Large Language Models
Section K Pseudocode of IVQ-CBM
Section Quantitative Interpretability and Casual Faithfulness
Section M Analysis of Information Leakage and Effective Concept Learning
Section N Sensitivity Analysis of Commitment Cost Parameter
B REMAINING RELATED WORK
Vector Quantization. VQ has played central role in learning discrete latent representations. Early work on Neural Discrete Representation Learning (Van Den Oord al., 2017a) introduced codebook-based quantization that enables end-to-end training through nearest-neighbor commit- ment objective. Follow-up variants such as VQ- VAE-2 (Razavi et al., 2019) further enhanced gener- ative fidelity via multi-level discrete hierarchies. Beyond architectural extensions, series of studies examined the optimization challenges VQ. Rotation-bel based VQ (Fifty et al., 2025) restructures the quantization space to stabilize code assignments, whereas linear-layer VQ (Zhu al., 2025)
proposes lightweight transformations to mitigate representation collapse and improve codebook utilization. VQ has also been explored within broader perceptual tasks: fector-Quantized Vision Foundation Models (Zhao et al., 2025) leveraged discrete visual tokens for object-centric learning, and scaling studies, such 100K-VQGAN (Zhu et al., 2024), demonstrated that extremely large codebooks can maintain high utilization. More recent designs like MGVQ (Jia et al., 2025) adopt gulti quantization increase representational granularity and improve generalization.
Despite these advances, prior works rely on explicit quantization, where the quantized vectors re- place continuous features in the forward pass. Such a hard discrete bottleneck is often beneficial for generative modeling but is misaligned with CBMs, which require preserving rich, distributed semantic information and allowing many-to-many relations between patches and concepts.
C THEORETICAL ANALYSIS
C.1 MOTIVATION: THE FUNDAMENTAL LIMITATIONS OF A SINGLE GLOBAL FEATURE
A prevailing and influential paradigm in the evolution of Concept Bottleneck Models is the use of a single global feature from a pre-trained vision model, such as CLIP, as the sole representation of an input image. Specifically, these models typically rely the final [CLS] token from Vi- sion Transformer (ViT) or the feature vector generated by a Global Average Pooling (GAP) layer in Convolutional Neural Network (CNN) (He et al., 2015). The implicit assumption underpinning
15

---

Published as a conference paper at ICLR 2026
this approach is that a single nigh-dimensional vector (e.g.. D 768) can sufficiently and com- prehensively encapsulate all semantic information relevant to downstream tasks. In other words, posits that this single, condensed vector is a complete basis for identifying all pertinent concepts and, ultimately, the final class label.
However, argue that this seemingly efficient simplification is, in fact, illusion of sufficiency that conceals fundamental flaws. While it may suffice for simple, object-centric classification tasks, its limitations are starkly exposed when confronted with visually complex scenes that demand fine- grained understanding.
Our central thesis is that the reliance single global feature vector is an untenable design sim- plification for building robust, interpretable, and broadly applicable CBMs. This simplification is particularly fragile in scenarios where:
The image contains multiple, spatially distinct objects or concepts.
The critical differentiating information lies in subtle, local details or textures.
The key features are inherently distributed or non-centralized.
This argument is not speculative but is grounded in the unavoidable theoretical limitations inherent to the mechanisms that generate this global vector. We prove this by deconstructing its three fatal flaws.
1. Irreversible Loss of Spatial Information. The generation of a single global vector is, by its very nature, process that destroys spatial structure. For CNN-based GAP layer, the mechanism averages a feature map of size H x W x D across its spatial dimensions to produce a 1 X 1 x D vector. This operation is mathematically permutation invariann;; long as the set of local features remains the same, their spatial arrangement can be arbitrarily altered with little to no change in the resulting global vector. This is analogous calculating the word frequency of document while discarding the sentences and paragraphs. We know what content is present, but we have permanently lost how it is organized. Consequently, any task requiring answers to "where?" or an understanding of spatial relationships is impossible for model that has discarded all coordinate information.
For the Transformer-based [CLS] token, while its computation involves a spatially awware self- attention mechanism, the process is ultimately one of aggregation and summarization. The final [CLS] output is an abstract vector that has encoded spatial relationships into its dimensions, but the explicit, original topological structure is lost (Park & Kim, 2022). The attention maps exist during computation, but the final vector itself does not retain this map. Conclusion: A single global vector actively discards the two-dimensional structure of an image, which is its first fatal flaw (Yu et al., 2022).
2. The Information Bottleneck and Feature Suppression. A vector of fixed dimensionality has a finite capacity for information, creating a natural bottleneck (Butakov et al., 2024; Gao et al., 2026b). Within an image, different regions, objects, and textures must compete for representation within this fixed bandwidth. In this process of feature competition, strong signals (from large, prominent objects) will disproportionately dominate the final vector's representation, while weak signals (from small subtle objects) are easily averaged experiment: a high-resolution CT scan where 99.9% of the image consists of healthy lung tissue (a strong signal) and only 0.1% contains a small, early-stage pulmonary nodule weak signal). During the global aggregation process, the features representing healthy tissue will overwhelm the vector, making it nearly impossible for the faint but critical signal from the nodule to survive this democratic aggregation. Conclusion: The limited capacity of single global vector forces a lossy compression that sy systematically sacrifices the fine-grained or low-prevalence information critical for complex tasks.
suppressed (Li et al., 2023). Consider a thought
3. Representational Failure for Multiple Instances and Concepts. The core promise of a CBM is a clear mapping from image features to concepts. A single global vector is powerless to man- age this when mo-ay"to-many" relationships concepts coexist in an image (e.g.. "striped texture," " "pointed ears," and "furry texture")) ) a single vector is forced to entangle these disparate pieces of information (Li al., 2024). This entanglement
required. When multiple independent semantic
16

---

Published as a conference paper at ICLR 2026
leads to severe representational ambiguity. An activation in a 21 dimension might represent concept A, concept B, or an inseparable combination A and C. This fundamentally violates the goal of CBMs to be interpretable and intervenable. An ideal concept bottleneck should have distinct "chan- nels" corresponding to disentangled concepts (Xie et al., 2025). For example, an intervention on the "pointed ears" concept becomes meaningless if its representation is coupled with the "furry texture" concept, as we cannot modify one without affecting the other. Conclusion: To model multiple con- cepts independently and clearly, model requires access multiple, separable visual features prior to the concept alignment stage. A single global vector cannot meet this fundamental requirement.
In summary, drawing from the current paradigm of CBMs that rely on single global feature, we have proven through theoretical analysis that this is a fundamentally flawed simplification. Its three fatal flaws-the loss of spatial information, the creation an information bottleneck, and the failure to represent multiple concepts clearly-are inherent to its generation mechanism. This provides solid and compelling theoretical foundation for our proposed paradigm: establishing a many-to- many mapping between local image features (patches) and semantic concepts.
C.2 PROOF INTRINSIC MANY-TO-MANT COUPLING
Proposition 1. Within the proposed IVQ- framework, (i) each learned Visual Concept in- evitably depends on multiple visual patch embeddings, and (ii) each visual patch embedding evitably contributes to multiple Visual Concepts.
Proof. We prove the two parts of the proposition separately.
1. FORMAL DEFINITIONS AND NOTATION
We adopt the notation from the method section.
Visual Patch Embeddings Zp = [2i,..., 21.] E RL are the patch embeddings from the ViT encoder E
Concept Queries Q = [q...... Bli.....qK.. qK] E RxX is the matrix of learnable concept queries.
Soft Assignment Matrix A E RL*K, where its element A₁* represents the soft-assignment weight of patch concept k:
Ayk exp(-I1z, q
Σド= exp(-11p -1zz q
Visual Concepts M [m....., m = A ZT E RRKD the matrix of aggregated Visual
Concepts. The k-th visual concept is given by m = ビド= Azazj.
Concept Prediction
The activation score for concept k denoted us, the cosine similarity between the visual concept m, and its corresponding textual concept embedding Tke i.e., U/ m The final prediction is č₁ (Ux), where a is the sigmoid function for the BCE || loss.
2. PROOF: "ONE CONCEPT-MULTIPLE PATCHES" IS ALMOST CERTAIN
This proposition states that for any given Visual Concept m,, its final representation and subsequent prediction é are influenced by multiple patch embeddings.
Lemma (Sufficiency)) Let A₁ be the soft-assignment weight for patch j and concept k. If there exist two distinct patch indices # j such that their assignment weights to concept l are non zero (i.e., Ask > 0 and Agk > 0), then the concept prediction simultaneously depends both patch embeddings 2; and 2,, as demonstrated by their non-zero gradients:
Oêx dix +0 and
f 0.
Oz;
Oz.
17

---

Published as a conference paper at ICLR 2026
Proof of Lemma. We apply the chain rule to compute the gradient of the concept prediction é₃ with respect to patch embedding 2j. Since dēg a' (vx) des and a (vx) * 0 almost everywhere, we
day only need to analyze the gradient of the activation score, dz:
ozy
dvz.
Ooy
Ouk 3mL
Ōz, 0m: dzj
The visual concept m, - Ei= 2 The derivative of ma with respect to 2j involves two components, as the assignment weights Als also depend Zj:
0mp dzj
0Aik dzj
Z
Direct Term
Attention Torm
where I is the identity matrix and @ is the outer product. The Direct Term Aşs I non-zero as long as the assignment weight A5ı is non-zero. This term captures the direct contribution of patch 2, the weighted average. The Atlention Term captures the indirect influence of changing 2j on the attention weights of all other patches and is generally nen-zero.
Since Ask strictly positive (due the exponential function in softmax), the Direct Term ensures that 0m, is non-zero. The gradient 9mg also non-zero in general. Therefore, the total gradient oz, non-zero. This holds for any patch j with a non-zero assignment weight, proving the lemma.
ddg
02,
Probability Argument. The condition of the lemma, Ask > and Ayæ > O, is certain to hold.
The assignment weight A₁и is calculated as exp(-liz, - 9x1l3) divided by sum of such terms. Since the squared Euclidean distance is finite and the exponential function is strictly positive for all finite inputs, every assignment weight Asæ is strictly greater than zero for any {1..... , L} and k {1, K}. Consequently, every patch has a non-zero influence every concept. The "one concept-multiple patches" relationship not just almost certain; structural certainty of the Magnet Attention mechanism.
3. PROOF: "ONE PATCH- MULTIILE CONCEPTS" IS ALMOST CERTAIN
This proposition states that a single patch embedding z, will meaningfully contribute to the forma- tion of multiple Visual Concepts.
Gradient-based Argument. Consider the update to patch embedding 2, during backpropaga- tion. driven by the concept alignment loss Leencops Σ LBCE(ČK, ct). The gradient of the loss with respect to z, is:
eonnce Oz;
Heuncer d62 Ocr 0z,
(10)
Č4
Ck Cx
OC2 Oz,
Cx(1 (1
Erur Signal
Patch Inffuence
deg is non-zero for all concepts k. There- As established in the previous section, the influence term Oz, fore, the model has non-zero prediction error for than one concept (i.e., & - Cg j 0 for multiple k), the total gradient received by patch z, will be a sum of contributions from all those concepts. Since natural images typically contain features relevant multiple concepts, the SGD update will pull in direction that is a composite of gradients from multiple concepts, making patch z, influential for all of them.
Softmax Competition Argument. We formalize the argument that single patch is unlikely to contribute to only one concept by quantitatively analyzing the conditions required for the assignment distribution to become nearly one-hot.
18

---

Published as a conference paper at ICLR 2026
Let's start by introducing temperature parameter 7 > 0 into the softmax function, which controls the sharpness of the output distribution. The assignment weight of patch j to concept is given by:
exp( - 2, 9 13/
AJa
Σ k'=1 1 exp 1 Ok' 12/T)
For the original formulation, we can simply consider r = 1. A "one-hot" assignment, where patch contributes almost exclusively single concept, means that for some concept k", its weight Azk- approaches 1.
Let k" be the index the concept query closest to the patch embedding 2j:
k arg
min kE (E......
1|, 1x
To quantify the significance of this closest distance, we define the minimum distance margin Amin for patch Zj as the difference between the squared distance to the second-closes query and the closest one:
min (Ilz; q - ||z q 112) KEE
min
large Amin 2 0 indicates that has a clear "winnerr" concept query, while small Amin suggests high competition. Now, establish a lower bound for the maximum assignment weight Aya-
Asx-
E K exp (-
1| 11x--99/ 9ar
||z, 9:-
1
By definition, |22 - 1 sum in the denominator, yielding the inequality:
qu-1lž Amin for any k K This allows bound the
Ask- 2
1 + (K - 1)e- Amin//'
This inequality provides a precise condition for a one-hot assignment. For Ajk- to be nearly (e.g., Ayk- 2 e), the margin-to-temperatuu ratio Awin/T must be large, specifically Amin 2 r log (--)). other words, patch can only be exclusively assigned one concept if its embedding 2j is geometrically well-separated from all but one concept query.
Such a large margin geometrically improbable in ditennsinnii spaces due the concen- tration of measure phenomenon (Vershynin, 2018), which states that distances between random points tend to be tightly clustered. It is far more likely that 2j will be reasonably close to several queries, leading to small Amin and thus distributed (non-sparse) set of assignment weights. The soft nature of the attention mechanism, combined with these geometric properties, ensures that cach patch almost certainly contributes to multiple concepts.
4. CONCLUSION
The Magnet Attention mechanism, by its design, establishes dense, many-to-many coupling be- tween patch embeddings and visual concepts.
1. One Concept Multiple Patches: This occurs with structural certainty due the nature of the soft attention mechanism where all weights are non-zero.
2. One Patch to Multiple Concepts: This is highly probable outcome under standard train- ing conditions. The competitive nature the softmax function makes a one-hot assignment (where one patch contributes only concept) an unstable and non-generic solution in a high-dimensional space. Furthermore, the training dynamics, driven by multi-concept error signals, actively steer patch embeddings be useful for multiple concepts.
Therefore, the patch-concepp mapping in the proposed model is intrinsically many-to-many, provid- ing robust foundation for learning comprehensive and interpretable visual concepts. n1
19

---

Published as a conference paper at ICLR 2026
C.3 THEORETICAL ANALYSIS: OPTIMIZATION DYNAMICS AND RANK PRESERVATION
In this section, we provide a formal analysis of how the optimization dynamics of the proposed IVQ loss explicitly counteract representational collapse and preserve the rank of the feature space.
1. Optimization Dynamics as Restoring Force. Recall the commitment loss component of the IVQ objective defined in Eq.
1 Lermmit 3
Σ || sg(ex,)1 j=1
(11)
where 2, E RR. is the j-th patch feature, and Ck2 is its nearest neighbor in the codebook Cug. During backpropagation, the gradient of this loss with respect to a specific patch feature zj is given by:
Va, Poommitt 23(z; cg
(12)
This gradient can be interpreted physically as a restoring force in the high-dimensional feature space. It actively pulls every patch embedding 2j towards its assigned semantic prototype Cky- Unlike stan- dard contrastive losses that primarily push features apart, this dynamic acts as a structured gravi- tational pull, clustering the continuous distribution of patch features into compact regions centered around the learned prototypes.
2. Geometric Interpretation and Rank Lower Bound. Representational collapse manifests rapid decay in the singular values of the feature matrix Zp E R LxxD causing features to degenerate into a low-dimensional subspace (i.e., (Z,) < K). The IVQ mechanism imposes a geometric constraint that counteracts this degeneracy.
Empirically, as shown in our ablation study (M K) and UMAP visualizations (Figure 9), the codebook prototypes fci....,,kk cK) } converge to a set of well-separated, linearly independent vectors. Geometrically, these prototypes span a support subspace Scode R with an effective rank of approximately K.
By minimizing the commitment loss Livq, the optimization process acts as a force pulling the rows of Zp towards these diverse prototypes. Assuming the input image contains diverse visual elements that activate subset these distinet prototypes, the feature matrix Zp is effectively regularized to span the same subspace as the active codebook vectors:
7pp Live, span( { S Scode-
(13) Since the codebook maintains full rank (+s K), it acts set of semantic anchors that prop open the feature space. This imposes an implicit lower bound on the feature rank, ensuring that representa- tions maintain sufficient dimensionality to encode diverse semantic concepts rather than collapsing onto single manifold.
Structured vs. Indiscriminate Diversity. This analysis also clarifies why IVQ outperforms general-
ceneral-purpose purpose regularization methods like Barlow Twins or Spectral Regularization (as shown in Fig. 6).
General Regularization: Methods that penalize cross-correlation or maximize spectral entropy encourage indiseriminate diversity. They force features to be orthogonal regardless of semantic content, which can lead to over-regularization where noise or irrelevant textures are amplified to satisfy the rank objective.
IVQ Regularization: Our method fosters stractured diversity. It preserves feature rank specifically along the semantic directions defined by the clinical concepts (the codebook). The feature space is allowed to be low-rank within concept cluster (compressing intra- class variance) while maintaining high-rank separation between different concepts (pre- serving inter-class variance). This alignment between optimization dynamics and semantic structure is the key driver of IVQ-CBM's superior performance.
C.4 THEORETICAL ANALYSIS FOR GRADIENT CONVERGENCE
The total loss function Liaal is differentiable with respect to all trainable parameters 0 of the pro- posed model. This ensures that gradients are well-defined, permitting stable model training via gradient-based optimizers.
20

---

Published as a conference paper at ICLR 2026
Proof. The proof proceeds by analyzing the differentiability of each component of the total loss.
Formal Definitions. Let {0E, Φ Φy; ΦQ) denote the set of all trainable parameters, where 0g are the parameters of the ViT-based visual encoder &, QA the learnable queries in the Mag- net Attention module A, Øy are the parameters of the projection head f, and ΦQ is the learnable codebook of the Vector- cistor-Quantizer Q.
The total loss a weighted sum of its components:
Liaal (0) Lels + Leoncept + LIvQ.
(14)
By the sum rule of differentiation, each component loss differentiable with respect to 0, then Loaa is also differentiable. We analyze each component in turn.
Differentiability of the Main Prediction Path (Las and Leoncept). The main prediction path com- putes visual concepts from patch embeddings Zp Eg (x) IR LxD and then calculates the clas- sification and concept losses. The aggregated visual concepts, denoted M RxXD, are derived
as:
M Agx
(15)
and the final logits are obtained after projection head for- The Magnet Attention operator Aøa involves computing squared Euclidean distances, applying 'softmax" function, and performing weighted sum via matrix multiplication. The projection head for consists of standard neural network layers (e.g.. Linear, LayerNorm). All these operations continuously differentiable.
The losses Lass and Leonopp are computed using the cross-entropy function, which is smooth and differentiable with respect to its inputs. Therefore, by the chain rule, both losses are differentiable with respect to the parameters in their computational graphs, namely {θg, O 6.)
Differentiability of the VQ Regularization Path (Civo). This part is critical as the Vector- Quantizer Qşo contains a non oon-differentiable argmin operation for selecting the nearest codebook vector C₁2 Φq for given input patch feature zj: k- argmin ||z, e 1l
(16)
The derivative of this discrete selection is zero almost everywhere, which blocks gradient flow. The IVQ loss, however, is formulated to circumvent this issue:
Livo = sg p - Q (2p) + 7Z sg Qac [1
(17)
Codebonk
Commitment
where sgl-1 denotes the stop-gradient operator (equivalent :detach()').
We analyze the gradient of each component:
Gradient w.r.t. codebook ΦQ: The gradient for Φq flows only through the Codebook Loss term, as the Commitment Loss detaches the quantizer's output. The gradient is:
Live V ai || |
Qoo (2p)
(18)
This gradient is well-defined and updates the codebook vectors to move closer to the en- coder's features.
Gradient w.r.t. encoder 0g: The gradient for the encoder's parameters 0g (which pro- duce Zp) flows only through the Commitment Loss term. The gradient with respect to the encoder's output is:
Vz,Civo 3 Vz, Z, sg|2so (2,))
(19)
= 2,3 (Zp. -sgi-sgiuoo Sgi (Z))
This gradient is well-defined and is equivalent to that of Mean Squared Error loss. It effectively pulls the encoder's output Z, towards the selected (but detached) codebook vectors without passing through the non-differentiable argmin operation.
The use the stop-gradient operator correctly decouples the updates, ensuring that computable gradients are available for both the encoder and the codebook.
21

---

Published as a conference paper at ICLR 2026
Conclusion. We have established the differentiability of all components of the total loss function.
The prediction losses, Less and connceps are differentiable with respect to (fg, A Qy)-
The regularization loss, Livo, provides well-defined gradients for both the codebook ΦQ and the encoder Oz.
Since Liaial is linear combination of these differentiable components, its gradient V Conaall is well- defined and can computed via standard backpropagation. The existence of valid gradient is a necessary condition for the convergence of gradient-based optimization algorithms. Therefore, the model architecture is theoretically sound for training.
D IMPLEMENTATION DETAILS
Experimental Setup. To ensure a fair comparison, we reproduce all baselines within our experi- mental framework and adopt Biomedical-CLIP (Zhang et al., 2025) for our method and baselines. Based on this architecture, we insert a projection layer to map visual features to the textual feature space, thereby aligning their dimensions. We then apply L2 normalization to regularize the aligned feature representations. We employ an exponential learning rate scheduler with a warm-up period and utilize the AdamW (Loshchilov & Hutter, 2017) optimizer for training. The initial learning rate and batch size are set to le-4 4 and 32, respectively. All experiments are conducted using Python 3. 9, PyTorch 2 5... and single NVIDIA RTX 4090 GPU, We also conduct experiments within CLIP to validate the universality of representation collapse (Cherti et al., 2023).
Concept Generation. For the concepts associated with each dataset, prompt Gemini 2.5 Pro to generate textual descriptions for each class. The generated concepts are then cross-validated using GPT-4o to ensure their quality and relevance. Drawing upon the work of (Panousis et al., 2024), build hierarchical framework for concepts based on a coarse-to- principle.
E ANALYSIS OF FEATURE RANK DYNAMICS
To empirically validate our method's ability mitigate representational collapse, we analyze the dynamics of the feature rank during training. The rank of the feature matrix serves proxy for the diversity and richness of the learned representations. features are diverse and non-redundant, while decline in rank-known as rank collapse-suggeet suggests that the feature space has become degenerate, hindering the model's ability to learn distinct concepts.
sustained high rank indicates that the
In this section, we present series of experiments comparing our method against various baselines. The results consistently demonstrate the effectiveness our approach maintaining high-rank feature representations.
Comparison with State-of-the-Art Baselines. As illustrated in Figure 4, first compare model with several leading concept-based methods. Our approach (Ours) consistently maintains high and stable feature rank across all eight datasets throughout the training process. In stark contrast, baseline methods such as Explicd, MVP-CBM, and others exhibit a noticeable decline in feature rank, succumbing to varying degrees of representational collapse. This result highlights our method's ''s superior ability to preserve the expressive power of the feature space compared to existing approaches.
Comparison on Standard Benchmarks. illustrated in Figure 5, we extend our analysis to standard vision benchmarks, ranging from CIFAR-10 to large-scale datasets like ImageNet. Con- sistent with our observations in the medical domain, our approach (Ours) maintains a robust and stable feature rank throughout training across all five datasets. In stark contrast, baseline methods succumb to varying degrees representational collapse, a phenomenon that becomes notably more severe complex datasets. Specifically, MVP-CBM and DOT-CBM exhibit drastic decline in fea- ture rank on ImageNet and Places365, indicating a failure to preserve feature diversity at scale. This empirical evidence confirms that representational collapse is fundamental bottleneck in CBMs,
22

---

Published as a conference paper at ICLR 2026
and highlights method's superior ability to generalize and preserve expressive power even in large-scale classification scenarios.
Ablation Study on the IVQ Module. To isolate the contribution of our proposed IVQ module, we conduct a crucial ablation study, with the results presented in Figure 6. The comparison is stark: the model equipped with our IVQ module (w/ IVQ, solid lines) successfully sustains a high feature rank on both training and validation sets. Conversely, the model without it (w/o IVO, dashed lines) experiences a sharp drop in rank, mirroring the behavior of the baseline models. This provides compelling evidence that the IVQ module is the key component responsible for preventing feature space degeneracy and maintaining representational diversity.
Comparison with Representation Regularization Techniques. Furthermore, we extend our analysis to include other common representation regularization techniques in Figure While some regularization methods may also help in maintaining higher feature rank, it is crucial to note key insight: high feature rank is necessary, but not sufficient, condition for superior perfor- mance. Simply forcing features to be diverse (e.g., via decorrelation penalties) does not guarantee that they are meaningful or well-aligned with the downstream task. As shown in the figure, our method not only preserves rank effectively but also achieves this in a way that structures the feature space for better concept learning, ultimately leading to improved overall performance (as shown in section 4.2). This distinguishes our approach from methods that might artificially inflate rank without enhancing semantic representation.
ISIC
IDRID
Retina
NCT
100
150
130
BUSI
SIMM
Cardin
CMMD
120
140
352
Epoch
Eboch
Epoch
Epoch
Figure 4: Comparative analysis of feature rank dynamies across eight datasets. Our proposed method (Ours) successfully maintains a high and stable feature rank, while haseline methods, including Explied, MVP-CBM, DOT-CBM, and COOP-CBM, exhibit varying degrees of rank collapse during training.
Figure 5: Comparative analysis of feature rank dynamics across five general domain datasets. Our proposed method (Ours) successfully maintains a high and stable feature rank, while baseline methods, including Ex- plied, MVP-CBM, DOT-CBM. and COOP- CBM, exhibit varying degrees of rank collapse during training.
F VISUALIZATION OF CODEBOOK IN IVQ
To qualitatively evaluate the semantic knowledge captured by the IVQ codebook. we provide a comprehensive visual analysis across all six medical imaging datasets. In the following figures,
23

---

Published as a conference paper at ICLR 2026
Val (m) (VO)
Train (w/ MD)
Vol (Wil 1901
Train Fwil IVO)
140
170
103
192
302
152
BUSI
SIIMACR
Cardlo
CMMD
190
183
300
Rank
145-
270
220
150
1.00
35
100
20
60
00
100
300
Epoch
Figure Feature rank dynamies models with (w/ IVQ, solid lines) and without (w/c IVQ, dashed lines) our IVQ module, shown on both training and validation sets. The clear gap demonstrates the effectiveness of IVQ in preventing rank collapse.
NO
Spectral regudarization
Barlow Twins
Gram Loss
SIIM
ISIC
BUSI
150
Feature Bank
150
A
100-
20
20
120
20
40
800
200
122 140
20
50
Epoch
Epoch
Epoch
Figure 7: Feature rank dynamics of IVQ against regularization baselines on the SIM, ISIC, and BUSI datasets. IVQ (blue) and Barlow Twins (green) successfully maintain a high rank, while Spectral regularization (orange) unstable and Gram Loss (red) suffers collapse.
24

---

Published as a conference paper at ICLR 2026
(a) CMMD (Mammography)
(b) BUSI (Breast Ultrssund))
(c) ISIC (Dermoscopy)
(d) IDRID (Fundus Photography)
(e) Cardio (Chest X-ray)
(Chest X-ray)
Figure 8: Visualization the learned codebook prototype mappings across six medical imaging datasets. each dataset, we show (a) the input image, (b) the concept activation heatmap where each color corresponds to the most active prototype for that patch, and (c) the overlay. The color legend maps each prototype to a human -efinnd clinical concept.
visualize the prototype index maps for representative images. For each image patch, we identify the nearest codebook prototype via an argmin operation and color-code the patch according to that prototype's index. This process generates concept activation heatmap, revealing which learned visual prototype is most dominant in each region of the image. The results consistently demonstrate that model learns semantically meaningful and clinically relevant concept representations that align with the diagnostic reasoning of medical experts.
Analysis on CMMD (Mammography). As shown in Figure 8a, the resulting mapping demon- strates highly logical process that mirrors clinical reasoning. The prototype for Mass Margin precisely delineates the lesion's contour, while those for Mass Shape and Calcifical on Features correspond its internal characteristies. Crucially, the prototype for Associ ated Features extends its focus beyond the lesion's 's border, probing the surrounding parenchyma for signs of structural distortion a key indicator of malignancy.
Analysis BUSI (Breast Ultrasound). The visualization for BUSI in Figure 8b reveals struc- tured analysis of the breast lesion. The prototypes for Lesion Shape and Lesion Margin accurately capture the overall form and boundary of the hypoechoic mass. Progressing to finer de- tails, the prototypes for P costerios Features and Echo Pattern are activated on the inter- nal and posterior regions the lesion, which are critical assessments in the BI-RADS classification system.
Analysis on ISIC (Dermoscopy). In the dermoscopy example Figure 8c, the prototype mappings align well with clinical assessment criteria like the "ABCDE" rule. Prototypes for general attributes such as Shape and Color active across the broader lesion area. Mcanwhile, more specific and clinically crucial prototypes, like Dermoscopic Patterns and Texture, are correctly localized to the darker, diagnostically significant interior regions.
Most importantly, this visualization addresses the critical issue of concept ambiguity (Kim et al., 2023), where given image may lack specific concept entirely-a a key challenge for concept- based models. For instance, while this lesion has a distinct texture, it may not exhibit St reaks. Our model correctly reflects this by showing negligible activation for the corresponding prototype.
25

---

Published as a conference paper at ICLR 2026
This demonstrates faithful alignment, as the model does not erroneously foree prediction for feature that is not visually present, confirming that our learned codebook achieves a truly accurate and discerning concept assignment.
Analysis on IDRID (Fundus Photography). The analysis of the diabetic retinopathy case in Fig- 8d shows that the model differentiates between various pathologies. While general findings like Intraret inal Hemorrhages are mapped to wider areas, the prototypes for severe, high- risk pathologies such as Neovascularizan in and Vitreous Hemor renorrhage are correctly concentrated near critical anatomical structures like the optic disc and major vascular arcades.
Analysis on Cardio & SIM (Chest X-ray). The chest X-ray visualizations Figure 8e and Figure 8f) demonstrate strong anatomical grounding. Prototypes for global assessments like Cardiothcracic Ratio and Cardiac Silhouette Shape are broadly active over the heart. In contrast, prototypes for specific pathologies Pu monary Vascular Congestio on is mapped to the lung fields, while Costoph Angle Effusi on is correctly activated in the lower lobes of the lungs where pleural fluid cumulates. This showcases a spatially aware reasoning process that distinguishes between global shape and localized pathological signs.
precisely localized. For instance,
Codebook Visualization Reveals Structured and Disentangled Concepts. To qualitatively as- sess the structure the learned codebook, we visualize the code vectors using UMAP by projecting them into a 3D space. As illustrated in Figure 9, clear and consistent pattern emerges across all six distinct datasets. The learned codes for each dataset are well-separated, forming discrete and com- pact clusters with significant distance between them. This spatial separation is highly desirable, as indicates that the learned codes are disentangled and non-redundant. Each code has successfully converged to represent unique, semantically distinct concept, avoiding representational collapse where multiple codes might capture similar features. The consistency of this structured outcome across diverse medical imaging modalities-frot ultrasound (BUSI) and mammography (CMMD) demonstrates the robustness and generalizability of our method
to dermoscopy (ISIC) and beyond-ddmmonsratte in discovering meaningful basis of concepts. The formation of such a clean and well-structured codebook is foundational to the model's 's ability to make interpretable and reliable predictions.
Cando
Figure 9: UMAP visualization for the codebook various datasets. clear spatial separation between codes in each plot indicates a highly disentangled and non-redundant set of learned concepts. This consistent structure across all domains demonstrates the robustness of our method in discovering meaningful conceptual basis.
G VISUALIZATION OF CODEBOOK IN VQ
For comparison, we visualize the prototype mappings learned by a standard VQ baseline Fig- 10. This ablation study qualitatively demonstrates the limitations of standard VQ, which
26

---

Published as a conference paper at ICLR 2026
(a) CMMD (Mammography)
(b) BUSI (Breast Ultrasound)
(c) ISIC (Dermoscopy)
(d) IDRID (Fundus Photography)
Cardio (Chest X-ray)
X-ray)
Figure 10: Visualization of the learned codebook protolype mappings from standard VQ baseline. These mappings highlight a significant failure in capturing clinically-relevantl concepts. Prototypes are often scattered, misaligned with anatomical structures, and fail to differentiate between pathology and background, starkly contrasting with the precise mappings of our IVQ model (shown in Figure 8).
struggles to learn semantically meaningful and spatially-pree preccise concepts. In stark contrast to the clinically-aligned protorypes from our proposed IVQ method, the VQ baseline's mappings often re- veal significant lack of anatomical grounding. The resulting heatmaps show concepts activated in nonsensical locations, failing to distinguish between critical pathologies and irrelevant background, thus highlighting the necessity of our IVQ approach.
Analysis on CMMD (Mammography). The VQ mapping in Figure 10a fails to capture any mean- ingful anatomical structure. The prototype for Mass Margin (yellow) is nonsensically activated in the background adipose tissue rather than on the lesion's 's contour. Similarly, Mass Shape (light green) and Calc milcife ficatioo con Features (blue) scattered randomly, failing delineate the actual lesion or its internal characteristics. This mapping lacks the logical, clinically-relevant preci- sion required for diagnosis.
Analysis on BUSI (Breast Ultrasound). The BUSI visualization (Figure 10b) demonstrates near-total failure of the VQ model. The actual hypoechoic lesion is almost entirely ignored or mis- classified as the Posterior Features (dark blue) prototype, which dominates the entire image background. Critical concepts like Lesion Shape and Lesion Margin n are only sparsely and incorrectly activated in the surrounding tissue, showing no understanding of the target pathology.
Analysis on ISIC (Dermoscopy). This mapping (Figure 10c) shows poor conceptual differentia- tion. The prototype for Dermoscopic Patterns (light green) incorrectly dominates almost the entire image, including the clear background skin, indicating it has not learned a specific feature. Furthermore, Border and Shape prototypes are activated illogically inside the lesion rather than at its periphery, reversing the correct diagnostic process.
Analysis on IDRID (Fundus Photography). The VQ mapping for IDRID (Figure 10d) is chaotic and lacks anatomical precision. High-risk pathologies like Neovascul cercuuaarii ztt ion and Venous Beading scattered randomly across the retina, failing to co-localize with critical structures like the optic disc or major vascular arcades. This all-over-the-place activation suggests the model has only learned coarse pixel statistics rather than true, spatially- understanding of the pathology.
27

---

Published as a conference paper at ICLR 2026
Analysis Cardio & SIIM (Chest X-ray). The chest X-ray visualizations demon- strate severe anatomical flaws. In the Cardio example (Figure 10e), the prototype for Cardiothoracic Ratio (yellow) bleeds nonsensically into the lung fields and abdomen. Crit- ically, Costophrenic Angle Effusi on (dark blue) is activated in the *upper* lung zones, which is clinically impossible fluid accumulates the lung bases. Similarly, in the SIIM image (Figure 10f), the Costophrenic Angle (dark blue) is again misplaced in the upper chest, and the Visceral Pleural Line prototype fails the actual line of the collapsed lung.
(a) Original
(b) Ours
(c) DOT-CBM
(d) Explicd
(c) Original
(f) Ours
(g) DOT-CBM
(h) Explicd
Figure 11: Qualitative comparison of explanation heatmaps ISIC 2018 dermoscopy images (skin lesion classification). We visualize concept altributions using Grad-CAM-style activation maps (Selvaraju al., 2019) applied
concept bottleneck layer (or post-hoc concept predictor) each method, overlaid original images with jet colormap and 40% opacity. Our method consistently produces sharp. highly localized activations that precisely delineate clinically relevant lesion regions while effectively suppressing background skin and artifacts. In contrast, DOT-CBM and Explied generate diffuse, noisy, poorly localized patterns that frequently spill into surrounding healthy skin or adequately highlight diagnostic features the lesion.
H VISUALIZATION ANALYSIS WITH BASELINES
To empirically evaluate the faithfulness and localization quality of the learned concepts, we conduct qualitative comparison between proposed IVQ-CBM and two recent patch-based baselines: DOT-CBM and Explicd. We employ Grad-CAM (Selvaraju et al., 2019) to generate concept activa- tion maps, visualizing the spatial regions that contribute most significantly to the model's concept predictions. This analysis is performed across two distinct domains: medical imaging (ISIC 2018) and fine-grained object classification (CUB-200-201))
Medical Imaging (ISIC 2018). Figure 11 presents the visualization results on dermoscopy im- ages. The comparison reveals a stark contrast in the semantic coherence of the learned features. Our method (Columms (b) and (f)) produces sharp, object-centric activations that precisely delineate the lesion boundaries, effectively separating the pathological tissue from healthy skin. In contrast, the baselines exhibit severe visual degradation indicative of representational collapse. DOT-CBM (Columns (c) and (g)) displays diffuse, "cloudy" activation patterns that spill significantly into the background, suggesting failure to disentangle foreground concepts from noise. Similarly, Explicd (Columns (d) and (h)) suffers from scattered and disjointed activations that often miss the lesion center entirely. These visual artifacts confirm that without explicit regularization maintain feature rank, the resulting concept representations become informationally redundant and spatially ambigu-
General Object Classification (CUB-200-2011), To demonstrate generalizability. we extend this dataset in Figure 12. Consistent with the medical domain, our ap-
analysis to the proach generates well-localized heatmaps that cover discriminative avian parts (e.g., head, wings, torso) while suppressing complex background clutter. Conversely, the baselines struggle with lo-
28

---

Published as a conference paper at ICLR 2026
(a) Original
(h) Ours
(c)
(d) Explicd
(e) Original
(f)
(g)
(h) Explicd
Figure 12: Qualitative comparison of explanation heatmaps on the CUB-200-2011 dataset (fine-grained bird classification). We visualize concept attributions using Grad-CAM-style activation maps (Selvaraju et al., 2019) applied to the concept bottleneck layer (or post-hoc concept predictor) of each method, overlaid on the original images with jet colormap and 40% opacity. Our approach consistently generates sharp, well- localized activations tail) while effectively suppressing irrelevant background. By contrast, DOT-CBM and Explicd produce diffuse, over-smoothed, or noisy activation patterns that either leak into background areas or fail to adequately cover diagnostically relevant bird parts.
faithfully highlight the entire discriminative bird regions (head, breast, wings, and
29

---

Published as a conference paper at ICLR 2026
calization; their activations are either over-smoothed erroneously highlight background elements (e.g., branches or ground).
by precise localization and boundary adherence- not merely a result of better training dynamics but a direct phenomeno- logical manifestation of our high-rank feature space. By successfully escaping the low-rank trap via Implicit Vector Quantization, our model preserves the semantic diversity required for faithful and interpretable visual concept learning.
Conclusion.
The superior visualization quality of EITV-CBM-ccaraattee
I RESULTS WITH OTHER MULTI- BACKBONE
Val (w/ (VQ)
Train (w/ (VQ)
Val (w/o IVQ)
Train (w/o IVQ)
BUSI
Cardio
230
Feature Rank
225
210
220
200
190
180
210
10
20
40
50
20
40
60
80
100
CMMD
ISIC
250
200
Feature Rank
1.00
M
160
140-
120-
100
20
40
80
100
20
40
60
80
100
120
140
Epoch
Epoch
Figure 13: Ablation study on the impact of the IVQ component. The figure compares the feature rank dynam- ies of the model trained with and without the IVQ module with CLIP backbone, demonstrating its effect on representation diversity.
Our primary experiments are conducted on CLIP model pre-trained with biomedical cor- pus (Zhang et al., 2025) to build the proposed IVQ-CBM. This domain-speciff backbone ensures feature relevance for medical tasks. Within this main setup, we perform comprehensive ablation studies to validate the contribution each component. Figure provides a compelling visualization of crucial ablation study the IVQ module. The results uncquivocal: the model variant with- out IVQ (dashed lines) suffers a significant drop in feature rank during training, a clear indication of representational collapse. In stark contrast, the full model equipped with IVQ (solid lines) robustly maintains a high feature rank throughout the training process. Notably, on datasets such as BUSI and CMMD, even exhibits rank recovery after initial dips. This direct comparison underscores the indispensable role of IVQ in preserving feature space dimensionality and preventing catastrophic rank collapse.
A key question is whether the observed rank collapse specific to the domain-adapted biomedical encoder or if it represents more general challenge in representation learning. To investigate this, we replicated the experiment using general-purposs feature extractor. Specifically, we replaced the specialized backbone with standard pre-trained Vision Transformer (ViT-L/14) (Radford al., 2021), with weights from the OpenCLII project (Cherti et al., 2023) trained on the LAION-2B dataset (Schuhmann et al., 2022). As demonstrated inFigure 13, the results on four distinct datasets unequivocal. The models trained without our IVQ module (dashed lines) consistently suffer from a severe drop in feature rank, confirming that representational collapse not tied to a specific encoder. In stark contrast, the models equipped with IVQ (solid lines) robustly maintain high and stable feature rank throughout training. This confirms that rank collapse is a fundamental problem and establishes IVQ robust, model-agnostic solution for preserving representation diversity.
30

---

Published as a conference paper at ICLR 2026
J STATEMENT ON THE USE OF LARGE LANGUAGE MODELS
Throughout the preparation of this manuscript, we utilized large language models enhance the quality of the text. Specifically, we employed Google's Gemini Pro for tasks related to language refinement, including correcting grammar and spelling, improving sentence clarity, and ensuring consistent academic tone.
The core scientific contributions, including the formulation of the problem, the proposed methodol- ogy, the design and execution of experiments, and the interpretation of results, are entirely the work of the authors. All text generated or modified by the LLM was meticulously reviewed, edited, and revised by the authors ensure it accurately reflects our original ideas and findings. The authors bear full and final responsibility for content presented this paper.
K PSEUDOCODE OF IVQ-CBM
Algorithm Training Procedure of Our Proposed Method
Require: Training dataset D {2x c, y))
Require: Visual encoder F,, text encoder Fr, classifier head hets-
Require: Learnable concept queries Q RR
Require: Learnable IVQ codebook Cog E RD
Require: Commitment cost hyperparameter 3.
Require: Set of textual concept descriptions {tx MK JR
1: Initialize parameters for E,, hebs Q, and Cog-
2: Pre-compute text concept embeddings: Tş <- (t( for l: =1, 1,..., K.
3: for each training epoch do
4: for each batch (xr, c, v) sampled from D do
5:
>> 1. Visual Feature Extraction
6:
Zu
Zels Z + 2
zul + Ex(x)
Encode image
7:
Zp
Extract patch-level features
8:
Magnet Concept Aggregation Compute soft- soft-assignment matrix A E L between patches and queries:
9:
Ajk- + exp( 9..3)
10:
> For each patch j and query
M + A = exp(
[|x;
11:
Z
b Aggregate features into K Visual Concepts
12:
D 3. Concept-Texx Text Alignment
13:
+ diag(MT
Generate Concept Activation Vector (CAV)
14:
b 4. Final Classification
15:
p hvss(v)
Predict class logits from CAV
16:
D 5. Loss Computation
17:
Cross-entropy loss for task classification b BCE loss for concept supervision Impliciit Vector Quantization (IVQ) Regularization For each patch feature z, E Zp. find nearest codebook vector Ck₃ E Cug. Let Zg [Ck,..... Ck,. be the matrix of quantized features. Livo sg( sg() Z + 3 Z, sg(2,)
Lehs (CE(P, Looncept +- LBCE(V,
18:
20:
22:
IVQ loss A Overall Objective and Optimization
23:
C+ Cots + Loonoope + LIvO 25: Update parameters of Es, hew Q Cog using gradients from L.
Total loss
end for
27:
end for
L QUANTITATIVE INTERPRETABILITY AND CAUSAL FAITHFULNESS
While qualitative visualizations offer intuitive insights, rigorous validation requires quantifying how well the learned concepts align with human expertise and whether they causally drive the model's
31

---

Published as a conference paper at ICLR 2026
Table 6: Quantitative Human Evaluation Results. Two radiologists evaluated the interpretability of IVQ- CBM on localization quality (Likert Scale 1-5) and semantic consistency (Choice Accuracy).
Datasct
Task 1: Localization Accuracy
Task 2: Semantic Alignment
(Avg. Likert Score. 1 1-5)
(Avg. Choice Accuracy)
CMMD (Mammography)
4.32
90% (45/50)
BUSI (Ultrasound)
4.51
92% (46/50)
ISIC (Dermoscopy)
4.15
88% (44/50)
IDRID (Fundus Photography)
4.20
86% (43/50)
Average
4.30
8
decisions. To this end, we conducted two supplementary studies: a human evaluation with radiolo- gists and concept intervention test.
L.1 HUMAN EVALUATION WITH EXPERT RADIOLOGISTS
To assess the spatial and semantic alignment of our learned codebook prototypes with clinical knowl- edge, we invited two radiologists to participate in blinded evaluation study.
Experimental Protocol. We randomly sampled 50 images across four diverse datasets: CMMD (mammography). BUSI (ultrasound), ISIC (dermoscopy), and IDRID (fundus photography). The evaluation comprised two distinct tasks:
1. Concept Localization (Spatial): Experts were presented with image and target clin- ical concept (e.g., Mass Margin). They rated the model-generated activation heatmap on a 5-point Likert scale (1 Completely Misaligned, 5 Perfectly Aligned) based on whether the highlighted region corresponded to the clinical pathology.
2. Semantic Alignment (Conceptual); Experts viewed the image and the heatmap without the concept label. They were asked to identify the represented concept from multiple- choice list containing the ground truth and 3-4 plausible semantic distractors (e.g., distin- guishing Calcification from Mass Margin).
Results. detailed in Table 6, IVQ-CBM achieved high average localization score of 4.30/5.0 and a semantic alignment accuracy of 89 % These results quantitatively corroborate that our IVQ mechanism successfully anchors visual features to semantically meaningful and spatially accurate clinical concepts.
Table 7: Concept Intervention Test on CMMD. We measure the drop in the model's predicted probability for the "Mallgnant" class after manually intervening on specific concept activations. The substantial drop confirms the causal role of these concepts in the decision process.
Intervention Type
Avg. P (Malignant)
Relative Change
None (Baseline)
0.96
Intervene on "Mass Margin"
0.80
-1..7%
Intervene on "Mass Shape"
0.85
-1.55
Intervene on Both
0.72
-25.0%
L.2 CAUSAL FAITHFULNESS VIA CONCEPT INTERVENTION
A critical requirement for CBMs is causal faithfulness fainnin correlate with the prediction but actively drive it. We validated this via a counterfactual intervention test the CMMD dataset.
meaning the concepts should not merely
Methodology. We selected a test subset of 50 images where the model confidently predicted malig- nancy (P (Malligaant) 0.9). We identified clinically causal concepts for this decision, specifically Spiculated Margin and Irregular Shape. We then performed an intervention on the Concept Activa- tion Vector (CAV), denoted as Specifically, we "flipped" the activation scores of these malignant
32

---

Published as a conference paper at ICLR 2026
Table 8: Evaluation of IVQ-CBM the ANEC-S Metric. The results demonstrate minimal performance degradation when restricting model to the top-5 effective concepts, indicating robustness against informa- tion leakage.
Dataset
Domain
Full Model Acc (%)
ANEC-5 Acc (%)
Performance Drop (A)
CUB
Natural
75,93
75.21
-0.72%
CIEAR-10
Natural
87.92
86.85
-1.07%
ISIC
Medical
90.11
89.20
-0.91%
BUST
Medical
93.59
92.80
-0.79%
concepts by replacing them with the average scores derived from benign samples, while keeping the classification head hele frozen. We then measured the degradation in the model's predicted probability for the malignant class.
Results. Table 7 demonstrates the impact of these interventions. Modifying individual concepts re- sulted in notable decrease in confidence. Crucially, intervening both shape and margin concepts caused a substantial 25.0% drop in the predicted probability (from 0.96 to 0.72). This significant sensitivity confirms that the decision-makinn making logic of IVQ-CBM causally relies these high-level clinical concepts, rather than spurious correlations or background artifacts.
M ANALYSIS OF INFORMATION LEAKAGE AND EFFECTIVE CONCEPT LEARNING
Recent studies (Yan et al., 2023; Srivastava al., 2025) have highlighted critical challenge in CBM: the potential for information leakage in the bottleneck layer, where models may achieve high accuracy by encoding non-semantic noise rather than learning meaningful concepts. To rigorously validata the faithfulness of our approach, we evaluate IVQ-CBM using the A-NEC (Accuracy at Number of Effective Concepts) metric proposed by Srivastava et al. (2025) and provide a theoretical analysis of the mechanisms inherent to architecture that mitigate such leakage.
M.1 ROBUSTNESS EVALUATION ON THE A-NEC METRIC
The A-NEC metric assesses whether model relies on concise set of "effective" concepts or exploits diffuse sum of features (leakage). We conducted an evaluation on both natural (CUB, CIFAR-10) and medical (ISIC, BUSI) datasets. Specifically, we performed post-hoc pruning on the final classification layer, retaining only the top-5 contributing concepts for each class prediction (denoted as ANEC-5), and compared this restricted performance against that of the full model.
As presented in Table 8, IVQ-CBM retains over 98% of its original performance across all datasets even when restricted to utilizing only the top-5 concepts. This minimal performance drop indicates that the model's 's decision-maker is primarily driven by few highly relevant semantic concepts, rather than relying residual leakage distributed across irrelevant dimensions.
M.2 MECHANISM ANALYSIS: MITIGATING LEAKAGE VIA STRUCTURAL REGULARIZATION
While prior approaches mitigate leakage by enforcing sparsity constraints on classifier weights, IVQ-CBM addresses the root cause- cause-feature feature representation representation-ttruugg two complementary mecha-
nisms:
IVQ as a Semantic Filter. Information leakage often exploits high- high-frequency noise or low-rank degenerate subspaces that carry discriminative but non-semantic information. The proposed IVQ mechanism acts as a structure-inducing regularizer. By imposing the Commitment Loss (Eq. 4), continuous patch features are compelled to align with learned codebook prototypes, which function as semantic anchors. This process effectively filters out unstructured noise, as leakage patterns typically lack the statistical consistency to form stable clusters within the codebook. By enforcing semantic consistency with discrete prototypes, IVQ prevents the bottleneck from serving as generic conduit for pixel-level noise.
33

---

Published as a conference paper at ICLR 2026
Table 9: Performance comparison the ISIC dataset before and after concept refinement. Guided by the assignment frequency in IVQ codebook, replacing the dormant concept Rough Texture with Irregular Streaks leads to consistent gains.
Model Setting
Concept Configuration
ACC (%)
BMAC (%)
Initial
Includes Rough Texture
90.11
86.22
Retined
Includes Irregular Streaks
90.75
87.12
Performance Gain
+0.64
+0.90
Magnet Attention as Implicit Sparsity, The Magnet Attention mechanism aggregates features via a competitive Softmax dynamic. This introduces an implicit sparsity effect: patches must "vote" strongly for specific concept queries to be aggregated. Diffuse background noise or leakage, which typically manifests as weak, uniform signals across patches, is naturally suppressed by the exponen- tial nature of the Softmax function. Consequently, the resulting Visual Concepts M are composed of strong, concept-alignu signals, effectively achieving the goal of effective concept learning during the feature aggregation stage.
M.3 IVQ CODEBOOK AS A DIAGNOSTIC TooL FOR CONCEPT REFINEMENT
Beyond its role as a regularizer, the IVQ codebook offers a unique capability: it functions as a trans- parent diagnostic tool to audit the quality of predefined textual concepts. Since the commitment loss (Eq. 4) compels codebook vectors to serve as semantic anchors for visual patches, we hypothesize that the frequency of patch assignments each prototype serves as a quantitative proxy for that concept's 's visual validity.
To validate this, we conducted post-hoc analysis of prototype assignment statistics on the ISIC dataset. The analysis revealed stark dichotomy: while robust concepts such as Blue-whirish veil attracted tens of thousands of patch assignments, the concept Rough Texture received negligible attention (< 50 assignments across the entire validation set). This quantitative signal suggests that Rough Texture is tactile property ill- suited for 2D dermoscopy classification, effectively rendering it null concept in the visual domain.
Leveraging this insight, we implemented a closed-loop refinement process summarized in Table 9. We replaced the ineffective concept with more visually distinct alternative, Irregular Streaks, and retrained the model. As shown in the table, this targeted adjustment yielded tangible performance improvement, boosting Accuracy to 90.75% (+0.64%) and BMAC to 87.12% (+0.90%). This ex- periment demonstrates that IVQ-CBM provides a mechanism not only for learning concepts but for validating and refining the human knowledge hase itself.
N SENSITIVITY ANALYSIS OF COMMITMENT COST PARAMETER
The commitment cost hyperparameter B3 in the IVQ loss (Eq. 4) serves critical role in balancing the strength of the regularization, specifically controlling how tightly the visual encoder's output constrained to the learnable codebook prototypes. To empirically validate the model's sensi- tivity to this design choice, we conducted ablation experiments on three diverse datasets: ISIC (Dermoscopy), SIIM (Chest X-ray), and BUSI (Ultrasound), evaluating a standard range of values 3 E {0.1, 0.25, 0.5, 1.0].
As summarized in Table 10, the model demonstrates remarkable robustness variations in B. The performance fluctuations in both Accuracy and BMAC are negligible (typically < 0.5%) across all tested values. This stability suggests that the primary function of the IVQ loss to provide structural constraint that prevents feature collapse. Our empirical results indicate that within the examined range, the model not sensitive to the precise magnitude of 3, suggesting that the regu- larization remains effective as long as the commitment cost is set to a reasonable non-trivial value.
34

---

Published as a conference paper at ICLR 2026
Table Sensitivity analysis of the commitment cost 3 on ISIC, SITM, and BUSI datasets. The model shows robust performance across range of standard values, with the default setting (B 0.25) highlighted bold.
Commitment Cost (8)
ISIC (ACC BMAC)
SIIM (ACC / BMAC)
BUSI (ACC BMAC)
0.10
89.94 / 86.05
81.85 81.90
93.25 / 95.15
0.25 (Default)
90.11 / 86.22
82.01 82.01
93.59 / 95.38
0.50
90.02 / 86.14
81.92 / 81.95
93.41 / 95.22
1.00
89.85 / 85.92
81.70/ 81.65
93.10 / 94.95
35


Reviews:
[{"review_id": "TGEsqwIzdA", "reviewer": "Reviewer_Ycpf", "summary": "The paper identifies representational collapse as a key obstacle in CBMs. It proposes Implicit Vector Quantization (IVQ) to preserve feature diversity without hard quantization and Magnet Attention to explicitly model many-to-many patch–concept relations. Experiments on eight benchmarks show that IVQ-CBM prevents collapse, improves interpretability, and achieves state-of-the-art performance.", "questions": "Do the IVQ-CBM generalizes to broader multimodal reasoning or relational concept tasks? Are there any experiments to validate it?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. It proposes a novel view of representational collapse as low-rank degeneration and introduces IVQ, a soft regularizer that avoids hard quantization.\n2. The proposed method addresses a fundamental CBM limitation. It improves both interpretability and performance with broad applicability to multimodal learning.\n3. This paper is well-organized and clearly written. The motivation and method is easy to follow.", "weaknesses": "1. The interpretability assessment mainly relies on qualitative visualization and concept-level alignment metrics. Incorporating human evaluation or concept intervention tests would better validate the interpretability and causal faithfulness of the learned concepts.\n2. A theoretical analysis and a deeper connection between IVQ’s optimization dynamics and rank preservation would clarify its mechanism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XCFyCXZemK", "reviewer": "Reviewer_fJ93", "summary": "This paper proposes a novel method to utilize patch level visual representation to capture fine-grained relationships with concepts for training Concept Bottleneck Model (CBM).", "questions": "My primary concerns are lack of results on standard datasets and metrics and missing qualitative results. Please see weaknesses for more details.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a novel idea of utilizing patch level representation from CLIP for modeling fine-grained and detailed relationships between image regions and concepts.  \n- The authors demonstrate representation collapse in CBM training with patch-level details and propose novel implicit vector quantization to preserve feature diversity.", "weaknesses": "- The claim of evaluating proposed methods on diverse benchmarks is misleading. While the authors demonstrate results on 8 diverse medical datasets, they should also evaluate the proposed CBM on standard benchmark datasets including CIFAR, CUB, ImageNet, and Places [1,2,3,4]. Such comparisons are crucial for assessing the scalability and generalization of the proposed approach across general classification tasks with increasing dataset sizes.\n- The paper is missing qualitative comparison visualizing the concepts learned by the method and with other baselines.\n- Recent studies [3, 4] have shown that CBM training can leak information through the bottleneck layer, allowing models to achieve high accuracy without actually learning meaningful concepts. To address this issue, [4] introduced the A-NEC metric to enable fair comparisons between CBMs based on their concept learning ability and IVQ-CBM should be evaluated on this metric.\n- Notations and writing in section 3.2 such as $Z_{v}, Z_{q}$ are unclear.\n\n[1] Yang, Yue, et al. \"Language in a bottle: Language model guided concept bottlenecks for interpretable image classification.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[2] Oikarinen, Tuomas, et al. \"Label-free concept bottleneck models.\" arXiv preprint arXiv:2304.06129 (2023). \n\n[3] Yan, An, et al. \"Learning concise and descriptive attributes for visual recognition.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[4] Srivastava, Divyansh, Ge Yan, and Lily Weng. \"Vlg-cbm: Training concept bottleneck models with vision-language guidance.\" Advances in Neural Information Processing Systems 37 (2024): 79057-79094.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UToUBaItRo", "reviewer": "Reviewer_w787", "summary": "The paper addresses \\textit{representational collapse} in Concept Bottleneck Models (CBMs), where visual patch features degenerate into a low-rank subspace, consequently degrading concept quality. The proposed solution introduces Implicit Vector Quantization (VQ) to discretize visual features into a high-dimensional concept space. This acts as a regularizer to prevent the \"low-rank trap\" and is shown to improve both concept quality and overall task classification performance. Nevertheless, I am not an expert in Concept-based learning, so the following comments and questions could be based on potentially inaccurate understandings or biased perspectives, and I will raise the rating if the author could give a sound explanation.", "questions": "How sensitive is the performance (concept quality and task accuracy) to the choice of the commitment cost hyperparameter $\\beta$ in the VQ loss, and how does the size of the implicit codebook affect the trade-off between preventing collapse and maintaining expressive feature representation? Could the author give more empirical or theoretical evidence?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies a critical and pervasive challenge in CBMs (representational collapse) and provides a clear mechanism for its cause. Implicit VQ is a novel and well-justified technique for regularizing the feature space and enforcing feature diversity, directly tackling the collapse problem. \n\n2. The method claims to simultaneously enhance model interpretability (concept quality) and maintain or improve predictive accuracy.", "weaknesses": "The implicit VQ regularization might introduce computational overhead compared to standard CBM training. The concept learning process relies on pre-computed text concept embeddings, which may limit the discovery of new, fine-grained, or complex concepts not covered by the initial textual set (Algorithm 1, steps 2, 13).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWzqb0213w", "reviewer": "Reviewer_93wz", "summary": "This paper talk about Concept Bottleneck Models, which try to make AI more explainable by using human concepts. But they find a problem, the features collapse and lose information, making model not good. They propose a new method called IVQ and Magnet Attention to fix this, and they say it works better in many experiments. They also provide some interesting explanable visualzations.", "questions": "Please refer to the former section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Intuitive solution -- implicit vector quantization for binding human text concepts.\nClean model architecture design with consistent performance boosts.\nExplanable results -- the visualization of the concepts learnt aligns well with human knowledge.", "weaknesses": "W1 Typo\n---\nLine 178. \n`Building upon obtrained high-rank concept-aware features`.\nShould be obtained.\n\n\nW2 Unclear Presentation\n---\nFigure 2, top left.\nIt draws `color`, `texture` and `shape` as text encoder input, so it implies #codes should be equal to #color + #texture + #shape.\nHowever, Line 215 says that `M equals the number of textual concepts K`. Which one is true?\n\nBesides, what is the form of text inputs? Do all the baselines have access to this information? How are the correspondences between texts and visual concepts made, so as to calculate Equation (8)? The using of BCE suggests that the number of text items is equal to the number of visual concepts; but the ablation studies used different numbers of visual concepts -- How could BCE be calculated?\n\n\nW3 Unclear presentation\n---\nIn Equation (5), the normalization is conducted along dimension $K$, i.e., codes in the codebook. However, in Equation (6), dimension $L$, i.e., patch embeddings, are eliminated (weighted summed). Routinely, if you want to eliminate some dimension in a matrix multiplication, you should normalize along that dimension. Do the authors have any insights for not following the routine?\n\n```python\nZ = ...  # (L,D)\nQ = ...  # (K,D)\n# (L,1,D) (1,K,D) -> (L,K)\ndist = norm(Z[:,None,:] - Q[None,:,:], dim=2)\nA = softmax(-dist, dim=1)  # TODO XXX why not `dim=0` or `L` ???\n# (L,K)\nM = einsum(\"LK,LD->KD\", A, Z)  # TODO XXX `L` is eliminated\n```\n\n\nW4 Incomplete Analysis\n---\nFigure 7a.\nVisualizing the code indexes of explicit vector quantization is missing.\n\n\nW5 Incomplete Review\n---\nSection Related Works:\nVector-Quantization-related works should be reviewed here, given the VQ contributes a lot in the proposed method.\nPossible literatures include:\n- Neural Discrete Representation Learning.\n- Generating Diverse High-Fidelity Images with VQ-VAE-2.\n- Restructuring Vector Quantization with the Rotation Trick.\n- Addressing Representation Collapse in Vector Quantized Models with One Linear Layer.\n- Vector-Quantized Vision Foundation Models for Object-Centric Learning.\n- Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%.\n- MGVQ: Could VQ-VAE Beat VAE? A Generalizable Tokenizer with Multi-group Quantization.\n- ...\n\n---\n\nThat said, I am open to revising my ratings, if the author could address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)