LG-CAV: Train Any Concept Activation Vector with Language Guidance
Authors: Qihan Huang
Jie Song
Mengqi Xue
Haofei Zhang
Bingde Hu
Huiqiong Wang
Hao Jiang
Xingen Wang
Mingli Song
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainability for Computer Vision
Concept Activation Vector
Cross-Modal Model
Primary area: interpretability_and_explainability

Abstract: Concept activation vector (CAV) has attracted broad research interest in explainable AI, by elegantly attributing model predictions to specific concepts. However, the training of CAV often necessitates a large number of high-quality images, which are expensive to curate and thus limited to a predefined set of concepts. To address this issue, we propose Language-Guided CAV (LG-CAV) to harness the abundant concept knowledge within the certain pre-trained vision-language models (e.g., CLIP). This method allows training any CAV without labeled data, by utilizing the corresponding concept descriptions as guidance. To bridge the gap between vision-language model and the target model, we calculate the activation values of concept descriptions on a common pool of images (probe images) with vision-language model and utilize them as language guidance to train the LG-CAV. Furthermore, after training high-quality LG-CAVs related to all the predicted classes in the target model, we propose the activation sample reweighting (ASR), serving as a model correction technique, to improve the performance of the target model in return. Experiments on four datasets across nine architectures demonstrate that LG-CAV achieves significantly superior quality to previous CAV methods given any concept, and our model correction method achieves state-of-the-art performance compared to existing concept-based methods. Our code is available at https://github.com/hqhQAQ/LG-CAV.

LG-CAV: Train Any Concept Activation Vector with Language Guidance
Qihan Huang', Jie Song": Menggi Xue², Haofei Zhang', Bingde Hu', Huiqiong Wang', Hao Jiang' Xingen Wang',,, Mingli Song'
ZZejjang University, Hangzhou City University
Ningbo Innovation Center, Zhejiang University
Alibaba Group. Bangsheng Technology Co., Ltd.
lgh. huang, sjie, www.eiahangegg. tonyhu, huiqiong. wang_,aagg,aa newroot, brooksong))zzj. edu. mqxuečzuce. edu, en, aoshu. aoshu.jh@ar jaaaljjh@llbbbb inc. com
Abstract
Concept activation vector (CAV) has attracted broad research interest in explainable AI, by elegantly attributing model predictions specific concepts. However, the training of CAV often necessitates a large number of high-quality images, which are expensive to curate and thus limited to a predefined set of concepts. To address this issue, we propose Sraguage-Guuider CAV (LG-CAV) to harness the abundant concept knowledge within the certain pre-trained vision- langunlanguuge models (e.g., CLIP). This method allows training any CAV without labeled data, by utilizing the corresponding concept descriptions as guidance. To bridge the gap between vision- language model and the target model, we calculate the activation values of concept descriptions on common pool of images (probe images) with vision-language model and utilize them as language guidance to train the LG-CAV. Furthermore, after training high-quality LG-CAVs related all the predicted classes in the target model, we propose the activation sample reweighting (ASR), serving model correction technique, to improve the performance of the target model in return. Experiments on four datasets across nine architectures demonstrate that LG-CAV achieves significantly superior quality previous CAV methods given any concept, and our model correction method achieves state-of-the-arr performance compared to existing concept-based methods. Our code available https: //gi thub. com/hgghAq//GG-CAV
1 Introduction
Concept activation vector (CAV) [16] interprets the pre-trained black-box classification models (target models) by quantifying the significance of concept the model predictions. CAV provides intuitive insights to comprehend the intrinsic behavior of black-box models, elucidating the patterns behind their decision-making processes. Owing its simplicity and effectiveness, has been followed by numerous studies [10, 36, 11, 1, 40] and extended to diverse domains, such as recommender system [45], 3D shape generation [8], abusive language detection [25], etc.
However, the training of CAV usually necessitates an ample amount of high-quality images that accurately depict the corresponding concept. Unfortunately, in practical contexts, gathering adequate number of training images challenging especially when the number of concepts is extensive, thereby significantly impacting the quality (estimated using the proposed concept accuracy and concept-to-class accuracy) of the trained CAVs. Figure delineates the correlation between
Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Canope--t- Cleas Asc.
Concept Ape.
90-
14
855
80
75
8esNet18
8
8
VGG13
VGG13
70
DenseNet121
DenseNer121
40
50
80
100
10
30
40
70
100
Shot Number
Shot Number
Figure 1: The quality CAV is significantly affected by the number of training images. Here concept accuracy estimates whether the CAV faithfully represents its corresponding concept. Concept-to-class accuracy measures the similarity between the CAV and its strongly semantic-related class.
Language Descriptions
LG-CAV
Activation Values
Interpret
0.5-
Target Model CLIP Model
Ratio
VI. Model
Activate
Target Model
Probe Images
0.0 [0.0, Dj.. 0.300.2 0.3] 044 0, 0,610 6,71077008)
Activation Values
(A) Concept Knowledge Transfer
(B) Distribution of Activation Values
Figure 2: (A) LG-CAV is trained guided by activations of concept descriptions on the probe images from VL model. (B) The distribution of activation values on a concept named "Skksscraper" (from the Broden dataset [2]) in the target model (ResNet18) and VL model (CLIP) differs lot.
the number of training images for each concept and the quality of the trained CAVs on the Broden dataset [2]. It can be concluded that when the number of training images is small, the quality of the trained CAVs low, hindering CAVs from properly interpreting the model.
In recent years, the advent of foundational vision-language models (referred to as VL models, such CLIP (30J) establishes connections between images and text by mapping image features and text features into a shared feature space. These VL. models undergo pre-training on extensive image-text datasets, equipping them with the ability grasp a multitude of concepts. Inspired by this, to address the data-scarcity problem of CAV training, in this work we propose LG-CAV to utilize the abundant concept knowledge from VL models for more cheaply getting CAV for any concept given the concept descriptions, without being confined to specific pre-defined concepts.
The concept features extracted by VL model cannot be directly used for training the LG-CAV, as VL model and the target model operate within distinct feature spaces. To bridge the gap, our work ingeniously trains the LG-CAV by calculating its activation values on common pool of images (probe images), and making them mimic the activation values of the corresponding concept from VL. model these images, shown in Figure 2 (A). Therefore, L.GCAAV learns its corresponding concept according to the concept's 's existence degree (activation value) on the probe images from VL model.
However, directly applying the above framework is not guaranteed to improve the quality of LG- CAV (see experiments in subsubsection 4.1.2), because the calculated activation values from the target model and VL. model are in different distributions (see Figure 2 (B)). To tackle this problem, our work proposes a Gaussian alignment (GA) module to align the activation values from the target and VL models. Besides, reweighting (DSR) module into this framework to further improve the quality of LG-CAV. Detailedly, CE module strengthens the completeness of concept descriptions by employing data augmentations on the concept texts. DSR module optimizes the selection of prohe images by allocating higher training weights the probe images with more stable concept representation.
propose concept ensemble (CE) module and deviation sample
Furthermore, after training numerous high-quality LG-CAVs that can describe all classes in the dataset, work makes a considerable improvement on previous CAV methods by applying LG- CAVs to model correction on generic datasets like ImageNet. To this end, we fine-tune the target model to align the prediction of each class with its strongly- related concept, with proposed activation sample reweighting (ASR) module that allocates higher training weights to the samples activated more highly by the corresponding LG-CAVs.
We perform extensive experiments to validate the performance of our proposed method. Experiments demonstrate that LG-CAV achieves significantly higher CAV quality (concept accuracy & concept-to-

---

class accuracy) than previous CAV methods on the Broden & ImageNet datasets over nine backbones. Besides, we conduct model correction on the ImageNet CUB-200-201 & CIFAR-IO datasets over nine backbones. Experiments present that our method achieves significantly superior performance to other concept-based methods.
To sum up, the key contributions of this work can be listed as follows:
We propose LG-CAV tackle the data-scarcity problem of CAV training, which trained guided by the corresponding concept descriptions from VL model.
We propose a Gaussian alignment (GA) module, a concept ensemble (CE) module, and deviation sample reweighting (DSR) module further enhance the quality LG-CAV.
Beyond providing explanations, we apply LG-CAV to model correction, by proposing an activation sample reweighting (ASR) module.
Experiment results verify that LG-CAV achieves significantly higher CAV quality, and our model correction method outperforms existing concept-basee based methods remarkably.
2 Related Work
Concept Activation Vector (CAV). With the development and widespread application of deep learning [12, 31, 3, 46], it has become increasingly important explain the internal mechanisms of deep neural networks (e.g,, using concept activation vector (CAV)). Each CAV [16] is trained for a specific concept in the target model, and is used to quantify the importance of this concept to model predictions. Most existing CAV methods only utilize CAVs to interpret the target model. Concept Gradient [1] extends the original linear CAV to non-linear concept functions. which im- proves the interpretability of CAV without the linear separability assumption of CAV. OA-TCAV [40] proposes an adversarial training approach improve the quality of CAV. Differently, our method achieves significantly superior CAV quality to these methods, by transferring the abundant concept knowledge from VL model.
Vision-Languaae Models for Interpretability. CL.P-Dissect [27] and DISCOVER [28] utilize CLIPP model to describe the neurons inside the target model. Label-free CBM (26) and PCBM [47] utilize CLIP model generate additional concept annotations for concept bottleneck models [17]. These methods are limited to solely interpreting the target model and lacking the ability to improve the model performance using the explanation results.
Model Correction. Model correction methods aim improve the target model by introducing
corrective information into the model. Most existing methods [32, 22, 24, 19, 41] are limited to the
customized tasks with narrow scope (e.g., debias the color bias of model representations ColorMNIST dataset [22]). Some methods [11, 4] improve the accuracy of generic classification models, but they limited small-sized datasets. Differently, our method trains high-quality LG-CAVs that can describe all classes in the dataset, thus facilitating the task of model correction on generic datasets like ImageNet.
provide more detailed comparisons with the related methods in Appendix C.3.
3 Method
3.1
Preliminaries
The target model is pre-trained classification model that receives image as input and outputs K classification logits, with a backbone f and a final layer h. Detailedly, f extracts the image features (==) E TR D2 of (Dy is dimension size), and h is linear layer that projects f(æ) into K classification logits. Note h hf(x)) E K and hy (f))) R is the classification logit for class k.
Concept activation vector (CAV) [16] represents a concept for the target model. Specifically, given positive images (P.) and negative images (N.)) for the concept e, binary linear classifier is trained on internal features (f(ær) E Pa) and (fx)) : x (N) to discriminate e, with classification loss Lels. Finally, the CAV E R for is defined as the weight vector for in the classifier.

---

VL model [30] consists of an image encoder Jing that projects input image æ into image features Jing (x) E Rvv,, and text encoder Itext that projects input texts t into text features 9text (t) e RDv, After trained on a large-scale image-text dataset, Fimg(2) and Itext (t) are projected into the same feature space and can be directly compared.
3.2 Evaluation of CAV Quality
We propose two metrics (concept accuracy and concept-to-class accuracy) to evaluate the CAV quality, based on the definition that CAV quantifies the importance of a concept to the class prediction.
Concept accuracy. Concept accuracy estimates whether the CAV faithfully represents its corre- sponding concept. To this end, the accuracy Acc(U,) for CAV v, is calculated as the test accuracy of the binary classification model. Specifically, let C denote the set of all concepts, concept accuracy denotes cardinality of set):
Sconcopt is finally calculated averagely over all concepts (note that ||
Sconcept
1 |CT
Σ LEC
Acc(we)-
Concept-to-class accuracy. The original CAV simply determines whether the trained CAV w. has positive relation to a class l in the target model, by simply determining whether the angle between ve and Vhg (f(z)) (a=) (the gradients of classification logit for class & on fx)) is acute. However, this metric is too simplified to reflect the degree of connection between CAVs and classes. Therefore, we propose concept-to-
accuracy estimate the extent to which the CAV ve relates class according to the cosine similarity between UG and h (=)). We construct the ground-truth set (D) of positively-rehee
concept-clase pairs by calculating the similarity between the concepts and the [39] as used in CLIP-Dissed [27]) and selecting the concept-class pairs with the similarity exceeding a threshold c. Finally, concept-to-class accuracy Sconcopt, to class is calculated averagely over all ground-truth concept-class pairs D:
class names with a language model (like all all-mpnet-basee
U (hx(f(z))
Sconcept to class
(2)
[D]
Ue (h+((()))
(C,A)ED
3.3 LG-CAV
In this section, we first propose framework on how to transfer the concept knowledge from VL model to the LG-CAV, then propose three modules into this framework to further improve the quality of LG-CAV: Gaussian alignment (GA) module, concept ensemble (CE) module, and deviation sample reweighting (DSR) module.
3.3.1 Framework
The features of concept descriptions extracted by VL model cannot be directly used supervise the training of CAVs, because VL model and the target model have different feature spaces. Therefore, we propose an ingenious method that transforms the concept knowledge of VL model into activation values on a common pool of images (also named probe images, denoted as R) and trains the LG-CAV from these activation values, inspired by previous concept- based method [7] that adopts probe images to recognize common units of different models.
Specifically, this method consists of three steps to train LG-CAV v... (1) Calculate the activation
values {Act., (f (x)) : x E R} of u. on the image features (f(x) x E R) extracted by the target model. (2) Calculate the activation values (Act (c) (finng (z:) E R) of grext (c) on {Shing (x)
E R} with { (Act (r) (9img (x)) x E R), and the corresponding loss function LLG-CAV is shown in
E R) using VL model. (3) Train V. by aligning (Act,, (f(x))
Equation (note that || ||2 denotes the 1.2 norm, J. Itexts Fimg are freezed, and only n. is trainable).
LLG-CAV
1
Σ (Actu, (f(x)) Act (gling (aung(=)))) ZER
(3)
|
We calculate activation value as cosine similarity between two vectors (e.g. (f())) fe=)(
Te [ ), because cosine similarity is invariant to the norms of feature vectors which differ a lot in different models. Therefore, the LG-CAV learns to recognize images with the corresponding

---

Original CAV
Positive
Binary Classifier
Images (Pa)
cd
Positive Image Features
Nogative Images
(N.))
Negative Image Features
Original CAV
LG-CAV
DSR Module
Frozen Module
LG-CAV (To be trained for target model)
Similarity
Similarity
(a. P.)
Activation Value
Function
LIC-CAV
Target Image Features
9ime
VI. Activation Values
Probe Images (R)
VL Image Features
Similarity
Similarity
The learned concept
A dog animal Augment with white coat.
Ensemble
Srme
Text Features
Distribution of Activation
Distribution Activation
Values model)
Values (Target modal)
Module
GA Module
The original CAV is defined as the weight vector for its represented concept in the binary linear classifier. Bottom: The LG-CAV is learned by mimicking the activation values of its represented concept on the probe images R using VL model. Besides, three modules (GA module, CE module, and DSR module) are proposed to enhance the quality of LG-CAV.
Figure 3: Top:
concept and the images without the corresponding concept. Besides, compared with the original binary classification task for CAV training, the activation values encompass richer information about the extent to which the concepts exist in the images, thus facilitating the training of LG-CAV.
3.3.2 Gaussian Alignment Module
However, directly utilizing the above LLG-CAV is not guaranteed to improve the quality of CAVs (see experiments in subsubsection 1.2), because the activation values calculated from VI. model and the target model have significantly different distributions (due to the huge difference of feature space in the models). To address this problem, Gaussian alignment (GA) module aligns the distribution of activation values for VL model with that for the target model, based on the observation that the distribution of activation values resembles Gaussian distribution (Figure 2 (B)). GA module consists of three steps: (1) Calculate the cosine similarity for each pair of features in (f(x) æ E R}
simulate the activation values from the target model, which will be
f(())(()))))))))))))) f(a')
xt, x. E RJ. (2) Estimate the parameters (mean & standard deviation) of Gaussian distribution X ~N(Harrges Parggt)) for A (activation values from the target model), and X N N((VL oi for R. (activation values from VL model). (3) Calculate the transformation function for these two Gaussian distributions, then use it to transform each Act Actai (9rmg (Rimg(x)) to
(Act.rezaee) (9ing (x))
be Act Actyron grovcle) (gimg (x)), as shown in Equation 4.
Act growee (e) (#)) Act (cd (fimg(a)) PVL
Otarget + Ptarget
OVL. N(JVI,, VLL) into a standard Gaussian distribution (X (0, 1)), then transforms the standard Gaussian distribution into X N ((Haagett: P.rgett). ). as shown in Appendix A.
Detailedly, this transformation first transforms
3.3.3 Concept Ensemble Module
Concept ensemble (CE) module employs data augmentations the concept descriptions, thus enhancing the comprehensiveness of the concept. Specifically, instead of using single prompt like "a photo of the concept e" (that will be fed into 9texa), CE module uses multiple prompts (e.g., "a
5

---

bright photo of the concept c", "a cropped photo of the concept c") to describe c. These concept prompts follow the class prompts in the original CLIP model, as demonstrated in Appendix C.2. Next, Itext will encode these augmented prompts into text features, and generate the augmented text features Gtext (c) by averaging them.
3.3.4 Deviation Sample Reweighting Module
Deviation sample reweighting (DSR) module optimizes the selection of probe images, by allocating higher training weights the probe images that more stably represent the concept. To this end, DSR module estimates the weight the probe image æ according to the standard deviation of its similarities with the ground touund-tuuth positive images Pc. using three steps: (1) Calculate cos (2;, Pa)
f()))((()) || x' E Pa... (2) Calculate the standard deviation s Po) of cos P.). Note that std std,( Pe) R. and lower , (x, Pe) indicates more stable concept representation of (3) The weight (, )) is finally calculated by normalizing the opposite of std (x,P.) with a softmax operation, shown in Equation 5. Note that the averaged value of all sample weights equals
1, and the softmax function can be replaced by other normalization functions.
exp std (x, Pc))
( Po) |
(5)
Σ exp stdy (P))
s' E'
3.3.5
Loss Function
With the above three modules, the updated LG-CAV loss ÉLG-CAV calculated as in Equation Besides, when positive images Pe and negative images N₂, are provided, LLG CAV can be added into the training framework of original CAV to enhance the CAV quality, and the total loss function Ltotal will be Liotal= = Lcls + LLG-CAV (note that Lets is the classification loss for the original CAV).
LLC-CAV 1
Σ PER
[wy(o, ..).c (Acto, (Acto.(((x))) Achguae (c) (Nimg (x)))²
(6)
||]||
3.4 Model Correction
Due to the lack of high-quality CAVs that sufficiently relate to all classes in the dataset, most existing CAV methods are confined to explaining the local behavior of target model using a very limited number and variety of CAVs. Different from these methods, our proposed method can train sufficient quantity of high-quality LG-CAVs that relate to all classes in the dataset, thus having great potential to improve the performance of target model in an interpretable manner.
Specifically, our model correction method alleviates spurious correlation in the target model (i.e., incorrect dependence of class unrelated concepts) to improve the model performance. To this end, we fine-tune the target model to align the prediction of each class with its strongly-related LG-CAV. However, directly aligning the gradients for each class with the LG-CAV would easily interfere with other correct concepts and hurt the performance. To align them in a more soft manner, activation sample reweighting (ASR) module allocates different training weights to the images of each class, according to the activation values of the corresponding LG-CAV on them. Assume concept e strongly related class k; and let Iç denote the training images of class k then ASR module reweights image x of It in two steps (similar to DSR module): Act.,(((x) (f(z)) (the activation value of LG-CAV v. on æ). (2) Calculate the weight w fine tune-ttee/gn (x) by normalizing Acto, (fx)) with a softmax operation, as shown in Equation 7.
(1) Calculate
exp (Ache, (f(x)) exp (Actes (S(x'))
fine tune (x: = |Z+|l
Σ
Next, f Whne-tune (x)) will be used as the weight of image x in the classification loss during fine-tuning, In this manner, the target model learns to predict class from the samples activated more highly by the LG-CAV v., thus better aligning the prediction of class & with its strongly-related concept Besides, this method requires no further training on the backbone f (only uses f to extract image features and trains the subsequent layers), leading minimal training cost.

---

Table 1:
The comprehensive evaluation of concept accuracy (%) for different CAVs the Broden
dataset. The results are on nine backbones pre-trate pre-trained on ImageNet (Note that Res denotes ResNet, Dense denotes DenseNet) averaged over 4 runs with different seeds. Bold font denotes the best result.
Method
Res-18 Res-34 Res-50
Dense- 121
Dense-169
VGG-13
VGG-19
ViT-B ViT-L
Original CAV [16]
68.92
69.32
71.34
72.46
72.70
67.44
69.56
65.35
65.85
Text-to- Text-to-Coneept Concept [23]
70.04
71.35
72.40
73.67
74.19
68.35
70.24
67.22
66.27
OA-TCAV [40]
72.62
72.20
73.24
73.90
74.89
68.69
70.81
67.83
67.35
Ours
67.23
67.48
67.52
69.43
68,46
65.99
67.94
63.16
63.22
Ours + GA
74.89
73.47
74.19
76.28
74.70
69.63
72.31
68.99
68.41
Ours + GA CE
76.41
74.47
75.63
78.18
76.12
70.25
72.92
69.43
69.31
Ours + GA CE DSR
77.45
76.04
76.48
79.07
77.25
70.69
73.47
70.52
70.09
Table 2:
The comprehensive evaluation of concept-to-class accuracy for different CAVs on the Broden dataset averaged over 4 runs with different seeds.
Method
Res-18 18 Res-34 Res-50 Dense- 121 Dense-169 VGG-13
VGG-19 19
ViT-B
ViT-L
Original CAV [16]
6.20
7.02
7.20
6.08
6.54
5.40
5.53
7.22
7.97
Text-to- Text-lo-Concept [23]
9.48
9.06
8.73
7.42
8.33
7.52
7.07
10.70
11.09
OA-TCAV [40]
10.11
10.29
10.90
9.18
10.63
8.38
8.74
10.07
10.68
Ours
4.72
6.66
5.99
5.64
6.02
4.02
4.49
6.07
6.38
Ours + GA
16.72
16.61
16.92
15.47
15.81
14.55
15.04
17.52
17.83
Ours GA +
19.14
20.10
20.51
18.78
19.00
17.50
18.35
21.52
22.34
Ours + GA CE DSR
24.58
25.61
26.05
23.93
23.97
21.40
22.79
26.12
27.72
4
Experiments
4.1
The Quality LG-CAV
4.1.1
Experiment Settings
Datasets. We estimate the quality of LG-CAV the Broden dataset [2] (a popular concept-based dataset with 63,305 images for 1197 visual concepts). In the Broden dataset, each image may contain multiple concepts. Therefore, we collect positive samples for each concept by selecting the images containing only this concept, and randomly select the same number of images from other concepts as negative samples. Finally. the simplified Broden dataset consists of 17,746 images for 468 visual concepts. The probe images (R) for each LG-CAV are from ImageNet and the images of other concepts in the Broden dataset. Specifically, we select the most activated and the same number of most least activated images by VI model.
Backbones. We follow the original CAV work [16] train CAVs for the target models pre-trained on ImageNet (from the open-sourced PyTorch package [29]). These backbones include ResNet [12], DenseNet [14], VGG [38], and Vision Transformer [6].
Parameters. To simulate the absence of images for training CAVs in reality, we set the number of positive samples (P.) and negative samples (Nc) to be 10, and the remaining images will be used as the test set. The threshold e for determining positively-related concept-class pair is 0.6. For each CAV method, || (the number of probe images) is set to be 1000. The loss function adopted here is Ltotal since P. and N₂ are available.
use SGD optimizer [34] train the CAV for 10 epochs with learning rate of 1e-3.
4.1.2 Experiment Results
Concept accuracy. Table demonstrates that without sufficient data, the accuracy of original CAV insufficient to accurately represent a concept. The first version of LG-CAV (Ours) has lower accuracy than the original CAV when no other modules are added, due to the large difference in the distribution of activation values. The added GA module aligns the activation values from target model and VL model, and improves the concept accuracy by 5.83 points averagely. The added CE and DSR modules both effectively improve the concept accuracy, and the final LG-CAV outperforms Text-to-Concept and OA-TCAV (see Appendix C.3 for the analysis of them) by large margin.
Concept-to-class accuracy. Table demonstrates that our proposed modules also enhance concept- to-class accuracy, because the CAV that better represents concept more accurately correspond

---

Table 3: The comprehensive evaluation of accuracy (%) selected classes (40 classes) of ImageNet averaged over 4 runs with different seeds.
Method
Res-18
Res-34
Res-50
Dense-121 121
Dense-169
VGG-13
VGG-19
VIT-B
VIT-L
Original
90.65
91.00
92.50
92.00
92.75
88.55
90.90
94.55
93.15
HiBug [4]
90.38
90.82
92.69
92.36
92.60
88.74
91.07
94.74
93.46
Ours
91.16
91.79
93.06
92.91
93.16
89.21
91.43
94.94
93.66
Table 4: The comprehensive evaluation of accuracy (%) for different methods on ImageNet (note that
KD denotes knowledge distillation) averaged over 4 runs with different seeds.
Method
Res-18 Res-34 Res-50
Dense-121
Dense-169
VGG-13
VGG-19
ViT-B
VIT-L
Original
69.76
73.31
76.13
74.43
75.60
69.93
72.38
81.07
79.67
Concept Distillation [11]
69.46
73.06
75.77
74.04
75.46
69.80
72.29
80.86
79.51
KD [13]
69.93
73.49
76.27
74.68
75.99
70.06
72.46
81.15
79.77
Label-free CBM [26]
N/A
N/A
71.95
N/A
N/A
N/A
N/A
N/A
N/A
Ours
70.26
73.66
76.47
74.94
76.28
70.19
72.60
81.38
80.05
its strongly-related class. The final LG-CAV also has much higher concept-to-class accuracy than Texi-to-Concept and OA-TCAV.
4.2
Model Correction
4.2.1 Experiment Settings
Datasets. We employ our model correction method on three representative datasets: Ima- geNet [5] (large-scale dataset), CUB-200-2011 [43] (a popular dataset used by many concept-based methods), and CIFAR- 1-0 [18] (small-scale dataset). The probe images (R) for each LG-CAV are also the most highly and least activated images by VI. model from their respective datasets.
The target models pre-trained on ImageNet from Py Torch, and the target models pre-trained on CIFAR-100 and CUB-200-2011 are from another open-sourced Py TyTrchCVV package, following PCBM [47]. The VL model adopted here is CLIP model with ViT-L/14 as backbone.
Backbones.
Parameters. We use SGD optimizer to train the final classification layer for 20 epochs with a learning rate of le-3. Note that different from the experiments in subsection 4. the training of LG-CAVs for these concepts does not require the original classification loss Cets and DSR module, due to the lack of ground-truth positive samples P...
4.2.2 Experiment Results
We adopt two methods find the strongly-related concept of each class in the target model, corre- sponding to two types of datasets: datasets with few classes & datasets with many classes.
Datasets with few classes. We manually collect the concept descriptions of each class from Wikipedia for these datasets (e.g., the randomly selected subset of ImageNet with 40 classes (lmageNet-40)). The selected classes and their corresponding concept descriptions can be referred to in Appendix C.I.
Appendix B. demonstrates that the trained LG-CAVs have ability to distinguish whether images contain their respective concepts. Next, we utilize these LG-CAVs for model correction with the ASR module. As shown in Table 3, our model correction method effectively improves the performance of original pre-trained model (converted from the pre-trained 1000-classes model by removing other 960 classes in the final classification layer), by an improvement of up to 0.91 points.
Datasets with many classes. Collecting sufficient high-quality concept descriptions for datasets with many classes is a challenging task. Therefore, we instead acquire the concept descriptions of each class based on its comparison with its confused class, inspired by relative CAV proposed in the original CAV work. Specifically, for the class k. we first find the confused class to which images from class k are most likely to be mispredicted by the pre-trained model, then define the concept descriptions as "a photo of class k not 1t This approach is applied to the whole ImageNet, CUB-200 LUB-200-2011 and CIFAR-100.
8

---

80
Concppe-te Class Acc.
Concept Aec.
Reston Sclation
Cancept.in Class
Tonkon Scketton
Concept Acc.
20-
60
&
VGl--1
VI1-
ViSI
15
100
500
10001
1000
1500
2000
Backbones
Backbones
Number of Probe Images
Number of Probe Images
Figure 4: Ablation experiments on probe images (selection strategy & image number).
"Tiger Cat"
Classification Logit Grad-CAM
(Before Debug)
0.1762 ("Tabby Cat")
0.0947 ("Tiger Cat")
0.7488 (Activation Value)
(After
0.7592 0.7401 0.7378 0.7231
0,4408
0.4342
TO
0.1490 ("Tabby Cat")
Debug)
LG-CAV ("a cat unimal
0.1662 ("Tiger Cat")
(A) Activation values of LG-CAV for "a cat animal with orange stripes"
with orange stripes")
(B) example correction for "Tiger Cat"
Figure (A) Activation values of the LG-CAV & (B) Model correction example.
As shown in Table 4, our method improves model performance on the whole ImageNet in an interpretable manner based LG-CAV, surpassing original model (by up to 0.68 points), Con- cept Distillation knowledge distillation, and Label-free CBM (see Appendix C.3 for the analysis of them). Besides, the results on CUB-200-2000 (Appendix B.2) CIFAR- 100 (Appendix B.3) also verify the effectiveness of our method.
4.3 Ablation Study
Selection of probe images. In the above experiments, we select the most activated and least activated images probe images R. We compare this selection strategy with random selection in this subsection. As shown in the first two figures of Figure 4, the LG-CAVs trained with this selection strategy have higher quality, because the prohe images selected by this strategy contain richer information to represent the corresponding concepts.
Number of probe images. This subsection investigates how the number of probe images affects the quality of LG-CAV. As shown the last two figures of Figure 4, when the number probe images small, increasing the quantity of probe images can improve the quality of LG-CAV. However, when the number probe images reaches a certain level (saturation), further increasing the quantity of probe images does not improve the quality of LG-CAV.
Additionally, we provide more ablation experiments on the choice of VI. model, the coefficient of LG-CAV loss, and the depth of extracted image features in the target model in Appendix B.5.
4.4 Visualization Results
Activation values of the trained LG-CAV. Figure 5 (A) demonstrates the activation values of trained LG-CAV its highly-activated and lowly-activated images, indicating that the trained LG-CAV can accurately activate images that contain the corresponding concepts.
Examples of model correction. As shown in Figure 5 (B), image of "Tiger Cat" is misclassified as "Tabby Cat" by the target model (with ResNet18 as backbone) before model correction. During model correction, ASR module mitigates spurious correlation of the target model by aligning the prediction of "Tiger Cat" with its strongly-related concept "a cat animal with orange stripes". This image is activated by the LG-CAV of this concept with high activation value (0.7488), thus the classification logit for "Tiger Cat" increases after model correction. Besides, we utilize Grad-CAM [37] to attribute the prediction of "Tiger Cat" in the target model, and it shows that the attribution map focuses more accurately on the cat's body after model correction.
Furthermore, provide more visualization results in Appendix D.

---

Conclusion
In this work, we propose LG-CAV to address the data-scarcity problem of original CAV by transferring the extensive concept knowledge from VL model. Specifically, LG-CAV mimics the activation values from VL. model on the probe images learn these concept knowledge. Besides, we propose Gaussian alignment (GA) module, a concept ensemble (CE) module, and a deviation sample reweighting (DSR) module to further enhance the quality of LG-CAV. Furthermore, we go beyond previous CAV methods by generalizing LG-CAV to model correction, with human-understandabbe method that aligns the class predictions with the strongly-related concepts. Experiment results demonstrate that LG-CAV significantly improves the CAV quality, and model correction method outperforms existing concept-based methods by a large margin. We hope our work can provide inspiration for future interpretable methods based vision-language models.
Acknowledgements
This work supported by National Natural Science Foundation of China (62106220, U20B2066), Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies, and Zhejiang Province "Pioneering Soldier" and "Leading Goose" R&D Project under Grant 2023C01027.
10

---

References
[1] Andrew Bai, Chih-Kuan Ych, Neil Y. C. Lin, Pradeep Kumar Ravikumar, and Cho-Jui Hsich. Concept gradient: Concept-based interpretation without linear assumption. In ICLR 2023. OpenReview.net, 2023.
[2] David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations, In CVPR 2017, pages 3319-3327.
IEEE Computer Society, 2017.
[3] Jiawei Chen, Hande Dong, Xiang Wang, Fuli Feng, Meng Wang, and Xiangnan He. Bias and debias in recommender system: A survey and future directions. ACM Trans. Inf. Syst.,
41 41(3):67:1-67:39) 2023.
[4] Muxi Chen, Yu Li, and Qiang Xu. Hibug: On human-interpretable model debug. In NeurlPS 2023, 2023.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR 2009, pages 248-255. leee, 2009.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR 202/. OpenReview.net, 2021.
[7] Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In ICCV 2023, pages 1934-1943. IEEE, 2023.
[8] Stefan Druc, Aditya Balu, Peter Wooldridge, Adarsh Krishnamurthy, and Soumik Sarkar. Concept activation vectors for generating user-defined 3d shapes. In CVPR Workshops 2022, pages 2992-2999. IEEE, 2022.
[9] Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. Improving CLIP training with language rewrites. In NeurlPS 2023, 2023.
[10] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept- based explanations. In NeurlPS 2019, pages 9273-9282, 2019.
[11] Avani Gupta, Saurabh Saini, and P. J. Narayanan. Concept distillation: Leveraging human- centered explanations for model improvement. In NeurlPS 2023, 2023.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR 2016, pages 770-778. IEEE Computer Society, 2016.
[13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint 1503. 02531, 2015.
[14] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In CVPR 2017, pages 4700-4708. IEEE Computer Society, 2017.
[15] Qihan Huang, Jie Song, Jingwen Hu, Haofei Zhang, Yong Wang, and Mingli Song. On the concept trustworthiness in concept bottleneck models. In AAAI 2024, pages 21161-21168 AAAI Press, 2024.
[16] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In ICML 2018, volume 80 of Proceedings of Machine Learning Research, pages 2673-2682 PMLR, 2018
[17] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML 2020, volume 119 of Proceedings of Machine Learning Research, pages 5338-5348. PMLR, 2020.
[18] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
11

---

[19] Jungsoo Lee, Eungyeup Kim, Juyoung Lee, Jihyeon Lee, and Jaegul Choo. Learning debiased representation via disentangled feature augmentation. NeurIPS 2021, pages 25123-25133, 2021.
[20] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. Hoi. BLIP: bootstrapping language- image pre-training for unified vision-language understanding and generation. In ICMI. 2022, volume 162 of Proceedings of Machine Learning Research, pages
PMIR, 2022.
[21] Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for CLIP training. In NeurlPS 2023, 2023.
[22] Yi Li and Nuno Vasconcelos. REPAIR: removing representation bias by dataset resampling. In CVPR 2019, pages 1572-9581. IEEE Computer Society, 2019.
[23] Mazda Moayeri, Keivan Rezaei, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In ICML 2023, volume 202 of Proceedings of Machine Learning Research, pages 25037-25060. PMLR, 2023.
[24] Jun Hyun Nam, Hyuntak Cha. Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. In NeurIPS 2020, 2020.
[25] Isar Nejadgholi, Kathleen C. Fraser, and Svetlana Kiritchenko. Improving generalizability in implicitly abusive language detection with concept activation vectors. In ACL 2022, pages 5517-5529. Association for Computational Linguistics, 2022.
[26] Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui- Wei Weng, Label-free concept bottleneck models. In ICLR 2023. OpenReview.net, 2023.
[27] Tuomas P. Oikarinen and Tsui- Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In ICLR 2023. OpenReview.net, 2023.
(28) Konstantinos P. Panousis and Sotirios Chatzis. DISCOVER: making vision networks inter- pretable via competition and dissection. In NeurlPS 2023, 2023.
[29] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Z. Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS 2019, pages 8024-8035, 2019.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML 2021, volume 139 of Proceedings of Machine Learning Research, pages 2021.
PMLR.
[31] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R- R-CNN: towards real-time object detection with region proposal networks. In NeurIPS 2015, pages 91-99, 2015.
[32] Laura Rieger, Chandan Singh, W. James Murdoch, and Bin Yu. Interpretations are useful: Penalizing explanations to align neural networks with prior knowledge. In ICML 2020, volume 119 of Proceedings of Machine Learning Research, pages 8116-8126. PMIR, 2020.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR 2022, pages 10674-10685. IEEE Computer Society, 2022.
[34] Sebastian Ruder. overview of gradient descent optimization algorithms. arXiv preprint arXiv: 1609.047477 2016.
[35] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds; Benchmarks for subpopu- lation shift. arXiv preprint 2000:2008.04850 2020.
12

---

[36] Jessica Schrouff, Sebastien Baur, Shaobo Hou, Diana Mincu, Eric Loreaux, Ralph Blanes, James Wexler, Alan Karthikesalingam, and Been Kim. Best of both worlds: local and global explanations with human-understandable concepts. arXiv preprint arXiv:2106.0888 2021.
[37] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee localization. In ICCV 2017, pages 618-626 2017.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR 2015. OpenReview.net, 2015.
[39] Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. Mpnet: Masked and permuted pre-training for language understanding. In NeurIPS 2020, 2020.
[40] Rahul Soni, Naresh Shah, Chua Tat Seng, and Jimmy D Moore. Adversarial tcav-robust and ef- fective interpretation of intermediate layers in neural networks. arXiv preprint arXiv:2002.03359, 2020.
[41] Wolfgang Stammer, Patrick Schramowski, and Kristian Kersting, Right for the right concept: Revising neuro-symbolic concepts by interacting with their explanations. In CVPR 2021, pages 11199969.... IEEE Computer Society, 2021.
[42] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303. 2023.
[43] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[44] Kai Yuanqing Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds object recognition. In ICLR 2021. OpenReview.net, 2021.
[45] Hojin Yang, Tianshu Shen, and Scott Sanner. Bayesian critiquing with keyphrase activation vectors for vac-based recommender systems. In SIGIR 2021, pages 2111-2115, ACM, 2021.
[46] Jingwen Ye, Zunlei Feng, and Xinchao Wang. Flocking birds of a feather together: Dual-step GAN distillation via realer-fake samples. In VCIP 2022, pages 1-5. IEEE, 2022.
[47] Mert Yüksekgönül, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2023. OpenReview.net, 2023.
13

---

A Proof
To address the problem that activation values calculated from VL model and the target model have significantly different distributions (due to the huge difference of feature space in the two models), our proposed Gaussian alignment (GA) module first estimates the Gaussian distribution of activation values for the target model and VL model (XvL ~ N(HYL, oiL) and target ~
N (Ptargtt: farget)) )), then transforms each geea (c) (ging(2) to be Act greale)) (c) (2)g(())) according the transformation between these two Gaussian distributions:
Act Act gloos (e) (9ing (x)) Act (c) (9img(()))) - PVL.
Otarget + Itarget
(8)
OVL
Specifically, this transformation first transforms ~ N(VLL Vi) into a standard Gaussian
distribution (X (O, 1)). then transforms the standard Gaussian distribution into X N (Prarget: Farget), which can be proved in the following theorem.
Theorem:
The Gaussian distribution XVI. ~ NuVI,. Vi.) of activation values for VL model can be converted into Gaussian distribution X Ktargtt ~ "N(aarget of iarget)) of activation values for the target model with linear transformation: Xtarget XvI.-uVL
Otarget + Mtarget-
OVL
Proof. We first prove that XVL N(VVL, Vi.) can be converted into the standard Gaussian distribution Sttandard N(0, 1) with linear transformation: XVL-PNL
0vl.
As Gaussian distributions, the cumulative distribution function of Xv1. is Fxvi. P(XvI. S k) _ 1
VENOVL. exp (#-WyL)) drx and the cumulative distribution function of sstandard is Fxtundard (A:) =
20
P (tsanndrd S k) = - v2n exp ( )
X XVI.-PVI then the cumulative distribution function of X will be Fx(k)= P(X <k) ovL
P( XVL- < k) = P(Xv1. S le +vV) = - +ov. ViROVI. exp
(2 (z-pv1.) dr. Next,
20t
let = 2 OVL. + pVL. then dr = ovı,d, and (x = S V2nOVL exp (
(s-0vv.) y
Jovi,dz=
20
k 1 V2 exp )z Therefore, Fx (k) (the cumulative distribution function of X) is equal to
F Fx (k) (the cumulative distribution function of Xstandara), proving that X is identical with X Xstandard-
Likewise, X ttarget ~ (Mrargets of Larger) can be converted into the standard Gaussian distribution Xstandard (O, 1) with linear transformation:
Xtarget
2 Sarrge
X Xstandard XVL-HVL
Xtxrins -harget
OVL
Xstandard
diarget
Therefore, by combining these two equations, Xtargot be converted from Xv1. with linear transformation, as shown in Equation 10.
X Xtarget Xstandard Ttarget + Ptarget
XVL PVL -target + Mtaget
(10)
OVL
B Experiments
B.1 LG-CAV Quality
For the datasets with few classes, we manually collect the strongly-related concept descriptions of each class from Wikipedia. After training the LG-CAVs for these concept descriptions, first verify
14

---

Table 5: The averaged Recall 100 (%) of LG-CAVs ImageNet-40. Random CAV denotes the
randomly initialized CAV. Bold font denotes the best result.
Method
Res-18 Res-34 Res-50 Dense-121 Dense-169 VGG-13 VGG-19 VTT-B VIT-L
Random CAV
4.98
5.07
4.62
5.00
5.04
4.60
4.74
4.92
5.08
LG-CAV
76.25
73.13
76.88
75.63
79.38
83.75
85.63
84.38
78.75
Table The evaluation of accuracy (%) for different concept-based methods on the CUB-200-201 dataset over five backbones. The results of PCBM & Trustworthy CBM & Label-free CBM are from their original paper.
Method
Res-10 10
Res-12
Res-14
R6s-16
Res- 18
Original
72.23
72.73
75.23
76.35
76.67
PCBM [47]
N/A
N/A
N/A
N/A
58.80
PCBM-h [47]
N/A
N/A
61.00
Trustworthy CBM [15]
N/A
N/A
N/A
N/A
68,30
Label-free CBM [26]
N/A
N/A
N/A
N/A
74.31
Ours
72.74
73.14
75.66
76.67
77.31
the quality of LG-CAVs by calculating the Recall @ 100 performance of them. Specifically, given an LG-CAV, we use CLIP model (with ViT-L/14 as backbone) to find 50 test images that best match its corresponding concept as ground-truth, then calculate the Recall @ 100 by comparing them with the 100 most activated test images calculated from LG-CAV. Table 5 illustrates the averaged Recall @ 100 performance over LG-CAVs, indicating that the trained LG-CAVs have the ability to distinguish images containing their respective concepts.
B.2
Model Correction on CUB-200-2011
Table 6 demonstrates the experiment results of model correction on CUB-200-2011 over five back- bones (ResNet10, ResNet12, ResNet14, ResNet16, and ResNet20), indicating that our model cor- rection method shows superior performance to the original model. Besides, our method naturally exceeds other concept-biid baneept-baeed interpretability methods (PCBM [47], Trustworthy CBM [15], and Label-free CBM [26]) that sacrifice performance for the sake of interpretability.
B.3 Model Correction on CIFAR-100
Table 7 also verifies the effectiveness of our method on small-scale dataset (CIFAR-100) over five backbones (ResNet20, DenseNet40, PreResNer20, SEResNer20, and SEPreResNet20).
Table 7: The evaluation accuracy (%) for different methods on the CIFAR-100 dataset over five backbones.
Method
Res-20
Dense-40
Pre-Res-20
SE-Res-20
SE-Pre-Res-20
Original
70.36
75.10
69.78
71.46
71.69
Ours
70.87
75.59
70.19
71.87
71.94
B.4 TCAV Score
In the main paper, we use concept-to-clase accuracy to estimate the similarity between the CAV and its strongly semantic-related class. The original CAV adopts simpler but incomplete metric named TCAV score for this purpose, which estimates whether the trained CAV v, has positive relation to its positively-related class & by simply determining whether the angle between v. and (x)) (the gradients classification logit for class k) acute. Table 8 demonstrates that our proposed modules also effectively improve the TCAV score.
15

---

Table 8: The comprehensive evaluation of TCAV score (%) for different CAVs the Broden dataset. Bold font denotes the best result.
Method
Res-18
18 Res-34
Res-50 Dense- Dense-169
VGG-13
VGG-19
ViT-B
ViT-I.
Original CAV
76.15
82.31
83.85
83.08
90.00
88.46
86.15
81.54
83.85
Ours GA CE DSR
96.15
98.46
98.24
98.28
99.03
99.23
97.69
94.62
93.85
B.5
Additional Ablation Study
B.5.1 Different VL Models
We conduct ablation experiments on how the choice of CLIP model affects the quality of LG-CAV. As shown in Table and Table 10, the I.-CAVV trained from CLIP models with ViTs (ViT- L/14, VIT-B/16, and ViT-B/32) as backbone have higher quality than those trained from CLIP models with CNNs (RN50x16) 16) as backbone, because the former CLIP models have much higher performance (zero-shot accuracy) than the latter ones. Besides, Table 9 and Table 10 demonstrates that the quality of LG-CAVs increases in the order of ViT-L/14 - ViT-B/16 - ViT-B/32, indicating that the ViTs with larger patch sizes lead to LG-CAVs with higher quality. This is because VITs with larger patch size focus more on the overall concepts in the images rather than specific local details, making the learned concept features easier transfer the target model.
Table 9:
The comprehensive evaluation of concept accuracy (%) with different CLIP models in four target backbones (ResNet18, DenseNet121, VGG13, and ViT-B/16) on Broden.
Method
RN50x16
ViT-I/14
VIT-R/16
VIT-B/32
Res-18
76.12
77.45
77.85
78.22
Dense-122
77.89
79.07
79.46
79.59
VGG-13
70.02
70.69
71.11
71.39
ViT-B
70.15
70.52
70.88
70.99
Table 10: The comprehensive evaluation of concept-to-class accuracy with different CLIP models in four target backbones (ResNet18, DenseNet121 VGG13, and ViT-B/16) on Broden.
Method
RN50x 16
VIT-I/14
ViT-B/16 VIT-B/32
Res-18
24.13
24.58
25.98
26.23
Dense-12 121
23.65
23.93
25.23
25.56
VGG-13
21.52
21.40
22.33
22.78
ViT-B
25.92
26.12
26.26
26.94
Besides, adopt other VL models (EVA-CIIP [42], LaCLIP [9], CLIPA [21]) to train the LG- CAVs. These VL models are advanced variants of the original CLIP model with more accurate vision-language alignment, and the LG-CAVs trained with these VL models have higher concept accuracy and concept-to-class accuracy, as shown in Table 1l and Table 12.
B.5.2 Coefficient of LG-CAV Loss
Figure 6 demonstrates how the coefficient of LG-CAV loss affects the quality of LG-CAV. Initially, increasing the loss coefficient will increase the quality of LG-CAV (in both concept accuracy and concept-to-class accuracy). However, when it exceeds 3.0, further increasing it will decrease the quality LG-CAV.
B.5.3 CAV Quality on Intermediate Features
In the experiments, we utilize the image features extracted from the last layer of the backbone to train CAVs, because deep features better capture high-level concepts. Besides, we also conduct experiments on the intermediate features of the target model over three backbones (ResNet18 8, DenseNet121 and ViT-B). Specifically, the depth layer for extracting the intermediate features these backbones 13, 88, 11, respectively.
16

---

Table 11: The comprehensive evaluation of concept accuracy (%) with different VL models in four target backbones (ResNet18, DenseNet121, VGG13, and ViT-B/16) on Broden. These VL models all adopt ViT-L/14 backbones.
Method
Res-18
Dense- 121
VGG-13
VIT-B
Original CAV
68.92
72.46
67.44
65.35
LG-CAV (CLIP)
77.45
79.07
70.69
70.52
LG-CAV (EVA-CLIP)
79.90
82.13
73.53
71.69
LG-CAA (LaCLIP)
77.93
79.95
71.39
71.06
LG-CAV (CLIPA)
78.45
80.16
71.42
70.80
Table 12:
The comprehensive evaluation of concept-to-class accuracy (%) with different VL models in four target backbones (ResNet18, DenseNet121, VGG13, and ViT-B/16) Broden. These VL models all adopt ViT-L/14 as backbones.
Method
Res-18
Dense- VGG-13
VIT-B
Original CAV
6.20
6.08
5.40
7.22
I.-CAV (CLIP)
24.58
23.93
21.40
26.12
LG-CAV (EVA-CLIP)
25.60
24.79
22.20
28.56
LG-CAV (LaCLIP)
26.29
25.68
22.69
28.44
LG-CAV (CLIPA)
25.46
24.85
22.10
28.23
As shown in Table 13 and Table 14, our proposed modules can still effectively improve the quality of LG-CAV (in both concept accuracy and concept-to-clan accuracy) on the intermediate features. How- ever, compared with the features extracted from the last layer, the LG-CAVs trained from intermediate features have much lower quality. Besides, intermediate features have much larger dimension size than the features extracted from the last layer, resulting in enormous training costs. Therefore, these two factors hinder the CAVs trained from intermediate features for broader applications.
Table 13: The comprehensive evaluation of concept accuracy (%) for intermediate features in three backbones (ResNet18. DenseNet121, and ViT-B/16) of the target model on the Broden dataset. Bold font denotes the best result.
Method
Res-18
Dense-121
VIT-B
Original CAV
64.68
66.96
56.04
Ours
57.36
56.46
54.96
Ours + GA
68.21
70.65
58.56
Ours GA + CE
69.25
71.09
59.45
Ours GA CE DSR
70.99
72.31
60.84
B.6 Standard Deviation
The experiment results of our main experiments are averaged over runs with different seeds, but the standard deviations of experiment results are omitted in the main paper due space limit. Table 15, Table 16, and Table 17 demonstrate the standard deviation of experiment results, indicating that the results are relatively stable in the experiments of evaluation of CAV quality and model correction. The results of four representative backbones from ResNet, DenseNet, VGG, and VIT are demonstrated, and the results of other backbones are similar.
More Experiment Details
C.1 Concept Descriptions of 40 Classes in ImageNet
For the datasets with few classes (e.g., the randomly selected subset of ImageNet with 40 classes (lmageNet-40)) we manually collect the concept descriptions of each class from Wikipedia for these datasets, shown in Table 18.
17

---

Concept-o- Class Agc.
Concept Ace.
24
70
21
65-
Ros 18
18-
Res 18
VGG 13
VOG-13
60
15-
Danee-131
VIT-B
0.5
1.0
2.0F
3.0
4.0
12
2.07
3.0
Loss Cocfficient
Loss Coefficient
Figure 6: Ablation experiments on coefficient of LG-CAV loss. the experiments of the main paper.
denotes the loss coefficient used in
Table 14: The comprehensive evaluation of concept-to-class accuracy for intermediate features in three backbones (ResNet18, DenseNet121, and ViT-B/16) on Broden. Bold font denotes the best result.
Method
Res-18
Dense-121
ViT-B
Original CAV
0.29
0.29
0.56
Ours
0.20
0.23
0.48
Ours GA Ours + GA +
0.58
0.63
0.68
0.86
1.04
0.85
Ours + GA + CE + DSR
1.40
1.80
1.16
C.2 Data Augmentation Templates in the CE Module
Concept ensemble (CE) module employs data augmentations on the concept descriptions. Specifically, instead of using single prompt like "a photo of the concept c" (that will be fed into 9text), CE module uses multiple prompts (e.g., "a bright photo of the concept c", "a cropped photo of the concept describe c. The data augmentation templates demonstrated in Table 19.
C.3 Comparisons with Baselines
C.3.1 The Quality of CAVs
We compare LG-CAV with four CAV methods: original CAV [16), Concept Gradient [1], OA- TCAV [40], and Text-to-Concept [23].
Original CAV [16]. The original CAV is defined as the weight vector for the corresponding concept in the binary linear classifier that classifies the positive images and negative images for this concept. The original CAV has poor quality when the training images for the concept are insufficient.
Concept Gradient [1]. Concept Gradient extends the original linear CAV to non-linear concept functions, improving the quality CAV trained on the features extracted from intermediate layers of the target model. In particular, when the features are extracted from the final layer of target model (linearly separable), Concept Gradient identical to the original CAV.
OA-TCAV [40]. OA-TCAV proposes adversarial training approach to improve the quality of CAV. However, still suffers from the data-scarcity problem, and thus inferior to our method.
Table 15: STD of concept accuracy (%) for different CAVs on the Broden dataset. The results are on nine backbones pre-trained on ImageNet.
Method
Res-18
Dense-121 121
VGG-13
ViT-B
Original CAV [16]
68.92 + 0.35
72.46
- 0.54
67.44 + 0.34
65.35
+ 0.41
Text-to-Concep)) [23]
70.04
0.71
73.67
0.50
68.35
0.76
67.22
0.55
OA-TCAV [40]
72.62 0.18
73.90
0.40
68.69
1 0.30
67.83
0.32
Ours
67.23
0.41
69.43
65,99
63.16
Ours GA
74.89 = 0.57
76.28
0.26
69.63
0.49
08.99-0.66
Ours + GA + CE
76.41 + 0.44
78.18
0.41
70.25
0.61
69.43-1000
Ours GA + DSR
77.25
0.38
79.07
0.31
70.69
0.50
70.52 0.23
18

---

Table 16: STD of concept-to-class aceuracy for different CAVs the Broden dataset averaged over 4 runs with different seeds.
Method
Res- 18
Dense- 121
VGG-13
ViT-B
Original CAV [16]
6.20
0.49
6.08
+: 0.55
5,40 + 0.47
7.22 = 0.66
Text-to-Concen [23] OA OA-TCAV [40]
9.48 + 0.97
7.42
1.05
7.52 + 1.13
10.70 + 1.24
10.11 0.38
9.18 = 0.43
8.38 = 0,40
10.07 = 0.32
Ours
4.72 - 1.19
5.64
1644:00 0.77
4.02 1.10
6.07 = 1.13
Ours + GA
16.72 + 0.79
15.47
1.17
14.55
1.08
17.52 + 0.94
Ours + GA CE
19.14
0.94
18.78
0.76
17.50
+ 0.74
21.52
1.15
Ours GA CE DSR
24.58
1.16
23.93
1 0.90
21.40
1.25
26.12 1 0.91
Table 17: STD of accuracy (%) for different methods on ImageNet averaged over 4 runs with different seeds.
Method
Res-18
Dense- 121
VGG-13
ViT-B
Original
69.76
74.43
69.93
81.07
Concept_ Distillation
69.46 + 0.06
74.04
0.06
69.80
0.05
80.86 + 0.03
KD
69.93 + 0.05
74.68
0.04
70.06
0.04
81.15 + 0.03
Ours
70.26 + 0.04
74.94 = 0.02
70.19 0.03
81.38 = 0.02
Text-to-Concept [23]. Similar to our proposed LG-CAV, Text-to-Concepl leverages VL model to generate CAVs, by directly mapping the features of VL model into the feature space of target model. However, it roughly conducts feature mapping with a linear projection matrix, without specialized optimization for each individual CAV like our LG-CAV. Therefore, the quality of CAV trained with Text-to- Coxccto-Concept is inferior to LG-CAV. Besides, Text-to-Coneept can only be applied to small-sized datasets with few classes (e.g., IN9 dataset [44], Living dataset [35]), without generalization ability to large datasets like ImageNet.
C.3.2 Model Correction
We compare our model correction method with four baselines; Distilltti [11], Knowledge Distillation [13], Label-free CBM [26], and HiBug [4].
Concept_ Distillation [11]. Concept_ Distillation mitigates spurious correlation in the target model by directly aligning the gradients for each class with the LG-CAV using cosine similarity. However, this approach is not applicable generic datasets like ImageNet, because it would easily interfere with other correct concepts for each class and hurt the performance, as shown in Table 17.
Knowledge Distillation [13]. Knowledge distillation transfers the knowledge from VL model to the target model by transferring the probabilistic predictions from VL model. For comparison with our method, we freeze the backbone of target model and only train the final classification layer using knowledge distillation.
Label-free CBM [26]. Label-free CBM incorporates an intermediate concept layer into the target model, and makes class predictions based on the prior concept predictions. As concept-based inter- pretable model, Label-free CBM sacrifices more performance to achieve higher model transparency, and thus naturally performs worse than our method.
HiBug [4]. HiBug leverages pre-trained large language models like ChatGPT and pre-trained vision- language models like BLIP [20] interpret the target model, and repair the model by training it on the generated data from stable diffusion model [33]. HiBug is limited to small-sized datasets (ImageNet- 40) because the data generation cost is too large for large datasets.
D More Visualization Results
We provide more visualization results generated by our method. Specifically, Figure 7 demonstrates the highly activated images (and the activation values) of LG-CAVs for eight concepts on the ResNet18 backbone. Figure 8 demonstrates eight model correction examples for the target model on the ResNet18 backbone.
19

---

Class Name
Concept Descriptions
Electric Ray
round marine animal with a stocky tail A python animal with small triangular head A spider animal with red yellow portions near the body
African Rock Python
Yellow Garden Spider
Partridge
A bird animal with thick neck and rounded wings
Toy Terrier
A dog animal with white coat and short, high-set tail
Black and Tan Coonhound
A dog animal with long cars and a strong tail
English Foxhound
A dog animal with thick skull and long muzzle
Otterhound
A dog animal with dense shaggy coat and webbed feet
Norfolk Terrier
A dog animal with hard, wiry, and straight coat A dog animal with wiry and dense double coat, triangular head, dark eyes, and small V-shaped cars
Wire Fox Terrier
Golden Retriever
A dog animal with long tails and dark eyes
Australian Kelpie
A dog animal with a lean, muscular build and soft short coat
Siberian Husky
A dog animal with thick, double-coated fur, pointed ears and bushy tail
Toy Poodle
A dog animal with distinctive dense and curly coat A fox animal with a long, bushy tail, narrow, pointed muzzle, and thick, soft fur
Red Fox
Tiger Cat
A cat animal with orange, gold, and red stripes
Leaf Beetle
An insect animal with solid and tough body A gazelle animal with tan buff coats and white rumps
Gazelle
Bath Towel
A rectangular, thin object made of fabric
Bathtub
A long, usually rectangular container
Cassette
A flat, rectangular container made of plastic
Candy Store
A room with an assortment of sweets
Desktop Computer
A computer with a rectangular chassis rectangular piece of fabric material
Doormat
Gong
flat, circular metal disc A pressurized aerosol can A small, handheld tool with sharp blade A curved or bent piece made of metal or plastic
Hair Spray
Hatchet
Hook
Laptop Computer
A portable computer with a rectangular display screen
Tights
A garment similar to leggings but is thinner
Overskirt
A short skirt
Product Packet Packaging
container that holds the product A relatively flat object with long handle small, round container for serving soups A rectangular or square shape, with small lever or button
Paddle
Soup Bowl
Electrical Switch
Toilet Seat
A flat or curved seating surface on top of a toilet bowl A soft fabric with smooth and lustrous surface
Velvet Fabric
Wall Clock
A typically circular-shaped clock face with numbers creamy beverage with pale yellow or off-white color
Eggnog
Cliff
A vertical or nearivertical rock exposure
Table 18: Concept descriptions of 40 classes from ImageNet.
20

---

Data Augmentation Templates
photo the {).
low resolution photo the ().
rendering of the
graffiti of the ()-
bad photo of the ).
cropped photo of the ().
bright photo of the [].
drawing of the [].
photo of the cool ()-
a close-up photo of the (}-
a painting of the ()-
a pixelated photo the {).
a sculpture of the O-
a plastic ()-
photo of the dirty I.
a jpeg corrupted photo the (].
blurry photo of the ().
photo of the hard see {).
good photo of the {).
close-up up photo of the ()-
the origami ().
sketch of the ().
a photo of the clean ()-
photo of the large []-
photo of the nice [).
a photo of the weird [].
a photo of small ()- black and white photo the ()-
dark photo of the ()-
Table 19: The data augmentation templates for the concept descriptions in the CE module.
21

---

0.7623
0.7594
0.7532
0.7488
0.7407
0.7390
0.7372
0.7320
dog animal with brown coat
0.7640
0.7575
0.7492
0.7485
0.7402
0.7381
0.7361
0.7289
dog animal with long tail
0.7791
0.7684
0.7680
0.7632
0.7630
0.7550
0.7544
0.7450
a dog animal with white coat
0,7592
0.7401
0.7378
0.7231
0.7212
0.7194
0.7141
0.7133
animal with orange stripes
0.7754
0.7671
0.7668
0.7607
0.7575
0.7518
0.7415
0.7414
a bird animal with blue wing
0.7445
0.7436
0.7422
0.7416 a circular-shaped object
0.7406
0.7389
0.7296
0.7271
0.7234
0.7162
0.7155
0.7134 a round container
0.7088
0.7076
0.7053
0.7007
0.7881
0.7839
0.7714
0.7648 yellow liquid
0.7636
0.7572
0.7441
0.7410
Figure Highly activated images (and the activation values) of LG-CAVs.
22

---

"Bathtub"
Classification Logit
Grad CAM
"Hatcher"
Classification Logit
Grad CAM
(Before
(Before
0.2341 ("Hook")
Debug)
("Toilet Seat")
Debug)
0.1659 ("Buhtub"))
0.1316 ("Hatchee")
(Aclivation Value)
0.7865 (Activation Value)
(After
(After Debug)
0.1975 ("Hook")
Debug)
0.1554 ("Toilet Seat")
L.-CAVV ("a
0.2465 ("Bathtub")
LG-CAV ("a small,
0.1649 ("Hatchet")
rectangular container")
handheld lool")
An example of correction for "Bathtub"
An example of correction for "Hatchet"
"Egunge
Classification Logit
Grad-CAM
"Egenog"
Classification Logit
Grad-CAM
(Before
(Before
0.3602 ("Soup Bowl")
Dehug)
0.2985 ("Soup Bowl"")
Debug)
0,2868 ("Eggnog")
0.2640 ("Eggnog")
0,8470 (Activation Valuc)
(After
0.8465 (Activation Value)
(After
0.3348 ("Soup Bowl")
Debug)
("Soup Bowl") 0.3097 ("Eggnog")
Debug)
LG-CAV ('yellow liquid")
0.3881 ("Eggnog")
LG-CAV ('yellow liquid")
An example of correction for "Eggnog"
An example of correction for "Eggnog"
HHair Spray"
Classification Logil Grad-CAM
"Gone"
Classification Logil
Grad-CAM
(Before
(Before
0.2045 ("Switch"")
Debug)
0.2275 ("Lampshade")
Debug)
0.1296 ("Haiir Spray")
0.2141 ("Gong")
0,8518 (Activation Value)
(After
0,7382 (Activation Value)
(After Debug)
0.1691 ("Switch")
0.2056 ("Lampshade")
Debug)
LG-CAV ("a
0.1853 ("Hair Spray")
LG-CAV
0.2489 ("Gong")
pressurized can")
metal disc")
An example of correction "Hair Spary"
An example of correction "Gong"
"Wall Clock"
Classification Logil
Grad-CAM
"Tiger Cat"
Classification Logil
Grad-CAM
(Before
(Before
("Packet")
Debug)
("Tabby Cat")
Debug)
0.1494 ("Wall Clock")
0.1883 ("Tiger Cat")
0.7491 (Activation Value)
(After
0.7956 (Aetivation Value)
0. 0.1632 ("Packet")
Debug)
TTD
0.2233 ("Tabby Cat")
Debug)
LG-CAV ("a
0.1708 ("Wall Clock"))
LG-CAV ("a cat animal
0.2782 ("Tiger Cat")
circular object")
with orange stripes")
An example of correction for "Wall Clock"
An example of correction for "Tiger Cat"
Figure Model correction examples.
23

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
24

---

Justification:
We have discussed the limitations of the work in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have provided the full set of assumptions and a complete (and correct) proof for each theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided the details of the experiments in the paper.
Guidelines:
25

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have provided the source codes of this paper, and we use the public datasets in this paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/ /gudides/CoeSuumissse
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ ctaass.cc/uublicee http:///wgeeeeeeeeeeeeceeee.eeee htwpa/////ggdde/////dddiiiiiiiii Stt////ooeesssiiiiieiieei
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
26

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have specified all the training and test details for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the error bars about the statistical significance of the experi- ments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We have provided the computer resources for the experiments in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
27

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurss wwtpeaurrs..c.ppuuppeeeeeeeeccce
httpss//earrsssccppbbbccc/c Ett.ii//ieeiissueeellleeeleeeeee ines?
Answer: [Yes]
Justification: The research conducted in the paper conforms, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed the societal impacts of this work in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
28

---

Justification:
The paper has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: The work uses the publicly available datasets and pre- pre-trained models.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithee thcode..e.......oeeeoeoooooo.ee. com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification:
We have provided a detailed readme file for the source codes.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
29

---

Answer:
[NA]
Justification:
This work hasn"t conducted experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work hasn't conducted experiments with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "JHNTYTZxVj", "reviewer": "Reviewer_V8uU", "summary": "The paper introduces Language-Guided Concept Activation Vectors (LG-CAV), a method to train Concept Activation Vectors (CAVs) without labeled data by leveraging pre-trained vision-language models such as CLIP. LG-CAV uses concept descriptions to guide the training of CAVs by aligning the activation values of concept descriptions on a set of probe images. To improve the quality of LG-CAVs, the authors propose three modules: Gaussian Alignment (GA), Concept Ensemble (CE), and Deviation Sample Reweighting (DSR). The paper also introduces an Activation Sample Reweighting (ASR) technique for model correction, which enhances the performance of the target model. Experiments across various datasets and architectures demonstrate that LG-CAV outperforms existing CAV methods in terms of concept accuracy and concept-to-class accuracy.", "questions": "1. Equation 3 uses cosine similarity to calculate the activation values because cosine similarity is invariant to the norms of feature vectors as claimed by the authors. However, this reasoning does not convincingly explain why cosine similarity is necessary in this context.\n2. Additionally, in lines 153-155, the authors state that \"compared with the original binary classification task for CAV training, the activation values encompass richer information about the extent to which the concepts exist in the images, thus facilitating the training of LG-CAV.\" I question why richer information can be obtained here. Is there any empirical evidence to support this claim?\n3. How does the performance of LG-CAV vary with different types and sizes of pre-trained vision-language models beyond CLIP?\n4. What are the computational costs and training times associated with the LG-CAV method compared to traditional CAV methods?", "limitations": "See above.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written.\n2. The use of vision-language models allows for training CAVs without the need for labeled data.\n3. The introduction of GA, CE, and DSR modules improves the quality of LG-CAVs.\n4. Beyond generating explanations, the method is applied to model correction, leading to improved performance in target models.\n5. The results show substantial improvements in both concept accuracy and concept-to-class accuracy compared to existing methods.", "weaknesses": "1. The method proposed in this paper does not clearly address the data scarcity problem of the original CAV methods, which was highlighted at the beginning. Although the method is effective, it is not evident why it successfully addresses the data scarcity issue.\n2. The method heavily relies on the availability and performance of pre-trained vision-language models like CLIP, which may not always be accessible or optimal for all tasks.\n3. The introduction of multiple enhancement modules increases the complexity and computational requirements of the method, which may be a barrier to practical applications.\n4. The paper could benefit from a more detailed analysis of scenarios where LG-CAV does not perform well or fails to improve over traditional methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and effort to provide valuable suggestions for this paper! We will make rigorous revisions to the paper based on the review!\n\n> **Weakness 1. Why it successfully addresses the data scarcity issue.**\n\n**Response:** Previous methods can only train the CAV for the target concept using the **human-collected** positive & negative images. Our method tackles the data scarcity issue by transferring the abundant concept knowledge from CLIP model to LG-CAV for the target model. Specifically,  the LG-CAV is trained by learning the similarity scores of CLIP model on a pool of **unlabeled images** (which can be easily obtained from the Internet).\n\nFor example, for a concept named ``a cat animal with orange stripes'', CLIP model can calculate the similarity score of this concept on the unlabeled images, by comparing the text features and the image features. The images that are closer to this concept will be assigned a higher similarity score, owing to the excellent cross-modal ability of CLIP model. After learning these similarity scores, the LG-CAV also has the capacity to discern the similarity between an image and this concept, as shown in Figure 5 (A) of the main paper.\n\n> **Weakness 2. Relies on CLIP model.**\n\n**Response:** CLIP model has demonstrated its excellent cross-modal ability in numerous tasks. By transferring the ability of CLIP model to the target model, our experiments have verified that our method can be applied to universal concepts, e.g., the concepts from the ImageNet dataset and the Broden dataset. Although current CLIP models may have some limitations, the experiments in Section B.5.1 of the Appendix show that our method can be generalized to different types of CLIP models, indicating that it has great potential to be adapted with the next-generation CLIP models in the future.\n\n> **Weakness 3. Computation cost.**\n\n**Response:** Actually, our method only uses an alignment loss function to learn the activation values from VL model, without adding new parameters. Besides, the coefficients of the added modules (*e.g.*, the sample weights of ASR module) can be calculated **only once** before training, requiring no redundant calculation at each step of training and saving the training cost. As shown in Table 1, compared with the original CAV, our method increases only 6% to 7% training time but significantly improves the CAV quality.\n\nTable 1: Training time for 468 concepts in the Broden dataset on one A800 GPU.\n||ResNet18|DenseNet121|ViT-B|\n|:-----|:-----|:-----|:-----|\n|Original CAV|6.89 Hours|8.68 Hours|8.22 Hours|\n|LG-CAV|7.38 Hours (+7.11%)|9.22 Hours (+6.22%)|8.72 Hours (+6.08%)|\n\n> **Weakness 4. Failure cases.**\n\n**Response:** Our work may fail in some special datasets like MNIST, as CLIP model achieves only 88% zero-shot image classification accuracy on MNIST. Nevertheless, our method can be applied to universal images, and has the ability to be adapted to the next-generation CLIP models with better performance.\n\n> **Question 1. Why cosine similarity is necessary.**\n\n**Response:** Due to the difference in feature dimensions between the target model and VL model, other metrics (e.g., Euclidean distance) can cause significant variation in the similarity scores between features from the target model and CLIP model, making them unable to compare and resulting in poor performance. Cosine similarity naturally mitigates this problem with a normalization operation to constrain the similarity scores from different models within [-1, 1], making them comparable.\n\nBesides, the existing CLIP models also use cosine similarity to compute the similarity between text and image features. We follow this metric to maintain the performance of the CLIP model. As shown in Table 2, cosine similarity achieves better performance than the other two metrics on three backbones.\n\nTable 2: Concept accuracy & concept-to-class accuracy in the Broden dataset with different metrics.\n||ResNet18|DenseNet121|ViT-B|\n|:-----|:-----|:-----|:-----|\n|Cosine Similarity|77.45 & 24.58|79.07 & 23.93|70.52 & 26.12|\n|Euclidean Distance|56.99 & 4.29|57.42 & 5.24|60.38 & 4.86|\n|Pearson Correlation|76.25 & 21.22|77.87 & 22.79|69.77 & 24.29|\n\n> **Question 2. Why richer information can be obtained here.**\n\n**Response:** This method shares a similar theory with Knowledge Distillation (KD). In KD, the student model learns the classification logits (soft label) of an input image from the teacher model instead of the original hard label, because soft label encompasses richer information about the similarity of the image with every class. Likewise, the activation value in our method indicates the similarity of the image with the target concept, with richer information than only assigning a positive or negative label to the image in traditional CAV.\n\nTo verify this, we conduct an experiment by setting the activation values above a high threshold (0.8) to 1, and other activation values are set to -1. As shown in Table 3, the loss of the similarity information leads to worse performance.\n\nTable 3: Concept accuracy & concept-to-class accuracy in the Broden dataset with/without similarity information (hard label).\n||ResNet18|DenseNet121|ViT-B|\n|:-----|:-----|:-----|:-----|\n|Original|77.45 & 24.58|79.07 & 23.93|70.52 & 26.12|\n|Hard Label|75.28 & 20.69|76.40 & 17.04|68.10 & 16.66|\n\n> **Question 3. Different CLIP models.**\n\n**Response:** We have provided the performance of our method with different VL models in Section B.5.1 of the Appendix. Overall, the VL models with higher zero-shot image classification accuracy lead to higher performance on LG-CAV.\n\n> **Question 4. Computation cost.**\n\n**Response:** The training time compared between our method and the original CAV is shown in Table 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oiptxy5fhI", "reviewer": "Reviewer_TRvX", "summary": "This paper proposes LG-CAV, a method to train Concept Activation Vectors (CAVs) for any concept without labeled image data, leveraging knowledge from pre-trained vision-language models like CLIP. \n\nThe authors introduce several techniques to improve CAV quality, including Gaussian alignment, concept ensemble, and deviation sample reweighting. \n\nThey also propose using the trained LG-CAVs for model correction to improve classification performance.\n\nExperiments demonstrate superior CAV quality and model correction results compared to existing methods across multiple datasets and model architectures.", "questions": "No specific questions. This work is of high completeness.", "limitations": "No specific limtations. This work is of high completeness.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper is well written. The framework is integral, and every component is clearly described in detail. The whole pipeline is easy to follow.\n\n2. The authors propose a novel method with feature alignment loss functions and neural modules to bridge the gap between pre-trained vision-language models and the target classification model.\n\n3. The experimental results show the proposed method achieves the best performance.\n\n4. This paper introduces two new metrics (concept accuracy and concept-to-class accuracy) to evaluate CAV quality. These two metrics are reasonable.", "weaknesses": "1. Some of the proposed modules (e.g., Gaussian alignment) seem heuristic. It would be better to give some theoretical justifications.\n\n2. The method uses a set of probe images, but it's not clear how to ensure that the trained LG-CAVs generalize beyond these images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for investing your time and effort in offering valuable suggestions for this paper! We will rigorously revise the paper based on your review!\n\n> **Weakness 1.** Some of the proposed modules (e.g., Gaussian alignment) seem heuristic. It would be better to give some theoretical justifications.\n\n**Response:** Thanks for the advice! The theoretical analysis on our proposed modules are as follows:\n\n* Gaussian alignment module.\n\n  We first prove here that the mathematic expectation of the deviation between two trained LG-CAVs is positively correlated with the deviation between the distribution of activation values for their training (with Gaussian distribution as an example), indicating that misaligned activation distributions seriously disrupt the LG-CAV training.\n\n  Furthermore, we have provided proof of how GA module aligns the distribution of activation values from VL model to the target model, in Section A of the Appendix.\n\n  **Definition:** Suppose ${\\rm Act}_1^{\\rm gt},{\\rm Act}_2^{\\rm gt},...,{\\rm Act}_M^{\\rm gt}$ are the ground-truth activation values of $M$ probe images for the target model, which follow the Gaussian distribution $\\mathcal{N}(\\mu _ {\\rm gt}, \\sigma _ {\\rm gt}^2)$. $v_c\\in\\mathbb{R}^{\\rm dim}$ is the LG-CAV, ${\\rm Act}_i^{v_c}$ is the activation value of $v_c$ on the $i$-th probe image, and the loss function is $\\mathcal{L} _ {\\rm gt}=\\frac{1}{M}\\sum _ {i=1}^{M}({\\rm Act}_i^{v_c}-{\\rm Act}_i^{\\rm gt})^2$. Besides, the deviated activation values ${\\rm Act}_1^{\\rm shift},{\\rm Act}_2^{\\rm shift},...,{\\rm Act}_M^{\\rm shift}$ and loss $\\mathcal{L} _ {\\rm shift}$ are defined likewise. $v_c^{\\rm gt}$ and $v_c^{\\rm shift}$ are the LG-CAVs trained with these two losses, respectively. $v[k]$ denotes the $k$-th element of a vector $v$.\n\n  **Theorem:** For each element $k$, the mathematic expectation of $(v_c^{\\rm shift}-v_c^{\\rm gt})[k]$ is positively correlated with $\\mu_{\\rm shift}-\\mu _ {\\rm gt}$.\n\n  *Proof:*\n\n  $$\n  \\frac{\\partial\\mathcal{L} _ {\\rm gt}}{\\partial\\mathcal{v_c}}=\\frac{\\partial\\frac{1}{M}\\sum_{i=1}^{M}({\\rm Act}_i^{v_c}-{\\rm Act}_i^{\\rm gt})^2}{\\partial\\mathcal{v_c}} = \\frac{2}{M}\\sum _{i=1}^{M}({\\rm Act}_i^{v_c}-{\\rm Act}_i^{\\rm gt})\\cdot\\frac{\\partial{\\rm Act}_i^{v_c}}{\\partial v_c}.\n  $$\n\n  Gaussian distribution has such properties:\n  * For each $x \\sim \\mathcal{N}(\\mu, \\sigma^2)$ and two constants $a$ and $b$, $a x + b \\sim \\mathcal{N}(a \\mu + b, a^2 \\sigma^2)$.\n  * For each $x_1 \\sim \\mathcal{N}(\\mu_1, \\sigma_1^2)$ and $x_2 \\sim \\mathcal{N}(\\mu_2, \\sigma_2^2)$, $x_1 + x_2 \\sim \\mathcal{N}(\\mu_1 + \\mu_2, \\sigma_1^2 + \\sigma_2^2)$.\n\n  Next, at each step of gradient descent, $v_c^{\\rm gt}$ is updated as: $v_c^{\\rm gt} = v_c - \\gamma \\frac{\\partial \\mathcal{L}_{\\rm gt}}{\\partial \\mathcal{v_c}}$, and $v_c^{\\rm shift}$ is updated likewise. Therefore, substitute the above formulas into $(v_c^{\\rm shift} - v_c^{\\rm gt})[k]$, we have:\n\n  $$\n  (v_c^{\\rm shift} - v_c^{\\rm gt})[k] = \\frac{2 \\gamma}{M} \\sum_{i=1}^{M} ({\\rm Act}_i^{\\rm shift}-{\\rm Act}_i^{\\rm gt}) \\cdot \\frac{\\partial {\\rm Act}_i^{v_c}}{\\partial v_c}[k]\n  $$\n\n  $$\n  \\sim \\mathcal{N}(\\frac{2 \\gamma}{M} \\cdot \\sum _ {i=1}^{M} \\frac{\\partial {\\rm Act}_i^{v_c}}{\\partial v_c}[k] \\cdot (\\mu _ {\\rm shift}-\\mu _ {\\rm gt}), \\frac{4 \\gamma^2}{M^2} \\cdot \\sum _ {i=1}^{M} (\\frac{\\partial {\\rm Act}_i^{v_c}}{\\partial v_c}[k])^2 \\cdot (\\sigma _ {\\rm shift}^2+\\sigma _ {\\rm gt}^2)) =  \\mathcal{N}(A \\cdot (\\mu _{\\rm shift}-\\mu _{\\rm gt}), B \\cdot (\\sigma _{\\rm shift}^2+\\sigma _{\\rm gt}^2)).\n  $$\n\n  Note that $A$ and $B$ are unrelated to the activation distributions, thereby $(v_c^{\\rm shift}-v_c^{\\rm gt})[k]$ follows a Gaussian distribution, and its mathematic expectation is positively correlated with $\\mu_{\\rm shift}-\\mu_{\\rm gt}$. (Some steps are too simplified due to space limit, we will add the full proof into the revised paper.)\n\n* Activation sample reweighting module.\n\n  ASR module allocates higher training weights to the samples with higher activation values on the corresponding LG-CAV. We prove that with this strategy, the class weight in the trained linear classifier will have higher similarity with its corresponding LG-CAV, thus mitigating the spurious correlation problem.\n\n  **Definition:** $\\mathcal{I} _ k = \\\\{ {x_i} \\in \\mathbb{R}^{\\rm dim} \\\\}_{i=1}^{N}$ denotes the image features of all $N$ training images of class $k$. $u_k \\in \\mathbb{R}^{\\rm dim}$ denotes the class weight for class $k$ in the linear classifier (total $K$ classes in this classifier), and $z _ {i, k} = \\langle x_i, u_k \\rangle$ denotes the inner product. $\\omega_i$ is the weight calculated from ASR module ($\\omega_i > 0$), higher $\\omega_i$ indicates that the corresponding LG-CAV is more similar with $x_i$.\n\n  **Theorem:** The $u_k$ trained with $\\mathcal{L} = -\\frac{1}{N} \\sum_{i=1}^{N} \\omega_i \\cdot \\log \\frac{\\exp(z_{i, k})}{\\sum_{t=1}^{K} \\exp(z_{i,t})}$ (weighted cross-entropy loss) is more similar with the corresponding LG-CAV than the $u_k$ trained with $\\mathcal{L} = -\\frac{1}{N} \\sum_{i=1}^{N} \\log \\frac{\\exp(z_{i, k})}{\\sum_{t=1}^{K} \\exp(z_{i,t})}$ (original cross-entropy loss).\n\n  We leave out the proof here due to space limit, and will add the proof into the revised paper.\n\n> **Weakness 2.** The method uses a set of probe images, but it's not clear how to ensure that the trained LG-CAVs generalize beyond these images.\n\n**Response:** Our experiments demonstrate that the probe images with more diverse relation (activation values) with the target concept lead to higher performance of the trained LG-CAVs, because these probe images contain richer knowledge for discriminating the target concept. Therefore, in this work we choose to expand the range of probe images (using ImageNet) to increase their diversity and guarantee the generalization ability. Actually, the range of probe images can be easily enlarged because the unlabeled images can be easily obtained from the Internet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CXwbf6abzw", "reviewer": "Reviewer_sN5p", "summary": "This paper proposed a LG-CAV model that leverage the pretrained vision language model to train CAV without label. This includes a concept ensemble model that employ data augmentation on concept text, a DSR module that optmize the selection of probe image and a model to align prediction of class to concepts called ASR. Experiments showed that this framework achives higher CAV quality.", "questions": "see the weaknesses", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The methodology is carried out clearly. The problem is important and the authors have got some good results. The experiments show its proposed model’s effectiveness.", "weaknesses": "1.Since LG-CAV leverage pretrained model, I am wondering whether this framework can handle unseen class besides just supervised setting such as novel category detection/generalized category detection. For example, instead of pick 40 classes from ImageNet, could you use 20 for training and the other 20 for testing? \n\n2. From figure 5 we can see that not all concepts are related to the input sentence. I am curious how the similarity threshold is selected.\n\n3. Evaluation of concept-to-class accuracy will need human evaluation of concept. How this is done in detail?\n\n4. From Table 4, the improvement compare with LG-CAV and other baseline is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time and effort to providing valuable suggestions for this paper! We will rigorously revise the paper based on your review!\n\n> **Weakness 1.** Since LG-CAV leverage pretrained model, I am wondering whether this framework can handle unseen class besides just supervised setting such as novel category detection/generalized category detection. For example, instead of pick 40 classes from ImageNet, could you use 20 for training and the other 20 for testing?\n\n**Response:** The LG-CAVs trained on one dataset can be transferred to another dataset, by utilizing the knowledge from the pre-trained VL model. We verify this on a more challenging setting by transferring the LG-CAVs trained on ImageNet to the Stanford Dogs dataset and the CUB-200-2011 dataset, instead of splitting the 40 classes into 20 training classes and 20 test classes (since the used backbones pre-trained on ImageNet have access to the total 40 classes).\n\nSpecifically, we train the LG-CAVs for each class of Stanford Dogs and CUB-200-2011, using ImageNet data as probe images. Next, we use each trained LG-CAV as the weight of a binary classifier, and test its classification accuracy on the test images from the corresponding class and the same number of test images from other classes. Table 1 and Table 2 demonstrate the average classification accuracy of the LG-CAVs trained using only ImgeNet data can achieve 80% to 90%, without a large performance gap with the LG-CAVs trained on the original datasets.\n\nTable 1: LG-CAV accuracy on Stanford Dogs with different training data.\n|Training Data|ResNet18|DenseNet121|ViT-B|\n|:-----|:-----|:-----|:-----|\n|ImageNet|84.76|87.68|93.73|\n|Stanfords Dog|86.97|94.02|97.05|\n\nTable 2: LG-CAV accuracy on CUB-200-2011 with different training data.\n|Training Data|ResNet18|DenseNet121|ViT-B|\n|:-----|:-----|:-----|:-----|\n|ImageNet|77.53|80.41|85.53|\n|CUB-200-2011|80.86|84.52|87.60|\n\n> **Weakness 2.** From figure 5 we can see that not all concepts are related to the input sentence. I am curious how the similarity threshold is selected.\n\n**Response:** Actually, Figure 5 demonstrates the high activation values of a trained LG-CAV on the left images and the low activation values on the right images, which indicates that the trained LG-CAV can discriminate whether the images are related to the target concept. We will add more annotation text to this figure for clarification in the revised paper.\n\n> **Weakness 3.** Evaluation of concept-to-class accuracy will need human evaluation of concept. How this is done in detail?\n\n**Response:** Actually, the evaluation of concept-to-class accuracy requires no human evaluation. Instead, we follow the previous work CLIP-Dissect [1] to utilize a pre-trained language model to determine the similarity between a concept and a class, and select the concept-class pairs with high similarities as ground-truth. Next, for each selected concept-class pair, the concept-to-class accuracy of the corresponding CAV is evaluated according to the similarity between the CAV and the corresponding class in the target model.\n\n> **Weakness 4.** From Table 4, the improvement compare with LG-CAV and other baseline is limited.\n\n**Response:** Thanks for pointing out this issue. In Table 4, our model correction method freezes the backbone of the model and only trains the final classification layer, verifying that LG-CAV can mitigate the spurious correlation problem for performance improvement with minimal training cost. This paves the way for training LG-CAVs for the middle layers of the backbone, and using them to supervise the training of backbone for more performance improvement in the future.\n\n[1] Oikarinen et al., Clip-dissect: Automatic description of neuron representations in deep vision networks, ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper initially received mixed reviews. Reviewer V8uU expressed concerns, giving a negative score, about the rationale for the model’s success, its reliance on a pretrained model, and its increased complexity. However, the author rebuttal successfully addressed these concerns, prompting the reviewer to raise their score. Consequently, all the reviewers acknowledged the value of the work based on its novelty, presentation, and effectiveness. Based on these recommendations, the AC also recommends the paper for acceptance.
Decision: Accept (poster)