Auxiliary Losses for Learning Generalizable Concept-based Models
Authors: Ivaxi Sheth
Samira Ebrahimi Kahou
Venue: NeurIPS 2023 poster (2023)
Keywords: Interpretability
concept bottleneck models
explainability
TL;DR: Propose multi-task learning and orthogonal loss for concept bottleneck models

Abstract: The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts.  While beneficial, CBMs have been reported to often learn irrelevant concept representations that consecutively damage model performance. To overcome the performance trade-off, we propose a cooperative-Concept Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) to encourage the separation between the concept representations and to reduce the intra-concept distance. This paper presents extensive experiments on real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that our proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.

Auxiliary Losses for Learning Generalizable Concept-based Models
Ivaxi Sheth
HeSPh-Helmholtz Center for Information Security ivaxi. she.oiistthhccsp.eee
Samira Ebrahimi Kahou
École de technologie supérieure, Mila, CIFAR AI Chair
samira wttahiiii--khhhuttttaalll. ca
Abstract
The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human- undernsndddtttlbllleeee high-level concepts. While beneficial, CBMs have been reported often learn irrelevant con- cept representations that consecutively damage model performance. To overcome the performance trade-off, we propose cooperative-Cone Conccet Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-graine line-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) encourage the separation between the concept represen- tations and to reduce the intra-concept distance. This paper presents extensive experiments real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.
Introduction
The recent advances in deep learning have impacted various domains such computer vision 21. natural language processing [53], and speech recognition [6]. Recently, the deployment large models 8 [51] has led various concerns regarding privacy and safety since machine learning models are often considered black boxes. With the increasing use of such deep learning models in daily human life and their wide deployment, it essential to understand model behaviors beyond final prediction. Since neural networks are considered opaque decision-makers, makers, inaccurate decisions by models in applications such medicine [18] or autonomous driving (59) lead to catastrophe for humans. To understand the inner workings of such black-box neural networks, the field of XAI [32]
has emerged in recent times. Concept Bottleneck Models (CBMs) 27 are family of neural networks that enable human interpretable explanations.
Concept-based models introduce bottleneck layer before the final prediction. This bottleneck layer consists of humaa--nterpretable concept predictions. For example, in the context of images of animals, these concepts could be "mane" in the case of lion "black and white stripes" in the case of zebra.
WWrk done when the author was Mila, École de technologic supéricure.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Concept bottleneck modelss (CBMs) map input images to interpretable concepts, which in turn are used to predict the label. The intermediary concept prediction allows human supervisor interpret and understand the concepts influencing the label prediction. In addition to explainability, CBMs offers an interesting paradigm that allows humans to interact with explanations. During inference, supervisor can query for explanations for a corresponding label, and if it observes an incorrect concept-based explanation then the supervisor can provide feedback.
While CBMs present benefits with models' explainability, Mahinpei et al. 33 have shown that concept representations of CBM may result in information leakage that deteriorates predictive performance. It is also noted that CBM may not lead to semantically explainable concepts [34]. Such bottlenecks may result ineffective predictions that could prevent the use of CBMs in the wild.
Along with model transparency, another challenge that modern neural networks face is robustness to distributional shifts [32] Deep learning models fail to generalize in real-world applications where datasets are non-iid [17]. The absence of comprehensive study examining the behavior of
CBMs under distributional shifts a significant limitation, potentially impeding their application in real-world scenarios.
In this work, we propose cooperative-CBM (coop-CBM) model aimed at addressing the performance gap between CBMs and standard black-box models. Coop-CBM uses an auxiliary loss that facilitates the learning of a rich and expressive concept representation for downstream task. To obtain orthogonal and disentangled concept representation, we also propose concept orthogonal loss (COL). COL can be applied during training for any concept-based model to improve their concept accuracy. Our main contributions are as follows:
We proposed a multi-task learning paradigm for Concept Bottleneck Models introduce in- ductive bias in concept learning. Our proposed model coop-CBM improves the downstream task accuracy over black box standard models.
Using the concept orthogonal loss, we introduce orthogonality among concepts in the training of CBMs.
We perform an extensive evaluation of the generalisation capabilities of CBMs on three different distribution shifts.
We looked at using human uncertainty metric for interventions in CBMs during test-time.
2
Related Works
Concept-based Models Early concept-based models that involved the prediction of concepts prior to the classifier were widely used in few-shot learning settings ( 62] Other works propose to predict human-specified concepts with statistical modeling [28] 29). Unsupervised concept learning methods use concept encoder to extract the concepts and relevance network for final predictions [4. 43). Although these methods effective interventions. Concept whitening [[]] was introduced as a method to plug an intermediate layer in place of the batch normalization layer of Convolutional Neural Network (CNN) to assist the model in concept extraction. CBM [27] extends the idea by decomposing the task into two stages: concept prediction through neural network from inputs, and then target prediction from the concepts. Many works have proposed models built on CBMs to either improve the downstream task There has been a line of work extending
useful in the absence of pre-defined concepts, they do not enable
accuracy 44 60 26
or mitigate the concept leakage [33 20
CBMs to real-world applications such as medical imaging [58] 13 autonomous driving [44] and reinforcement learning [14 CBMs require annotated concepts which poses challenge for their applications to large-scale image datasets. Yuksekgonul et al. (58) propose using concept activation vectors (25 and Oikarinen et al. [8] used multimodal models such as CLIP 44) to annotate concepts for CBMs. Although these either require concept bank or suffer from pretrained model's biases [58]
Alternative losses (CE) loss. Several variants of the CE loss have been explored in the past to improve the discriminative Ranasinghe et al. [41], Vorontsov et al. 54 introduce the use of orthogonality in feature space to encourage inter-class separation and intra-class clustering. Our work builds upon 41 by introducing orthogonality constraints in the concept feature space.
The training of CBM and its variants typically involves the use of Cross Entropy
power of learned feature representations of data 19 46
56 39

---

3 Method
3.1 Background
Consider a standard supervised learning setting for classification task, where models are trained on a dataset D {zi, ) with N data samples. Standard models aim to predict the true distribution PM (z)zz) from input Although such setting has been proven effective on vision benchmarks, users are unaware of the detailed inner workings the model. Therefore, CBMs introduces intermediate prediction of human-understande understandable concepts before the model prediction.
In the supervised concept-bas model setting, the dataset uses additional labeled concept vectors E {0, 11sh where each element indicates the presence of one of high-level concepts. This allows supervised concept learning in addition to target learning. Following a simplistic causal graph for data generation, y - CBMs consist of two models. The first model fx-c maps the input image x: to concepts c, while the second model JC-Y maps the concepts c the label y.
CBMs can be categorized by their method of training JC-Y from the obtained concept representations f(c)x)) This could be done in the following manner: jointly, where both fx=c and gC-Y are trained simultaneously end-to-end, sequentially, where fx -c is trained first, after which JC-Y is trained using Py (c)xr) representations, and finally independently. where fx-c and gC-Y are trained individually and then combined.
Interventions Interventions are a core motivator of CBMs. The bottleneck model allows for interventions by editing the concept predictions. Since CBMs consider correcting the predicted concepts through interventions during test-time, the corrected concepts are not pooo-propagated through fx-c and 9C-Y. During test-time intervention the predicted concepts can be modified by a supervisor to their ground truth values, leading to "adjusted" concepts prediction. We represent the predicted concepts as è py (c(xx) and the modified concepts as č. We consider test-time interventions as an important aspect of explainable models in safety-critical applications. We hypothesize that model-supervisor interaction must lead to the development of symbiotic relationship between the model and the expert. Here, the expert learns about the potential causation between a concept and its corresponding label, and the model learns true concept values from the expert. We attempt to shine a light on these test-time interventions by simulating realistic scenarios by introducing human uncertainty.
3.2 Coop-CBM
While CBMs provide concept expla- nations behind prediction, it has been observed that this can come at expense of lower model accuracy compared to black-box standard mod- els (33] In this work, we propose coneept-based architecture, coop- CBM to improve the performance of CBMs downstream classification tasks.
fri
gifiii
Encoder
y
hix)
Coop-CBM model that consists of an encoder, concept learner f. auxiliary label learner h, and task label learning g. The encoder transforms input data into a feature representation, which is used by f to predict high-level con- cepts and h to predict supplemental auxiliary label, and finally g predicts the final task label conditioned on the con- cepts only.
Figure 1:
Motivation The different training paradigms in CBMs introduced by Koh et al. [27] give rise to differ- ences in their concept representations, ()(c)z). Koh al. (27] reports that joint CBMs have the highest task accu- racy among the different CBM train- ing procedures albeit still lower than standard models. Intuitively, this suggests that joint CBMs, which train both concept predictor and task predictor simultaneously, are able to encode the informa- tion about the task label y into concept labels better than sequential and independent CBMs. In the case of joint CBMs, backpropagation of task loss through the concept predictor aids the overall model in giving more accurate predictions. In this work, we aim to leverage such "soft" information

---

about the task to improve accuracy. Coop-CBi CBM aims to leverage soft label information in concept predictors better align the concept predictions to the corresponding label.
Model Coop-CBM introduces multi-task setting before the final prediction. Along with predicting the concepts c, we introduce the prediction of task labels in the concept predictor. This allows the model to learn relevant signals and inductive biases of downstream tasks in the concept learning phase. In essence, we now have model fx-o that predicts concepts from input x and new model x that predicts label y from input x. This enables the model to learn relevant knowledge about the task that could be absent in the bottleneck concepts c. Although this setting makes the model interpretable, since corresponding concepts to a label can be obtained, one loses the causal - - y property. Additionally, it does not allow test-time interventions, which is a key application of concept- based models that facilitates human-model interactions. Therefore, to maintain the original properties of CBM, Coop-CBM uses a label predictor g which takes the predicted concepts c from fx-c as input and gives the final label y as output. Hence to avoid confusion, we call the label prediction from h immediate label, v/. and the final task label, y. It must be noted that f and l share all but the last linear layer. Therefore the concept predictor f and g parameters 0, trained in the following manner:
i = =E Jargmax (log p(c, v/ >x; 0)] = =E Jargmax[log py(c|a;; 0) + log ph (a(y' (v'aa; 0)]
(1)
D
D
b E argmax llog p(ule; +)]]
(2)
Therefore coop- trained using a linear combination of three different CE losses: Le as concept loss, Ly as immediate label loss and Ly as task prediction loss.
arg arg mngmin[Cc((()..ccm + Cy (h(z),)) + C,(o(f(z),
(3)
In summary, we argue that the introduction of an immediate label introduces the concept predictor to learn meaningful information about the task while still being interpretable. The hx-Y" model intuitively acts as a regularizer for meaningful concept prediction.
Mutual information perspective We hypothesize that by using coop-CBM. the concept predictor acquires better knowledge about y. In particular, this can be beneficial when fine-grained concept annotations not available. We, therefore, suspect that the mutual information (MI) between the input image, concept representations, and the label becomes richer and more expressive as compared to CBM (27]. One way quantify this is by visualising the MI planes throughout the training, similar to Zarlenga et al. [60].
3.3 Concept Orthogonal Loss
Following the current CBM literature, we cross-entropy loss to train each of the models, fx fx- hx-yy' and JC-y in coop-CBM. In model f, each concept is learned via independent and separate classifiers. Given their binary representation, it is intuitive to improve the embedding space of concepts by increasing separability. To do so, we introduce the concept orthogonal loss (COL). By incorporating COL into the training process, we aim to enhance the overall separability of the concept embeddings, leading to improved performance and interpretability of the coop-CBM model.
Motivation Due to the variations in training strategies employed in different CBM models, the resulting concept representations can exhibit varying levels of accuracy. The concept accuracy refers how effectively the learned concept representations align with the ground truth or human-defined concepts. Coop-CBA was concerned with the predictive performance of the task, but here we focus on the concept label accuracy. Higher concept label accuracy signifies improved interpretability. As observed by Koh et al. (27], the concept accuracy of joint CBMs models is lower than other variants because the concepts learned are not completely independent of each other, also called leakage by (33. Hence, increasing the inter-concept distance and intra-concept clustering throughout the concept vector for the entire dataset can allow the model learn beyond co-dependent concept representations.

---

COL In addition to CE loss for learning concepts, we introduce novel concept orthogonal loss by conditioning orthogonality constraints on concept feature space. The disadvantage of CE loss is that it does not set specific distance separation between different concept representations in the feature space. Consider the CE loss for each concept prediction:
eN
CCe(e,))
=
Σ C1
c,oo(C; - (1 - c; c:)log(1 &))
(4)
Using this traditional CE loss for each concept, in Equation difference between the predicted probability distribution and the true probability distribution of the binary concepts. The model, fe, attempts learn probability distribution for when a concept is active or inactive respectively. CE does not explicitly enforce separation between concepts.
4 we are essentially minimizing the
With concept orthogonal loss (COL), we enforce the separation in the latent representation of the concepts. Our aim with COL is to group similar features together while ensuring that features belonging to different concept classes do not overlap with each other. COL Lcoe enforces inter- concept orthogonality and intra-concept clustering. We define inter-concept separation and intra- concept similarity as d1 and d2 respectively. We enforce orthogonality constrain via cosine similarity. We define the LCoL loss in the shared last layer, of coop-CBM before concept and auxiliary label predictions, We enforce COL constraints within each batch, B.
d1 = Σ
90 ||/11|
d2 = Σ
4 qi
(5)
11:1l
||
J. =/=c)
J.EB.
4775"
aEA
where ||-I| denotes the Frobenius norm.
Using the cosine distances, dj and d2, we simultaneously aim to increase the distance between different latent concept representations and decrease the distance between representations from the same concept. We can introduce hyperparameter, À to accordingly give weightage to either d1 or d2- The similarity loss d1 between the feature representation two samples corresponding to the same concept aims to push the dN towards which means that the feature representations of same concept should be as similar as possible. As for the dissimilarity loss, the goal is to push the loss towards O, which enforces that the feature representations of different class samples should be as dissimilar as possible. Therefore we consider the absolute value of d2.
Lcor (1 - di) + X(22|
(6)
It is important to note that CE loss is applied to each concept binary classification task, which measures the difference between the predicted class probabilities and the true labels. The introduction of COL encourages the network to learn features that are both discriminative and non-redundant among concepts at an intermediary network level. By combining the COL and CE losses, the network trained to learn discriminative that separate each concept and useful features for classifying when concept active. A benefit of COL is it can be universally any concept- model to encourage orthogonality between different concepts.
In this section, we introduce two auxiliary losses, one to improve the task accuracy using multi-task setting and the other to improve concept representation in latent space, leading improved concept accuracy. The final loss a linear combination (a, 3, 7 are hyperparameters for weighting in Equation ? of concept and task losses along with immediate and concept orthogonal losses.
arg min mi + 3Cy (h(z).)) Ly(g(e) v) (Co
(7)
3.4 Interventions
Koh et al. 27 demonstrated the potential of CBMs for facilitating human- model interaction and improving task performance during inference. But it can be time-consuming and costly to have domain experts go over each concept, hence some of the recent and concurrent works proposed to use uncertainty as a metric to select interventions.
propose a lightweight approach that strategizes the supervisor-model interaction. Our method intuitive and considers three aspects of intervention:
5

---

1. Uncertainty of concept prediction CUS represents the confidence of the model to predict latent concepts, fx-C.
2. Supervisor confidence for concept correction SCS represents the reliance the supervisor to intervene and subsequently correct the concepts accurately.
3. Importance of concept for label prediction CWS denotes the significance of each concept for the subsequent downstream task.
Chauhan et al. 10] propose to optimize interventions over small validation set using CUS. In comparison, consider access the validation unrealistic. Shin et al. 49), Sheth et al. 48 evaluated interventions more comprchensively and studies the behavior of CBMs during inference by selecting CUS and CWS metries. We additionally take into account supervisor's confidence in domain knowledge, SCS and their expertise in correcting the concepts. Concurrent work [12] also looked at human uncertainty for CBMs in depth. Unlike previous works that evaluate test-time interventions on the test splits of respective datasets, we also analyze test-time interventions in OOD setting in the Appendix
Experiments
For evaluation consider several im- age classification benchmark datasets. Our work performs in-depth empirical anal- ysis of the effectiveness of concept-basse based models in the presence of different distribu- tional shifts simulating real-world scenarios where data is diverse.
Model type
CUB
AwA2
TIL
Standard [No concepts]
82.3
96.2
51.1
#0.2
=0.1
=0.9
Independent CBM 27
76.0
94.9
47.4
+0.1
-0.3
=1.0
Sequential CBM
27
76.3
94.6
47.9
10.2
10.2
10.9
Joint CBM 27
80.1
95.4
49.6
Baselines We consider the models pro- posed by Koh et al. 27 as our baseline. Additionally, we compare our performance with recent concept-bast based models that are built on CBM (60, 20). Due to biases intro- duced during automatic concept acquisition as mentioned by the authors of Yuksekgonul [58], we consider it be an unfair baseline to compare generalization proper- ties. They also vary in number of con- cepts considered which can also damage the performance and are limited by either the presence of concept bank application (CLIP will fail to generate concepts for TIL dataset), making an unfair comparison.
10.1
10.1
20.7
CEM 600
82.5
96.2
51.3
=1.3
CBM-AR 220
81.6
95.9
49.5
=1.
Coop-CBM (ours)
83.6
96.6
53.4
=0.3
#0.1
=0.8
+ COL
84.1
97.0
54.2
=0.2
=0.1
+0.9
Table 1: Model accuracy on CUB, AwA2 and TIL datasets
Datasets We Caltech-UCSI Birds-200-2 200-2011 (CUB) [55) dataset for the task of bird identifica- tion. Every dataset image contains 312 binary (eg: beak color, wing color) concepts. We additionally Animals with Attributes 2 (AwA2) (57] dataset for the task of animal classification. The dataset contains 85 binary concepts. We use all of the subsets of the Tumor-Infiltrating I Lymphocytes (TII)) dataset for cancer cell classification.
42
5
Results and Analysis
The primary metric used for the downstream classification task is accuracy. We use the same metric to evaluate the effectiveness of the intervention. We first evaluate the different model performances on the test data split of respective datasets and report the task accuracy 9C-sY for coop-CBM model variants.
Coop- CBM improves task accuracy The evaluation of the performance of model is based on the final prediction accuracy. Table we compare the performance of coop-CBM against other bascline models. We first observe that CBMs experienced significant drop in performance compared

---

to the standard model that did not use concepts. Our proposed model, coop-CBN, with immediate label prediction achieves state of art accuracy and statistically significant results on every dataset. We have observed a significant improvement in the performance of the CUB 1.8% increase from the standard model) and TIL.++3.11 1% increase from the standard model) datasets. This finding important as it suggests that machine learning models can be designed to overcome the high accuracy vs interpretability tradeoff. Our performance can be further boosted by introducing orthogonality among different concepts. It must be noted that CUB is fairly densely annotated dataset, which might not always be realistic, hence we also benchmark our model by training on fraction concept sets. We also observe similar trend in results in concept-scaree settings (see Appendix D.2). This suggests that our method is robust to concept selection, which can beneficial scenarios where the number of available concepts is limited or expensive to obtain.
COL improves concept accuracy Previ- ously in Table [ we observed that adding concept orthogonal loss to coop-CBM im- proved its downstream accuracy. in Table 2 we study the impact of adding COL to baseline concept models. Our experiments show that adding COL improves the con- cept accuracy by a significant margin, espe- cially in joint CBM, CEM, and CBM-AR settings. known pitfall of CBMs is, con- cept leakage 33 could be potentially pre- vented by increasing the separation between their concept representations. By maximiz- ing the inner product between the concept can
Model type
w/o COL
COL
Independent CBM
27
96.6
97.2
40.0
=0.1
Sequential CBM
96.6
97.2
20.0
=0.1
Joint CBM 27
93.2
96.4
20.1
+0.2
CEM 660
94.8
97.0
20.2
#0.1
CBM-AR 20
94.2
96.8
=0.1
#0.1
Coop-CBM (ours)
93.9
97.3
=0.2
:0.2
Table 2: Concept prediction accuracy for each model before and after adding COL for CUB dataset
embeddings of different concepts, ensure that each concept is represented in a separate and distinet direction in the embed- ding space. This helps in preventing the models from relying on irrelevant concepts. Further to intuitively understand the differences in the concept representation of our model, compute the histogram for the predicted concept logits. From 3igure]]wa [we see that Coop-CBM+COL. minimizes the concept loss better (with help from the auxiliary loss which aids representation learning), which results in clearer separation of logits.
Clipping concept values avoid concept leakage Further, we employ clipping of concept prediction proposed by Mahin- pei et al. (33) to further mitigate information leakage in throught two experiments. For the first ex- periment, we trained the model by clipping the predicted con- cept values to "hard" labels. Sec- ond, we trained the model as we have described earlier in the pa- per (using soft labels) and eval- uated the test set by clipping "hard" labels. From the above ex- periments, we conclude that the model is able to learn good representation of the concepts without necessarily leaking information.
CUB
TIL
Model
Std
Expl
Exp2
Std
Expl
Exp2
Coop-CBM
83.6
82.1
83.0
53.4
52.6
52.8
10.2
20.5
40.8
10.9
+COL
84.1
83.2
84.0
54.2
53.5
54.1
10.2
10.2
20.6
40.9
20.9
40.8
Table 3: Testing for information leakage in our proposed model. Std standard conditions when joint probabilities are learned to predict the final task, no clipping. Expl During training, clipped the predicted concept values to "hard" labels. Exp2 During the evaluation, we clipped the predicted concept values to "hard" labels.
Accounting for Human uncertainty for interventions As discussed earlier, higher concept accuracy also improves the test-time interventions as seen in Figure While other works used concept weights and uncertainty as metries to select the interventions, our work introduces a more realistic setting by introducing human uncertainty additionally. The previous works do not account for human error or certainty. Albeit human uncertainty difficult to quantify since it is often subjective, use the concept visibility data in the CUB dataset to quantify the confidence score, SCS. The
7

---

CUR
CIR
TI
C1R
97
03
23
50
100 Single Ince ventons
250
200
50
.50
10C Single Interventions
Incorrect interventt
Single interventions
Figure 2: L-R:11 Accuracy intervention graph using joint CBM while including supervisor uncertainty. 2)Accuracy vs intervention graph in presence incorrect interventions by the supervisor using joint CBM. 3) Comparing different model's random interventions the TIL dataset. 4)
Comparing different model's random interventions on the CUB dataset.
annotators rank visibility from presents accuracy curve with increasing number of interventions on the vanilla CBM model during test-time for proper and fair comparison to [49]. It suggests that considering SCS leads to the most meaningful interventions. On intervention comparison between different concept-based models, we observe that coop-CBM leads to significant interventions that improve downstream performance, especially in TIL dataset where both image descriptive and non-descriptive concepts are present.
being 'not visible' and being 'definitely visible'. Figure
In summary, the coop- CBM model and the addition of the concept orthogonal loss help improve both task and concept accuracy without sacrificing interpretability. which was common tradeoff in previous methods. This result demonstrates the potential for concept-base models be more effec- tive in human-AT interactions, especially in domains where expert intervention and interpretability are critical such as healthcare.
5.1 Performance under distribution shifts
5.1.1 Background spurious correlations
Shortcut-based biases 17 exist in many datasets, where deep learning easily learn spurious features. the presence of shortcuts, the model can learn to use spurious features to approximate the true distribution of the labels, opposed learning core features. can be of partic- ular interest evaluate the performance of concept-based models in the presence of shortcuts. Furthermore, using explain- able concepts to facilitate human-modee interaction could help reduce the impact of these biases.
CUB
CelebA
Model
In
Out
In
Out
Standard
86.9
27.7
96.5
76.3
20.3
12.8
#0.5
#1.8
Independent CBM 27
81.1
30.9
93.8
76.2
+0.5
22.3
20.6
#1.9
Sequential CBM 27
81.2
30.8
93.8
76.1
10.5
+22
20.6
#1.9
Joint CBM
83.6
32.9
94.7
80.1
10.4
13.
20.5
#1.
CEM 660
84.0
34.0
95.2
81.0
10.4
$2.9
20.4
11.7
CBM-AR 20
82.6
33.6
95.9
79.7
10.6
12.5
20.5
12.0
The shorteut we consider here is spuri- ous correlation to the background color in CUB dataset. Loosely following the experimental setup of (5). we correlate the background of species of bird its corresponding label. For the dataset, segment the bird images and add colored background to all of the images.
Coop- (oop-CBM(ours)
84.9
35.4
95.4
81.3
10.3
12.5
10.6
11.9
+ COL
85.8
36.2
95.9
82.0
10.4
12.7
2007
22.7
Table 4: Accuracy of different models under distributional shift background spurious correlation.
Each class here is correlated a randomly generated color background with probability of 80% for the train set. The in-domain test set contains images with similar color background probability as the train set while the correlation in the out-domain test set reduced to 30%. We also consider the hair color-based shortcut induced in Large-scale CelebFaces Attributes (CelebA) dataset where we focus on gender classification between males and females. We aggregate and then construct a modified version of CelebA that learns the shorteut of blonde hair color with women similar to [3].

---

In Table 4 we report the in on-domain and out-domain accuracies of our baseline models and proposed coop- for CUB and CelebA datasets. Our results show the robustness of coop- and COL to background spurious correlations achieving state-of-art results among the concept-based models. We observe an interesting trend for the CUB dataset. The concept-basee models may have lower accuracy standard model in the in-domain setting but in the out-domain setting, all of the baseline models including ours have higher aceuracy than the standard model. While we do not observe similar definitive trend in CelebA dataset, it is evident that most of the models, CEM, joint CBM and coop-CBM are superior when the test data contains more images men with blonde hair. This experiment suggests that the concept- based models able to generalize better to unseen data outside the training distribution, which can be attributed their ability to learn more invariant features through the explicit conditioning and learning of concepts. This is promising result it indicates that concept-based models, particularly coop-CBM may be more robust to extreme distribution shifts.
In Appendix E we evaluate the influence of interventions in out-domain settings for the biased CUB dataset. Realistically, the user may fine-tune the model distributional shift after deployment to improve the predictive performance, but in the circumstances where the labeled shifted dataset is absent, we suspect that interventions could greatly help if they are cheaper to obtain. In such cases, we especially argue that SCS score benefits the intervention selection.
5.1.2 Image corruptions
Distributional shift can occur due to various factors such changes in the data collection process, changes in the environmental conditions under which the data is collected,
changes in the underlying population that the data represents. The Corruptions dataset [22 35 a collection of image corruptions designed to evaluate the robustness of computer vision models. Some of the corruptions are realistic OOD settings such as snow, while others could be less likely in nature such impulse noise. We introduce 7 such corruptions (Gaussian Noise, Blur, Zoom Blur, Snow, Fog, Brightness and Contrast) onto CUB, AwA2 and TIL datasets. We report detailed results on CUB and average accuracy for AwA and TIL datasets in Table
CUB
Model
AwA2
TIL
1
2
3
5
6
7
Avg
Standard
65.2
62.0
56.7
60.1
74.1
72.0
53.4
63.3
79.3
38.4
10.3
60.7
20.5
+0.2
1.0.8
20.6
B0.4
Independent CBM 27
61.4
62.1
57.1
59.7
73.6
69.8
52.9
62.3
78.4
36.6
10.6
2044
10.3
1:0.7
20.5
10.2
Sequential CBM M7
60.3
61.9
56.5
58.5
72.7
71.2
52.0
61.8
78.2
36.3
10.35
10.2
20.6
10.4
10.8
20.3
1.0.7
Joint CBM 27
63.1
64.5
57.4
60.6
73.8
72.3
51.8
63.4
79.1
37.1
+0.4
20.8
20.3
10.7
2-0.6
20.2
20.6
CRM 60
66.1
61.4
57.3
61.0
74.2
71.6
53.4
63.6
79.7
38.5
CBM-AR 20
64.8
61.7
57.2
59.4
73.3
70.4
52.9
62.8
79.6
36.8
+0.6
+0.4
20.S
+0.3
+0.7
#50.5
+0.2
Coop-CBM (ours)
67.2
63.5
59.0 O
60.9
75.4
73.2
53.3
64.6
80.9
40.6
+0.3
+000G
+ COL.
67.8
63.9
58.7
61.5
75.8
73.4
53.3
64.9
81.5
40.9
220.4
20.8
20.3
20.7
20.6
#0.2
20.6
Table 5: Comparison of concept-based models on image corruptions on CUB, AwA2 and TIL datasets.
Based on our evaluation of distributional shifts in Table 5 we found that incorporating an auxiliary loss in the form of multi-task setting can help CBMs achieve competitive downstream accuracy overall. We notice that the standard black box models although do perform better in the presence of "contrast" corruption in CUB dataset. Although coop-CBM outperforms other explanation models. In general, it is interesting to note that coop-CBM has better generalization performance in the presence of spurious correlations than in the presence of corruption. This may be because spurious correlations are introduced as a shortcut within the training data, and concept-based models are designed to learn invariant features that are robust to such shortcuts. Regardless, coop-CBM's superior generalization property across different corruptions suggests that the model is able to effectively filter out irrelevant information/noisee in the data.
5.1.3 Noise concept correlation

---

In Section 5.1.1 1 and Section 5.1.2, we conducted experiments concerned with distribution shifts image space, in this section, we introduce the eval- uation of CBMs by simulating distribution shifts in the concept space. To investigate the potential risks of spurious correlations in concept models, we introduced Gaussian noise to the binary con- cepts. By altering the standard deviation (a) the Gaussian noise, effectively correlated the shorteut (here noise level) with the image through the concept. To simulate more realistic setting, instead of adding distinct noise a for each class species, aggregate random groups of species and add same a to them. In our experiment for the CUB dataset, add 10 different levels of noise (simulated by σ) to groups of 20 species labels (200 total classes). For AwA2, we create groups of 10 classes. This approach allowed us simulate the possibility of introducing unintended correlations between the concepts and the images. By study- ing the effects of these correlations on the performance of the concept models, we gain insights into the robustness and reliability of the models in handling contaminated concepts. We observe that by introducing a separation between different concepts through COL, our model performs significantly better than the rest of the baselines.
Model type
CUB
AwA2
Independent CBM 27
69.7
80.1
10.6
=0.4
Sequential CBM 22
69.6
80.3
+0.3
=022
Joint CBM 27
71.0
81.3
+0.4
=0.4
CEM 60
71.2
81.9
20.6
=0.3
CBM-AR 20
71.1
81.5
10.4
-0.3
Coop-CBM (ours)
71.9
82.5
10.6
-0.2
+ COL.
72.7
83 2
10.4
:0..2
Table 6: Accuracy of different models under distributional shift-noise concept correlation
Future work and Limitations
In this work, we introduced coop-CBM, novel concept-driven method to balance Al model in- terpretability and accuracy. We utilized the Concept Orthogonal Loss (COL) to improve concept learning and applied coop-CBM to various datasets, achieving better generalization, robustness to trade-offs.
spurious correlations, and improved eecuracy-interpretablll
However, our approach has limitations. It relies on labeled concept vectors, which can be challenging in domains with limited annotations and face biases in concept annotation methods. potential future work could be to extend it to methods that do not assume concept label [58 38 Further, we used accuracy as a metric evaluate concept leakage, in the future, would be interesting to explore other metrics beyond the accuracy of concept prediction. A future extension of COL could be to evaluate which concepts should be explicitly orthogonalized. We recognize that our model has a few hyperparameters to be optimized. Furthermore, our model assumes that learned concepts align closely with human notions, but this alignment isn't 't always perfect, affecting comprehensibility. Future research could improve the accuracy of concept-based models by providing meaningful explanations and incorporating additional evaluation metrics. Another potential direction could be assess the mutual information and therefore establish theoretical grounding describe the superior performance of coop-CBM.
Discussion and Conclusion
In this work we proposed two significant contributions to the paradigm of concept-based models. First, we introduced a multi-task model that predicts an intermediary task label along with concept prediction. This is particularly helpful when dense and relevant concept annotation is absent, such as TIL dataset. Second, we introduced orthogonality constrain in the concept representation space during training via concept orthogonal loss. This loss increases inter-concepi separation and decreases intra- intra-concept distance. For both of our proposed methods, we perform extensive experiments on diverse datasets and different distributional shifts. We observe that the bottleneck layer before the final prediction enables concept-base based models to exhibit robustness to spurious correlations in background. Coop-CBM along with COL achieves state-of-art performance for both task accuracy and concept accuracy. Our work indicates that coop-CBM and COL have a strong ability to adapt and generalize well across diverse datasets and real-world scenarios.
10

---

8 Acknowledgements
We would like to thank Vincent Michalski and the reviewers for engaging in discussions on the earlier version of the paper. The authors would like to thank Google, CIFAR (Canadian Institute for Advanced Research) and NSERC (Natural Sciences and Engineering Research Council of Canada) for supporting and funding the research and Digital Research Alliance of Canada for the compute support.
References
[1] Julius Adebayo, Justin Gilmer, Ian J. Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. ArXiv, (bb//180000337,, 2018.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances neural information processing systems, 31, 2018.
[3] Mohammed Adnan, Yani loannou, Chuan- Chuan-Yung Tsai, Angus Galloway, HR Tizhoosh, and Graham W Taylor. Monitoring shortcut learning using mutual information. arXiv preprint WMXiv:226.110034 2022.
[4] David Alvarez-Melis Self-explaining neural networks. 2018.
[5] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk mini- mization. arXiv preprint DXXiv:9907.0280) 2019.
[6] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end iitention-based large vocabulary speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4945 4949. IEEE, 2016.
[7] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus Klaus-Robert Müller, and Woj- ciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, 2016.
[8] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learnrrs. Advances in neural information processing systems, 33:18777-190 2020.
[9] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. arXiv preprint arXiv:2007.07375, 2020.
[10] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430, 2022.
[11] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nat. Mach. Intell., 2:772-782, 2020.
[12] Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Llia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference AI, Ethics, and Society, pages 889,8899111 2023.
[13] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine fine-grained debugging and analysis. Advances in Neural Information Processing Systems,
2022.
[14] Devleena Das, Sonia Chernova, and Been Kim. State2explanation: Concept-based explanations benefit agent learning and user understanding. arXiv preprint 2rXiv:2309.12482. 2023.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint 2020.
11

---

[16] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2015.
wttp://..r./aaa//5...0222 0214
[17] Robert Geirhos, Born-Herrik. Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11) (11):656663, 2020.
[18] Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L Beam, Irene Y Chen, and Rajesh Ranganath. review of challenges and opportunities machine learning for health. AMIA Summits on Translational Science Proceedings, 2020:191, 2020.
[19] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) 06), volume pages 1735-1742. IEEE, 2006.
[20] Marton Havasi, Sonali Parbhoo, and Finale Velez. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems, 2022.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[22] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 12261, 2019.
[23] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
[24] Mark Ibrahim, Melissa Louie, Ceena Modarres, and John William Paisley. Global explanations of neural networks: Mapping the landscape of predictions. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, 2019.
[25] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[26] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.0155: 2023.
[27] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. ArXiv, 04./2007.04112 2020.
[28] Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision, pages 365-372. IEEE, 2009.
[29] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition, pages 951-958. IEEE, 2009.
[30] Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in neural information processing systems, 19, 2006.
[31] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August 15(2018):11, 2018
[32] Luca Longo, Randy Goebel, Freddy Lecue, Peter Kieseberg, and Andreas Holzinger. Ex- plainable artificial intelligence: Concepts, applications, research challenges and visions. In Machine Learning and Knowledge Extraction: 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2020, Dublin, Ireland, August 25-28. 2020, Proceedings, pages 1-16. Springer, 2020.
12

---

[33] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. ArXiv, abs/2106.13314, 2021.
[34] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? ArXiv, ftsss2105.04255 2021.
[35] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv: 190::0740777444 2019.
[36] Grégoire Montavon, Wojeiech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. ArXiv, abs/1706.07979 2018
[37] Yasunobu Nohara, Koutarou Matsumoto, Hidehisa Soejima, and Naoki Nakashima. Explanation of machine learning models using improved shapley additive explanation. In Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics, pages 546-546, 2019.
[38] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui- Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06 2023.
[39] Ce Qi and Fei Su. Contrastive-centes loss for deep neural networks. In 2017 IEEE international conference on image processing (ICIP), pages 2851-2855. IEEE, 2017.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[41] Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12333-12344 2021.
[42] J. Saltz, Rajarsi R. Gupta, Le Hou, Tahsin M. Kurç, Pankaj Kumar Singh, Vu Nguyen, Dimitris Samaras, Kenneth R Shroyer, Tianhao Zhao, Rebecca C. Batiste, John S. Van Arnam, Ilya Shmulevich, Arvind U. K. Rao, Alexander J. Lazar, Ashish Sharma, and Vésteinn Thorsson. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell reports, 23 1:181-193.e7, 2018
[43] Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286-10295. 2022.
[44] Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsuper- vised concepts. IEEE Access, 10:41758-41765, 2022.
[45] Andrew M Saxe. James McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 2013.
[46] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
[47] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128:336-359, 2017.
[48] Ivaxi Sheth, Aamer Abdul Rahman, Laya Rañee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou. Learning from uncertain concepts via test time interventions. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurlPS 2022, 2022.
[49] Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302. 2023.
13

---

[50] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 2818 2-288 2016.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arX11:2302.13397, 2023.
[52] Asher Trockman and Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXi:2104.07167, 2021.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[54) Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning, pages 3570-3578. PMLR, 2017.
[55] Peter Welinder, Steve Branson, Takeshi Mita. Catherine Wah, Florian Schroff, Serge J. Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[56] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning ap- proach for deep face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pages 499-515. Springer, 2016.
[57] Yongqin Xian, Christoph Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Putlern Analysis and Machine Intelligence, 41:2251-2265, 2019.
[58] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv: 15480, 2022.
[59] Éloi Zablocki, Hédi Ben- Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision-based autonomous driving systems: Review and challenges. International Journal of Computer Vision, 130(10):2425 240-222222,2222. 2022.
[60] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprini arXiv:2209.0005: 2022.
[61] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv preprint arXiv:2301 1036,
[62] Yaohui Zhu, Weiqing Min, and Shuqiang Jiang. Attribute-guice feature learning for few-shot image recognition. IEEE Transactions Multimedia, 23:1200-1209 2020.
14

---

A More Related Works
Exlainability Post-hoc explanations aim to provide insight into why a particular prediction or decision was made by the model. These may be in the form of heatmaps, rule sets, or feature importance scores. Global explanations (24 aim to learn the overall generic features in order to explain black-box models by the use of explanators. These explanators are often simple Machine Learning (ML) models. Local explanation methods [1 exhibit explainability by exploring the intrinsic workings of the neural network. This can be executed by propagating layer-wise feature relevance 77. Selvaraju et al. [47 propose a gradient-based saliency mapping technique that perturbs inputs by injecting noise. Deconstructing the nonlinear model to simpler sub-functions [36] was another proposed method to interpret model's predictions. Nohara et al. [37] propose a torreerrce approach to feature importance estimation. Popularly, saliency heatmaps for feature importance visualization have been used [2]. Post hoc explanations although do not address the fundamental issue of model transparency, as they generated externally to the model and may reflect the true reasoning of the model's 's internal mechanisms which can be addressed by concept- models like ours.
Orthogonality Orthogonality in neural networks has been extensively studied in the literature, with various approaches proposed to enhance model performance and interpretability. X and Y have explored orthogonal regularization techniques, which impose orthogonality constraints on weight matrices or feature representations during training. These methods have been shown to improve generalization and reduce overfitting. Saxe et al. [45] have introduced orthogonal initialization meth- ods, which initialize weight matrices using orthogonal transformations. This initialization strategy has been found to aid training convergence and stabilize the learning dynamics of deep networks. Trockman and Kolter [52] have proposed techniques to enforce orthogonality constraints specifically in convolutional filters convolutional neural networks (CNNs). By imposing orthogonality on the filters, these methods enhance the representational power and robustness of CNNs. In work, introduce orthogonality to concept feature space.
B Experiments
The details regarding datasets and hyperparameters are mentioned below. Our codebase is available at
cotp//b//nncccca and is built upon from open source repos [27 41
B.1
Dataset Details
CUB The CUB-200-2011 dataset is a collection of 11788 images that are used for fine-grained visual categorization. There are 312 concept attributes that are binarised following the Koh et al. [27] work. While most existing studies use subset of these concepts, we have chosen to use the entire concept bank in our models and baselines, addressing the fairness issue of subgrouping concepts as highlighted by [49] The primary task classify 200 different species of birds. We also utilize the meta-data of this dataset to obtain human uncertainty.
TIL 42 dataset contains Tumor- Infiltrating Lymphocytes Maps from TCGA HE Whole Slide Pathology Images. The dataset contains tumor maps from the most common cancer tumor types. We all 13 subsets of the TCGA dataset, therefore constituting 13 cancer types. Although the popular task for such a dataset is necrosis classification, we modify the task to be a classification for different (here 13) tumor types. The advantage of medical images is that their meta-data is readily available from diagnosis. The metadata includes information such as the origin of the tumor, age, gender, and size of tumor cells, which are converted to concepts. We follow We wnd up with 185 binary concepts following the pre-processing.
(48) for the dataset pre-processing.
AwA2 Animals with Attributes dataset contains over 37,000 images of 50 different animal species, each labeled with 85 distinctive attributes. These attributes can include various characteristies such as color, shape, or behavior, providing rich source of information for concept bank.
15

---

B.2 Experimental setup
For CUB|55] dataset, we trained using 128 batch size with SGD optimizer with 0.9 momentum and learning rate of 10-2. The feature extractor was Inception V3 as concept encoder model.
For AwA2 57 dataset, trained using 128 batch size with Adam optimizer with 0.9 momentum and learning rate of 5 x 10-3 The feature extractor was VIT as concept encoder model.
For the medical dataset, we create the task of classification divided on the basis of cancer types for TIL dataset [2) We generate a concept attributes from the meta-data. We use a traditional 70%-10%-20 random split for training, validation, and testing datasets. Additionally,
trained
using 64 batch size with SGD optimizer with 0.9 momentum and learning rate of 10 2 The feature extractor was Inception [50) as concept encoder model.
For m-CelebA 31 dataset, we train using 64 batch size with Adam optimizer with 0.9 momentum and learning rate of x 10-3 for 500 epochs. The feature extractor was Inception [50 as a concept encoder model.
Across all of the models for tasks, we use weight decay of factor of x 10-5 and scale the learning rate by factor of 0.1 if improvement has been training. We also train using an early stopping mechanism i.c. if the validation loss does not improve for 200 epochs, we stop training.
validation loss for the last 15 epochs during
In this paper, we introduced two auxiliary losses, one to improve the task accuracy using multi-task settings and the other improve concept representation in latent space, leading improved concept accuracy. The final loss a linear combination (a, _3, y are hyperparameters for weighting in Equation [7) of concept and task losses along with immediate and concept orthogonal losses.
arg ())
(, +Ccor(a))
For the hyper-parameters of Equations
and / values to 0.01 for all of the experiments.
Model
Task
Concept
8 = 0.01
84.1
97.3
8 = q 1.0
83.0
96.8
= 3 = =01, 0.01
83.4
97.4
3 = 0.01. 0.1
83.2
97.4
8 = 0.1.7 0.1
83.4
97.5
0.1, = 0.01
84.0
97.3
== 0.01, - 0.01
84.2
97.0
Table 7: Different weightage coop-cbm with COL CUB dataset
CEM hyperparameters We would like to point out that we used the same concept a weightage hyperparameter for each of the model. literature, for all [27 60 20) of the methods used the same concept weightage and we follow the same convention. Looking at our selected hyperparameter, the most divergent value could not find an ablation study around this hyperparameter in the original paper, we continued to use the same value as rest of the models. We acknowledge that our results might be skewed due this reason.
CEM (0). The original CEM paper selected the a 5. Since we
B.3 Resources used
Our codebase was built upon the open codebase of 27 We trained on Linux-based clusters mainly on V100 GPUs and partially on A100 GPU. The following table includes compute timing for each epoch for every bascline.
16

---

Model type
CUB
TIL
AwA2
Standard
57s
33s
272s
CBM
68s
39s
286s
CEM
78s
46s
297s
CBM-AR
87s
50s
313s
Coop-CBM (ours)
61s
41s
289s
Table 8: Compute timings of baseline models on different datasets on a V100 GPU.
C Further experiments on COL
C.1
Disentanglement
Disentangled features allow for more intuitive understanding of the underlying factors that influence the concepts. Several research efforts have explored the benefits and applications of disentangled representations [23]. One of the notable characteristics of Concept Orthogonal Loss (COL) is its ability to induce disentanglement in the concept space. By incorporating COL into the training process, the model learns separate and represent concepts in more distinct and independent manner. This disentanglement is achieved by enforcing an orthogonal relationship among different concept representations. To evaluate disentanglement, we use the Oracle Impurity Score as proposed by Zarlenga et al. [61]. The metric essentially aims to detect for impurities in soft representations of concepts. We use this metric to compare against Joint-CBM. The OIS score for Joint-CB.M on the CUB dataset was 0.19 while on coop-CBM with COL 0.14 which shows better disentanglemen in concept learning for Coop-CBM COL model showing the benefit of using COL on top of any model ad-hoc.
COL encourages the model assign orthogonal directions different concepts, thereby reducing the overlap and correlation between them. As a result, each concept becomes more independent and captures specific aspect or attribute of the input data. This disentanglement in the concept space enables better interpretability and facilitates a clearer understanding of how different concepts contribute the model's decision-making process.
Through disentanglement, COL enhances the separability and discriminative power of the learned concept representations. It allows the model to focus relevant and informative aspects of the data while minimizing the influence irrelevant or redundant features. This disentanglement not only improves the interpretability of the model but also contributes to its overall performance by reducing concept interference and enhancing the model's 's ability to generalize to new and unseen data.
By promoting disentanglement in the concept space, COL provides valuable tool for understanding and analyzing the inner workings of concept-based models. It opens up opportunities for further research and exploration into how disentangled concept representations be leveraged for various tasks, including transfer learning, domain adaptation, and model debugging.
C.2
When is concept orthogonality relevant
One obvious question and could be an interesting future work could be to devise what explicit concepts must be orthogonal. Our work assumes that every concept must be orthogonal but potentially there could be an application where it could be beneficial to include partial orthogonality. Devising an optimal point for when to use COL could be great future work building our work. We attempt to provide justification using empirical analysis and some intuition in this section.
Duplicating concepts We consider scenario where input concepts are intentionally duplicated to create a high degree of concept correlation. In our experiment from Table 9 we duplicated 10%. 25%, 50% and 100% of concepts and added them to the original concept bank. This is a worst-case representation "similar concepts". From the table, we see that the duplication of concepts does not impact the concept or the task accuracy. Additionally, this experiment contributes to the broader understanding of how COL performs in various scenarios. We observe that the performance not significantly impacted for concept duplication if we add COL.
17

---

10%
25%
50%
100%
Task accuracy CUB
83.9
83.7
83.2
83.4
=0.1
#0.7
+0.5
+0.2
Task accuracy TIL
54.0 20.7
54.1
53.8 #1.2
53.8
10.1.1
Table 9: Evaluating the robustness of COL loss in the presence of concept correlation. We randomly duplicate a percentage the concept bank and evaluated our model Coop-CBM+COL on the CUB and TIL datasets.
Histogram of concept logits for joint-CBM and coop-CBM To gain further insight into the effect of COL, we computed histograms of the activations of the penultimate layer (to which the loss is applied) and saw that the histogram of CBM+COL is very sparse (with a large peak at O and a much smaller one 1) in contrast the vanilla CBM.
Histogram concept logits for joint-CBM
coop-CBM+COL
joint-CBM
100
80
Frequency
60
40
20
0.0
0.2
0.4
0.6
0.8
Value
Figure Concept logits histogram comparison on TIL. dataset
Intuition The dissimilarity loss d2 encourages independent concept prediction. This is particularly important to reduce leakage in CBMs and improve the robustness of concept explanations. For the connection between entanglement of concept predictions and leakage, refer the reader to the introduction section Havasi et al. [20]. Motivated by this insight, we introduce d2, to encourage disentanglement of the penultimate layer of features for concept prediction. We agree that this may result in an overcomplete representation, induced by the opposing forces of dj and d2 for samples with partially overlapping concepts, and multiple feature groups may contribute to the same concept. However, it might be difficult to achieve disentanglement otherwise, and as our experiments show, COL improves the concept representation and accuracy, including in the out-of- domain settings. To gain further insight into the effect of COL., we computed histograms of the activations of the penultimate layer (Figure 3 and see that the histogram of CBM+COL very sparse (with large peak at 0 and a much smaller one at 1) in contrast to the vanilla CBM. This suggests that COL may encourage learning of an overcomplete sparse feature space, the elements of which encode various combinations of concepts, and the last layer of learns to introduce invariance the prediction of each concept G; with respect to specific combinations with other concepts си by linear combination. Additionally, it must be noted that this regularization is only applied to the penultimate layer before concept prediction which means low-level features are still free to share weights. Essentially, we believe COL encourages learning sparse over-complete dictionary of features with concepts still partially entangled different combinations. The concept prediction layer in f then learns linear combination of these specialized features to introduce invariance
18

---

with respect to the specific combinations. In fact, when studying histograms of activations of the penultimate layer, we observe that activations with COL are indeed very sparse in contrast to CBMs without COL. Given a sparse dictionary of combinations of concepts as induced by COL, the task of disentanglement of concept prediction would ideally reduce linear combination of dictionary elements. This is the intuition behind sparse coding (see [30]. In our case, the sparsity is induced indirectly as a result the arthogonality-basee loss formulation. Our experiments show that this approach significantly facilitates the overall optimization of CBMs, improving concept accuracy and downstream performance.
C.3 Effect of lambda on COL
We experimented with different loss weights for À in our experiments and the model+COL seemed be fairly robust with different values of A. We have put those results in Table 5 of the rebuttal PDF on CUB and TIL datasets. While fine-tuned value of À might show good performance, regardless, the model still able to beat the performances of other baselines. We observed that 0.05 as a good tradeoff between performance and uncertainty across datasets.
observe that
Dataset
A=0.55
X=0.1
A=
1=1.0
>=10.0
CUB
84.1
84.1
83.8
84.0
83.6
40.5
10.3
TIL
54.2
54.0
54.3
53.6
54.1
20.9
20.6
40.8
11.0
Table 10: Effect of À on the coop-CBM+COL model. We observe that the orthogonal loss is fairly robust to hyperparameter selection.
D Further experiments
D.1 Comparison against models with automated concept acquisition
38 and [58] used pre-trained model CLIP which was trained on a massive corpus of data to obtain concepts. This can potentially introduce inherent biases from pretraining into the concepts. This was also brought up in the Limitations and Conclusion section of [58). Furthermore, the dissimilarity in the concepts employed in these works adds complexity to establish fair and meaningful comparison. Moreover, we wish to emphasize that neither of these works directly compare with CBM variants in their main paper, except for [58 which appears in Appendix C. Also is not possible to compare with realistic medical datasets CI.IP fails to generate meaningful concepts. Regardless, we have compared both the methods with our method on CUB+OOD datasets and our model outperforms the accuracy of (38] and (58), which is lower than the standard model. We believe it will be interesting future to include our model methodology and COL in the respective models.
Model type
CUB
OOD-CUB
corr-CUB
Standard
82.3
27.7
63.3
PCBM
78.4
33.4
62.7
PCBM-h
80.9
32.1
62.9
LF-CBM
81.0
33.8
63.9
Coop-CBM
83.6
35.4
64.6
+ COL
84.1
36.2
64.9
Table 11: Comparison of Posthoe CBM and Label-Free CBM with our proposed methodology. OOD-CBM refers to Exp 5.1 relating to spurious correlation generalization. Coor-CBM refers to Exp 5.2 relating to image corruption. Unfortunately, Exp 5.3 could not be conducted due to the different concept bank.
19

---

D.2 Coop-CBM and COL in the presence of sparse concept labels
Concept labeling could be labor-intensive task and hence it is important to understand the most optimal point of operation. We randomly select subset of concepts and train baselines on the subset. Due to concept and task prediction at the same level, we observe coop-CBN provides inductive bias for the downstream task.
Model
10%
25%
50%
100%
Standard
82.3
82.3
82.3
82.3
=0.4
20.5
20.3
+0.2
Independent CBM 27
33.6
62.0
74.7
76.0
=0.5
-0.3
20.1
40.2
Sequential CBM 27
45.5
62.0
74.3
76.3
=0.3
=0.1
=0.2
+0.2
Joint CBM 27
75.2
78.5
79.6
80.1
=0.1
=0.5
20.3
+0.1
CEM 60
76.1
79.3
79.5
82.5
=0.5
=0.2
#0.1
+02
CBM-AR 20
73.4
77.8
79.7
81.6
20.4
20.4
10.4
Coop-CBM (ours)
81.4
82.9
83.7
83.6
20.3
20.5
20.4
10.3
+ COL
82.4
83.2
83.9
84.1
=0.4
20.3
20.5
10.2
Table 12: Task accuracy CUB with sparse concept annotations (% fraction of concepts)
Model
10%
25%
50%
100%
Independent CBM 27
94.2
95.3
96.7
96.6
0.2
20.1
10.3
10.0
Sequential CBM 27
94.2
95.3
96.7
96.6
001
20.3
10.2
10.1
Joint CBM
90.7
92.0
93.5
93.2
=0.3
10.1
CEM 660
93.6
93.9
94.1
94.8
=0.2
+0.2
CBM-AR 20
93.1
93.7
94.0
94.2
10.1
Coop-CBM (ours)
92.5
92.9
93.6
93.9
=0.3
=0.2
=0.1
#0.2
COL
96.6
96.8
97.1
97.3
0.2
20.1
20.3
10.2
Table 13: Concept accuracy CUB with sparse concept annotations (% fraction of concepts)
Model
10%
25%
50%
100%
Standard
51.1
51.1
51.1
51.1
=1.2
10.9
Independent CBM 27
15.4
25.6
43.2
47.4
+1.
Sequential CBM 27
19.8
27.9
43.9
47.9
20.7
10.9
Joint CBM 27
43.5
44.9
46.1
47.6
11.
20.6
10.7
CEM 60
46.1
47.9
49.2
51.3
20.9
11.3
CBM-AR 20
46.1
48.8
49.2
49.5
=1.1
20.8
+10
Coop-CBM (ours)
50.3
51.7
52.8
53.4
=1.4
40.8
+ COL
51.0
52.2
53.5
54.2
=1.0
21.5
+0.9
Table 14: Task accuracy TIL with sparse concept annotations (%
fraction of concepts)
20

---

Model
10%
25%
50%
100%
Independent CBM 27
92.3
94.5
95.7
96.4
=0.5
=0.3
=0.7
+0.2
Sequential CBM
92.3
94.5
95.7
96.4
=0.6
=0.1
=0.8
#0.3
Joint CBM 27
91.7
92.3
93.2
93.9
20.7
20.5
10.2
10.6
CEM 60
93.2
93.8
94.2
94.4
20.8
20.6
10.3
10.7
CBM-AR 20
93.6
93.8
94.2
94.2
2.02
20.8
10.5
10.6
Coop-CBM (ours)
93.2
93.8
93.9
94,2
0.3
20.7
10.4
10.8
+ COL
96.4
96.6
97.0
97.1
=0.4
=0.2
=0.6
+0.5
Table 15: Concept accuracy TIL with sparse concept annotations (% fraction of concepts)
Model
10%
25%
50%
100%
Standard
96.2
96.2
96.2
96.2
=0.2
=0.1
#0.1
Independent CBM 27
42.4
67.6
89.4
94.9
=0.4
=0.2
#0.1
+0.3
Sequential CBM
52.6
71.5
91.7
94.6
2033
2.66
10.2
10.2
Joint CBM 27
89.2
91.8
94.0
95.4
20.
20.4
10.3
10.1
CEM 60
89.9
93.1
95.5
96.2
00.6
1033
10.1
10.1
CBM-AR 20
91.0
92.6
93.8
95.9
=0.2
=0.1
20.1
20.0
Coop-CBM (ours)
92.6
94.2
96.1
96.6
=0.3
=0.1
=0.2
#0.1
+ COL
92.9
95.4
96.5
97.0
=0.1
=0.1
=0.3
20.6
Table 16: Task accuracy AwA2 with sparse concept annotations fraction of concepts)
Model
10%
25%
50%
100%
Independent CBM 27
94.7
95.8
97.3
97.7
10.3
Sequential CBM 27
94.7
95.8
97.3
97.7
Joint CBM 27
93.1
94.2
94.8
95.2
10.1
CEM 60
92.6
93.2
94.8
95.6
CBM-AR 20
93.5
93.7
95.0
95.4
=0.2
=0.1
20.4
+0.3
Coop-CBM (ours)
93.5
94.6
95.2
95.7
=0.3
10.4
+ COL
96.6
96.8
97.1
98.4
10.1
Table 17: Concept accuracy AwA2 with sparse concept annotations (%
fraction of concepts)
21

---

D.3
Detailed results with image corruptions
TIL
Model
2
3
5
7
Avg
Standard
35.5
34.7
39.3
35.7
41.6
39.9
43.9
38.4
+12
#1.7
+1.4
#1.1
=1.6
$1.3
#1.0
Independent CBM 27
32.8
31.9
39.5
34.3
40.8
32.9
43
36.6
1.55
11.0
+1.7
11.2
11.
11.6
1..3
Sequential CBM 27
33.0
31.6
39.9
34.6
40.2
32.4
42.9
36.3
1.33
114
11.1
11.6
1.5
11.0
11.7
Joint CBM
33.3
32.3
41.6
38.2
41.2
33.3
39.8
37.1
11.4
113
11.2
11.7
11.0
11.5
E1.1
CEM 60
35.2
34.8
40.0
37.3
42.1
33.6
46.5
38.5
+1.1
=1.6
#1.5
#1.0
=1.
#12
+13
CBM-AR 20
35.4
35.1
39.7
38.4
41.6
42.8
25.6
36.8
#1.7
#1.
#1.3
#14
=11
#1.
#1.
Coop-CBM (ours)
36.7
36.2
41.4
37.6
43.3
39.9
49.1
40.6
#1.0
=1.7
+12
11.3
=1.
#1.1
#1.
+ COL
37.2
36.5
41.5
38.3
43.0
44.4
45.4
40.9
#1.
#1.5
#1.0
#1.7
=1.2
#0.9
#1.0
Table 18: Comparison of concept-based models on image corruptions on TIL datasets
E
Further about interventions
Concept Uncertainty Score
A significant part of the intervention selector constitutes Concept Uncertainty Score (CUS) which denotes the model uncertainty for the prediction of concepts. We calculate epistemic uncertainty that arises due to model parameters and lack of training samples. Realistically, labeled medical data often scarce, encouraging the application of epistemic uncertainty quantification the predicted concepts. We use Monte-Carle Carlo dropout [16] to model epistemic uncertainty, with random dropout rate of 0.2. We apply the dropout before the prediction concepts. For an image æ; predicting concepts G..-K where K is the total number of concepts, we evaluate T softmax probabilities {p.)7-1 for each concept prediction. We measure the uncertainty for each concept which we refer to as H(-). We compute the entropy-based uncertainty for each concept the measure of the expectation of the information inherited in the possible outcomes random variable. Using the uncertainty metric, calculate the overall uncertainty concept vector H.
N
1 T
H((-)
Σ 2m
Pg (c)x log
Σ (c(x,))) T 7m1
(9)
N
T m
Concept Weightage Score
The second part of the intervention selector score is signified by Concept Weightage Score (CWS), which accounts for the importance concept in the final downstream prediction task. Using CWS, the intervention selector is able to prioritize the concept for intervention that is deemed to change the final prediction significantly. We define the weightage score as 3. The fyle) linear one-layer Multi-Layer Perceptron (MLP) which helps in defining 3.
N
8, = C5
Wiy
(10)
1=1
Supervisor Confidence Score
Finally, in the intervention selector, we consider the reliability of the annotator via Supervisor Confidence Score (SCS). For example, while a histopathologist can identify diseases across human tissues and organs, they often have more specialized and nuanced areas of focus. It is therefore
22

---

beneficial for the model to request additional information from histopathologists from their expert knowledge. This in practice prevents ambiguous or inaccurate concept correction. The SCS is variable across each annotator and is represented by
Finally, bringing the three desiderata of concept selection for intervention, the final intervention selector score is linear combination of all of the scores.
T
)) + k2
ATSelect ki *
Σ
Σ
Ps (c(c))))) log T
Σ Po pp((lz))
wayl + hig *
N
(11)
where k1, 62, f3 are hyperparameters for importance weightage each of the scores.
Intervention allows the model to query the most significant concepts. In our case, it hypothesized that the most uncertain concepts will build a symbiotic relationship between the human and the model. The supervisor decides the threshold Ze to correct the concept prediction. Realistically, in medical scenarios, due to the professional's 's limited availability, we would like to optimize the number of concepts to intervene. Setting lower threshold is a trade-off decided by the user. In contrast to other works that perform group interventions 27 by intervening on group of similar concepts, we perform single interventions. Group interventions require clustering of concepts on the basis of their similarity which is not realistic as such information is not always available. Therefore, single interventions are performed to minimize the dependence on human priors.
Algorithm Intervention selector Pseudocode
X < input image
Cr f- intermediary concepts
Y label
f Image to concept prediction model
+ Concept to label prediction model
Cn...C g(X)
for i = 1,2.....7 do
H4: CUS(c)
3: CWS(G)
7. SCS(c:)
G = k * Hi + 1:2 3g + his * %
end for
é Cth - threshold
Cthr < intervene on threshold valued
Y = f(()...o))
F Example of OOD data
Figure L: example from trainset, example from in-domain testset, R: example from out-domain testset from spurious background correlation CUB synthetic dataset, class Black Footed Albatross
This similarity metric is pre-detined by humans
23

---

Accuracy vs interventions CUB
965
76
CMS
74
Accuracy
72
70
200
250
300
Number of single interventions
Figure 4: Interventions on Joint CBM in the presence of image corruptions on CUB dataset
Accuracy interventions CUB
Mander((aa))
84
SC5
CWS
02
Accuracy
80
78
76
74
0
50
100
150
200
250
300
Number of single interventions
Figure 5: Interventions on Joint CBM in the presence of image corruptions on CUB dataset
G Limitations and Future Work
In this work, we proposed novel concept-based approach, coop-CBM, to enhance the interpretability and accuracy trade-off in AI models. We introduced the Concept Orthogonal Loss (COL) to improve concept learning and employed Coop-CBM on various datasets and evaluation scenarios. Our results demonstrated superior generalization, robustness to spurious correlations, and advancements in the accuracy-interpretabilty trade-off.
While our proposed approach has shown promising results and made significant contributions, it important recognize certain limitations. Firstly, the reliance on labeled concept vectors poses challenges in domains where concept annotations are limited costly to acquire. Furthermore, current concept annotation methods suffer from biases and a lack of domain knowledge, indicating the need for further improvements in this area. Our work does not address unsupervised concept acquisition methods, but instead focuses on a model architecture that can be applied regardless the concept acquisition approach.
While our concept-based approach offers interpretability, it assumes that the learned concepts align closely with human-ume undiansnnnnsttaaaaleeee notions. However, there is possibility that the learned concepts may not always perfectly align with the intended interpretations, which can pose challenges in terms of comprehensibility and explainability. Future research could delve deeper into understanding explanations and their alignment with human understanding. thereby exploring ways to improve fidelity of concept-based models in providing accurate and meaningful explanations. Future research could aim to incorporate additional evaluation metrics that assess the transparency, fairness, and robustness aspects of concept-baa based models.
24

---

Overall, while our approach shows promising results and addresses important concerns in the field of explainable AI, it is important to be aware of these limitations and continue advancing research to overcome them and further enhance the applicability and reliability of concept-based models.
25


Reviews:
[{"review_id": "Pq9uiOAnOs", "reviewer": "Reviewer_G8F4", "summary": "This paper proposes auxiliary losses to learn concept-bottleneck models: a coop task loss, and a COL (orthogonal concept loss). The experiments show that the proposed coop-cbm can perform 1-3% better than standard black-box models and outperform other CBM models on three datasets: CUB, AwA2 and TIL.  \n\n==================\npost-rebuttal: the authors have addressed my concerns and questions in the rebuttal. I would like to increase the rating from 4 to 6. ", "questions": "1. It's not clear why the concepts are necessary need to be orthogonal. Could the authors explain? \n\n2. In eq 5, how do you define intra-concepts? It looks like you are using c_i= c_j, which means the concepts are exactly the same? Please elaborate the details. \n\n3. Line 228-229, why is it unfair to compare with post-hoc CBM [33]? Do you mean that it is expected that [33] with joint training can outperform coop-cbm? please clarify\n\n4. Please report computation cost for baselines and the proposed method in Table 1.\n\n5. How is Fig 1 calculated? Is it correcting the mis-classified images in the test time through intervention? Please give more details on how Fig 1 is plotted. ", "limitations": "I did not see a dedicated paragraph to discuss the limitation of the proposed work. ", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This topic is important\n2. The paper compared with 3 fairly new CBM baselines (CBM [12], CEM [35], CBM-AR [8]), but stronger baselines are not compared [33, 18]. The authors mentioned it's an unfair comparison to [33] in Line 228-229, but it's still not clear why, and why [18] is not compared either. ", "weaknesses": "1. The proposed idea is interesting, though the two losses (COOP loss Eq 3, and COL in Eq 6) seem to be proposed in prior works? The authors mentioned that COL is from [21], and it feels like the COOP loss in Eq 3 is similar to the non-interpretable \"residual\" terms in Post-hoc CBM [33].  So the main contribution seems to be integrating them together? Please clarify and compare if there's some misunderstanding here. \n\n2. The description of some related work seems not accurate: Line 74-76: what does it mean that \"suffer from pretrained model's biases\"? Post-hoc CBM [33] with residual terms can improve the downstream task accuracy. It looks like COOP can perform better is due to the residual loss + join training (since post-hoc CBM only training the last 2 layers)? Besides, the result in Label-free CBM [18] show that it is comparable to original downstream task performance without the residual terms. Please compare with [33] and [18] with ablation study.\n\n3. Large scale datasets results are missing (ImageNet)\n\n4. The exposition of the technical part can be improved (sec 3). For example, to describe f, g, h, it'll be useful to visualize the structure. phi and theta are not defined. Also, since f and h actually differ in one linear layer, why both of them still use theta? it seems phi is the parameter of g. \n\n\n\n\n**minor**\n- an extra s in Line 277 \"modelss\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback. \n>The authors mentioned it's an unfair comparison to [33] and why [18] is not compared either.\n\n[18] and [33] used a pre-trained model - CLIP which was trained on a massive corpus of data to obtain concepts. This can potentially introduce inherent biases from pretraining into the concepts. This was also brought up in the Limitations and Conclusion section of [33]. Furthermore, the dissimilarity in the concepts employed in these works adds complexity to establish a fair and meaningful comparison. Moreover, we wish to **emphasize** that neither of these works directly compare with CBM variants in their main paper, except for [33] which appears in Appendix C. Also it is not possible to compare with realistic medical datasets as CLIP fails to generate meaningful concepts. Regardless, we have compared [18,33] with our method on CUB+OOD datasets in Table 1 of rebuttal pdf, and our model outperforms the accuracy of [18, 33], which is lower than the standard model.\n>The two losses (COOP loss Eq 3, and COL in Eq 6) seem to be proposed in prior works. The authors mentioned that COL is from [21].\n\nOrthogonality and alternative loss functions have been active research areas. [21] focused on establishing orthogonality in the embedding space between features of distinct classes. Therefore, the loss function was added just before the classifier layer. However, the COL we propose aims to induce orthogonality among the different concepts predicted. This introduces orthogonality between the respective linear layers associated with each concept. As a result, our formulation considerably deviates from that of [21], as we're introducing orthogonality within a distinct space and across different dimensions. \n>It feels like the COOP loss in Eq 3 [...] Do you mean that it is expected that [33] with joint training[..]\n\nWe cannot make claims on how PCBM-h[33] would perform when jointly trained but it should be noted that the residual term in [33] is non-interpretable which might defeat the purpose of concept-based explanations while less accurate than the standard model. Hence unlike [33], we can achieve higher task accuracy while maintaining high concept accuracy (Table 2, main paper).\n>Large scale datasets results are missing (ImageNet).\n\nWe have validated our method on 3 diverse datasets: TIL is a realistic medical dataset, AwA2 is a large-scale 13GB dataset and we have performed multiple realistic evaluations on distributional shifts. Most works on CBMs do not evaluate on ImageNet apart from [18] whose concept acquisition methodology is different.\n>For example, to describe f, g, h, it'll be useful to visualize the structure. \n\nWe have added the figure in the rebuttal PDF.\n >Also, since f and h actually differ in one linear layer, why do both of them still use theta? it seems phi is the parameter of g.\n\n$\\theta$ is the encoder layer. It should be noted that f has multiple linear layers as well. Yes $\\phi$ is the parameter of g. We shall make it clearer in the camera ready version. \n>It's not clear why the concepts are necessary need to be orthogonal. \n\nBriefly mentioned in Section 3, we introduce orthogonality to increase the separation between different concepts. Independent CBMs[12] which take binary (0/1) concept values as input into the concept predictor, exhibit the highest concept accuracy, suggesting enhanced interpretability. Simultaneously [8,16] indicated that information leakage via \"soft\" concept labels leads to low concept accuracy compromising explainability. COL introduces orthogonality constraints among different predicted concepts, promoting their independence and reducing the potential for information leakage between them. By encouraging orthogonality between concept representations, COL promotes learning of distinct, disentangled, and meaningful high-level concepts. COL leads to three significant applications: 1) enhances concept representation and accuracy (see Table 2 main paper on CUB), 2) improves task accuracy 3) helps to make more meaningful interventions i.e. with a lower intervention budget, one can achieve higher accuracy. Orthogonality in concept learning is crucial for models aiming to balance accuracy, explainability, and prevent information leakage.\n>Computation cost\n\nWe have added this in Table 2 of the rebuttal PDF\n >Please give more details on how Fig 1 is plotted.\n\n* L->R-The first two graphs show the importance of using human uncertainty(SCS) as a metric to select interventions as compared to concept uncertainty(CUS) and weightage(CWS). We show this in two ways:\n* First we plot for accuracy of the CBM model (as used by other baseline metrics [3,26]) when the annotator correctly intervenes on the concept. Intuitively, the accuracy should increase.\n* Second, consider that scenario is similar to first except the annotator has changed. The annotator with ill intentions changes the concept values to incorrect concepts (instead of GT concepts). Now we plot all of the metrics and intuitively the accuracy would reduce.\n* L->R Last two graphs of Figure 1 compare the random interventions of different baseline models. Since SCS was present for CUB only, we consider random interventions for TIL and CUB\n>In eq 5, how do you define intra-concepts? c_i= c_j clarification.\n\nIn retrospect, we would like to rewrite as $d_1=\\sum_{\\substack{i,j\\in B,\\\\\\\\c_i^a = c_j^a\\\\\\\\a\\in A}}\\frac{q_i^T q_j}{||q_i|| ||q_j||};d_2=\\sum_{\\substack{i,j\\in B,\\\\\\\\ c_i^a\\neq c_j^a \\\\\\\\a\\in A}} \\frac{q_i^T q_j}{||q_i||||q_j||}$ where A is total concepts. $c_i=c_j$ brings concept representation of the same label concept closer\n>Limitations\n\nIn the original submission, it is part of Appendix B. We will move it to the main paper.\n\nWe would like to thank the reviewer again for the valuable feedback. We ran the requested experiments and attempted to answer all of the concerns of the reviewer and would be happy to answer any further questions."}, {"review_id": "MoW385ntcL", "reviewer": "Reviewer_Mi12", "summary": "The author proposes coop-CBMs + concept orthogonal loss (COL) to solve previous limitations in Concept Based Models with a focus to learn relevant concept representations that consecutively boost model performance.", "questions": "1. modify the structure/content arrangements in introduction section. \n2. do more literature review on section2 on the relationship between Concept-Based Model and explainability if applicable.\n3. please clarify the notations in method part.\n4. please clarify what is your final method, Coop-CBM or Coop-CBM + COL ?\n5. add classification results for only using Coop-CBM on CUB, AwA2 and TIL datasets.\n6. better to tell the significance of the proposed method, not just limited to some improvements in the benchmark dataset.", "limitations": "No. \n\nit would better to discuss the limitations of proposed methods ( Coop-CBM along with COL) in `Discussion and Conclusion` section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "\nIn general, the idea of coop-CBMs+COL is simple and effective. The method increases prediction accuracy on benchmark datasets and  seems to work well on noisy inputs.", "weaknesses": "* The overall structure of the paper is reasonable, but part of substructure in some sections are not well organized. In introduction section, the author first introduce definition of previous Concept bottleneck models, then pointing out its limitation.Then followed with the importance of studying the robustness of CBMs in various domains. After that, returns back to bottleneck of CBMs. Unless the author aims to describing totally different limitations of previous CBMs, it feels unnecessary.\n\n* In Section3, the author does not specify what does $\\hat{\\theta}$,  $\\hat{\\Phi}$ stands for,  it would be better to insert formula into the words such as a formula followed by it notations. Otherwise this description looks messy. \n\n* The author does not show experimental results for using CBM alone on classification Coop-CBM.  \n\n* It's confusing calling Coop-CBM both a component of the final strategy and the whole strategy incorporating it.There is a bit confusing that what is your final method. Coop-CBM or Coop-CBM + COL ?\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback. \n> The overall structure of the paper is reasonable, but part of substructure in some sections are not well organized.\n\nThank you for your feedback. We shall rewrite that section by moving the two paragraphs about distributional shifts and limitations of CBMs around.  We will update the camera ready version accordingly to improve readability.\n\n> the author does not specify what does $\\hat{\\theta}$, $\\hat{\\phi}$ mean.\n\n$\\hat{\\theta}$, $\\hat{\\phi}$  are the parameters of the encoder and $g$ correspondingly (please see Figure 1 from rebuttal PDF).\n\n> The author does not show experimental results for using CBM alone on classification Coop-CBM.\n\nIn our initial submission, we have already conducted comprehensive evaluations of CBM, coop-CBM, and the inclusion of COL, (please refer to Tables 1, 3, 4, and 5 in the main paper, along with Tables 1, 2, 3, 4, and 5 in the appendix). Furthermore, we have conducted an additional analysis in Table 2 by isolating the effect of COL within different baseline models. This analysis demonstrates that regardless of the chosen concept model, the incorporation of COL consistently enhances concept accuracy. \n> It's confusing calling Coop-CBM both a component of the final strategy and the whole strategy incorporating it.There is a bit confusing that what is your final method. Coop-CBM or Coop-CBM + COL ?\n\nAs per the title of the paper, we are suggesting losses that can be used to improve the generalization of concept-based models. We will add a sentence in the introduction to emphasize this. Our contribution in terms of the model is two fold: first to improve task accuracy for concept-based models via Coop-CBM, and second to improve the concept accuracy for concept-based models via COL. We empirically show that each of the strategies alone works but definitely the most effective strategy is to use Coop-CBM + COL. From Table 1,3,4,5 penultimate line, we observe that Coop CBM alone improves task accuracy but with COL, we can further enhance the task accuracy. As mentioned in our paper, information leakage is detrimental to concept accuracy and hence the explainability of the model. We thus add the COL loss to increase the independence of learning each concept from the other as observed in Table 2 of the main paper for CUB.\n\n> do more literature review on section2 on the relationship between Concept-Based Model and explainability if applicable.\n\nWe did our best to include all of the recent related references of Concept-Based models that were released until the Neurips deadline in the related works section of the main paper. We will add all new references mentioned by other reviewers. Please let us know if you think we are missing a particular reference and we will include it in camera ready. In the Appendix, section D, of the original submission we have included a further literature review on explainability. \n\n> please clarify what is your final method, Coop-CBM or Coop-CBM + COL ?\nOur contribution encompasses the introduction of both coop-CBM and COL. Both these methods, individually and when combined, exhibit improvements in both explainability, accuracy and test-time interventions.\n\n> add classification results for only using Coop-CBM on CUB, AwA2 and TIL datasets.\n\nThe main paper has classification results for using coop-CBM on CUB, AwA2 and TIL datasets in Table 1. In addition, we have results for coop-CBM only on all of our other tables as well in the original paper. The penultimate row of each of the tables mentioned that says “Coop-CBM” signifies classification results using only Coop-CBM. The “+COL” row signifies the classification result using both Coop-CBM+COL. We will add a sentence to each table caption to highlight this.\n\n> better to tell the significance of the proposed method, not just limited to some improvements in the benchmark dataset.\n\nThis work introduces two significant contributions to concept-based models. Firstly, we propose a multi-task model that predicts an intermediary task label alongside concept predictions. This is particularly useful when dense and relevant concept annotation is lacking, as seen in the TIL dataset. Secondly, we incorporate orthogonality constraints in the concept representation space through concept orthogonal loss during training. This loss enhances inter-concept separation while reducing intra-concept distance. Through extensive experiments across diverse datasets and distributional shifts, we observe that the bottleneck layer preceding the final prediction enhances the robustness of concept-based models against spurious background correlations. Coop-CBM combined with COL demonstrates leading performance in both task and concept accuracy. \n\nAdditionally, we would like to point out that most of the current CBM works do not perform extensive experiments on distributional shifts. Our paper is the first work that attempts to shed light on the impact of using CBM on multiple different distributional shifts. We have attempted to justify and motivate our work both empirically and intuitively. In Section 3, we introduced the intuition behind our contributions and then throughout the paper we provide analytical experiments which highlight each contribution’s effectiveness. \nPlease point us to any specific issues and we will address them during the discussion period.\n\n> it would better to discuss the limitations of proposed methods ( Coop-CBM along with COL) in Discussion and Conclusion section.\n\nWe have included limitations in Appendix B. Following your suggestion we will move this discussion to the Conclusion for camera-ready version.\n\nWe have attempted to answer all of the concerns of the reviewer and would be happy to answer any further questions. We thank the reviewer again for their feedback.\n"}, {"review_id": "P9zjH0mQCT", "reviewer": "Reviewer_1MWB", "summary": "The authors introduce an auxiliary task in CBM training to incorporate more information regarding the downstream task in the input representation for predicting concepts. They also introduce Concept Orthogonal Loss (COL) to enforce orthogonalization between input representations of different concepts, and vice versa. Experiments show superiority of their method over existing CBM baselines on the performance of the downstream task, for standard training settings as well as settings with manually-injected distribution shifts.", "questions": "I am willing to increase the score and accept the paper as I genuinely believe this work is a good solid step in terms of making concept-based models more viable if the following issues are resolved properly.\n\n### Risk of information leakage\nMy main concern is that coop-CBM improves downstream task performance because downstream task information is leaked through the concepts. This makes intuitive sense as the explanation of performance gain is the representation used to predict concepts contain more downstream task information. Actually I am not sure where the author stands on the topic of leaking information through the concepts. Does the authors believe that benign leakage of information is acceptable? \n\nPerhaps experiments could be repeated by clipping the concept predicted values in coop-CBM to hard labels (as suggested in Mahinpei et al's paper) to reduce the leakage and see if there is still performance gain (or maybe this is already the default behavior indicated somewhere in the work that I missed).\n\nThe risk of information leakage is that then the concept scores no longer purely represents the concept, but rather encodes the downstream task label in some implicit manner. Then the whole concept bottleneck idea breaks down. The bottleneck is no longer a bottleneck and explainability through concepts is no longer valid.\n\n### Assumption of all concepts should be orthogonalized\nThe authors extended the idea of concept whitening to CBMs through COL. However, orthogonalizing every concept may not be always beneficial (e.g. related concepts). Are the authors claiming orthogonalizing concepts is always beneficial? Is it possible to characterize when to adopt COL? Perhaps it is impossible to make definite statements about when COL would work but is it possible to at least provide a good heuristic that serves as a good-enough guideline?\n\nFor example, concepts in CUB are grouped into attributes of different parts of birds (e.g. body, head, wing). Does doing COL on inter-grouped concepts make sense? Perhaps something could be done with prior knowledge of which concept correlates more with others? Are there easy metrics that could serve as a proxies for concept correlation?", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Simple and straightforward presentation of the proposed methods. Very easy to read.\n* Proposed method makes intuitive sense.\n* Solid experiments.\n* Experiments on extensive settings to demonstrate the superiority of CBMs over standard models w/o concepts.", "weaknesses": "* Risk of information leakage: Mahinpei et al. have shown that CBMs may leak information of the target task through the concepts, also mentioned in the paper (P2L50). Whether the performance gain of coop-CBM comes from information leakage is not exactly clear.\n* Assumption of all concepts should be orthogonalized: The proposed COL implicitly assumes that all concepts should be orthogonal. The information gain shown in the experiments that this assumption holds true in the tested settings. In general, user selected concepts may not be entirely orthogonal. This method may be limiting when a user does not know whether adopting COL is appropriate given the chosen concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> My main concern is that coop-CBM improves downstream task performance because downstream task information is leaked through the concepts. This makes intuitive sense as the explanation of performance gain is the representation used to predict concepts contain more downstream task information[....]\n\n* We agree with the reviewer that information leakage is an important concern for concept-based models. As pointed out by the reviewer, and multiple times in our main paper, we have considered the impact of information leakage. This leakage occurs when the concepts that are supposed to be independent and interpretable become influenced by each other due to shared task-related information. \n* To overcome this issue, COL is designed to encourage orthogonality among the learned concept representations. Mathematically, COL adds an orthogonal regularization term to the loss function used during the training of concept classifiers. This term encourages the learned concept representations to be orthogonal to each other, thereby reducing any inadvertent inter-concept correlations. \n* One way to evaluate concept leakage in literature has been through concept accuracy.  COL enhances concept independence for concept-based models. In Table 2 (main paper), we evaluate the concept accuracy of each baseline model on the CUB dataset. Here, as one can expect the “hard” CBM has the highest concept accuracy in the absence of COL. But with the introduction of COL, the concept representation for each baseline was learned to reduce the information leakage. Coop-CBM+COL achieves comparable concept accuracy to Independent (hard) CBM with higher mean and comparable uncertainty. \n* This result aligns with the fundamental goals of explainability and interpretability, which are pivotal in ensuring that the concept-based model's representations remain meaningful and free from unintended correlations, ultimately enhancing the model's utility and reliability in various applications.\n\n>Perhaps experiments could be repeated by clipping the concept predicted values in coop-CBM to hard labels (as suggested in Mahinpei et al's paper) to reduce the leakage and see if there is still performance gain (or maybe this is already the default behavior indicated somewhere in the work that I missed).\n\nThank you for suggesting the experiment. Following your suggestion, we performed two sets of experiments on CUB and TIL datasets using coop-CBM.\n* First, we trained the model by clipping the predicted concept values to “hard” labels (Table 3, Exp1 in rebuttal pdf).\n* Second, we trained the model as we have described earlier in the paper (using soft labels) and the model evaluates on the test set by clipping to “hard” labels (Table 3, Exp2 in rebuttal pdf).\nFrom the above experiments, we conclude that the model is able to learn a good representation of the concepts without necessarily leaking information.\n\n>Are the authors claiming orthogonalizing concepts is always beneficial? Is it possible to characterize when to adopt COL? Perhaps it is impossible to make definite statements about when COL would work but is it possible to at least provide a good heuristic that serves as a good-enough guideline?\n\nWe would like to connect this answer with the previous answer. The motivation for COL arose to mitigate information leakage in coop-CBM and is applicable to other concept-based models too. The rationale for COL revolves around its ability to counteract the inadvertent propagation of task-specific information across concept representations. By imposing this constraint, COL effectively disentangles the influence of different concepts, fostering their independence and reducing the risk of cross-concept information leakage interpretability, and the effect of interventions. Also, please see below for detailed answers.\n\n>For example, concepts in CUB are grouped into attributes of different parts of birds (e.g. body, head, wing). Does doing COL on inter-grouped concepts make sense? Perhaps something could be done with prior knowledge of which concept correlates more with others? Are there easy metrics that could serve as a proxies for concept correlation?\n\n* Devising an optimal point for when to use a loss function could be great future work building on our work. Also, prior knowledge about concept groups cannot be assumed. One way to characterize the correlation could be through using mutual information, which can be an interesting followup work. \n\n* We would like to emphasize that COL can be used to improve the concept representation of any concept-based model. For instance, considering the case of Joint-CBMs, which has been known to exhibit information leakage, the incorporation of COL can effectively increase separation in concept learning leading to enhanced interpretability while mitigating the risk of information leakage (See Table 2 main paper). \n\n* In order to investigate the effectiveness of COL we propose a comprehensive experiment that deliberately explores an extreme case for evaluation in line with the reviewer's proposal. We aim to showcase the robustness of COL across different scenarios, extending the OOD experiments in the main paper.\nExperiment- We consider a scenario where input concepts are intentionally duplicated to create a high degree of concept correlation. In our experiment from Table 4, we duplicated 10%, 25%, 50% and 100% of concepts and added them to the original concept bank. This is a worst-case representation of “similar concepts”. From the table, we see that the duplication of concepts does not impact the concept or the task accuracy.\nAdditionally, this experiment contributes to the broader understanding of how COL performs in various scenarios.\n\nWe thank the reviewer again for the detailed questions and insightful additional experiment suggestions. We hope we addressed all concerns and are happy to answer and further questions during discussion period.\n"}, {"review_id": "dmalwht5yK", "reviewer": "Reviewer_K2hb", "summary": "This paper proposes Coop-CBM, a concept bottleneck model (CBM) trained using a novel multi-task loss and orthogonality regularizer. The proposed losses discourage undesired leakage in a CBM's concept representations while encouraging a better balance of their accuracy and interpretability. By incorporating these loss terms in their objective functions, *Coop-CBMs* are encouraged separate latent representations learnt for different concepts while reducing the intra-concept distance. This design enables effective concept interventions in these models while maintaining high concept and task accuracies. The authors perform an extensive evaluation of Coop-CBMs on four real-world datasets for image classification tasks, together with corrupted variants, and show that the proposed method achieves higher accuracy in all instances, compared to black-box models and previous CBM-based models, while remaining receptive to test-time concept interventions.", "questions": "After carefully reading the paper and the appendix, I had some trouble getting convinced by some of its results and proposed methods. My score is mostly influenced by this work's (1) lack of proper evaluation against existing baselines (specifically in how hyperparameters are selected), (2) its possibly counterintuitive results when the set of concept annotations is incomplete (and the lack thereof of an explanation for such unexpected results), and (3) the large amount of missing of details/explanations in some crucial components of this work that may lead to reproducibility issues. However,  I think my concerns may be addressed  by successfully addressing these questions:\n\n1. If you look at Figure 3 of [1], you can see an evaluation of interventions in the method I believe this work calls “CBM-AR” (called “Hard AR w/o side-channel” in [1]). In that figure, it is clear that the task accuracy of CBM-AR is significantly low when one uses a small number of concepts during training (as one would expect as the model’s bottleneck is extremely constrained). However, in the results shown in Table 1 of the Appendix, the same drastic change in accuracy is not observed for CBM-AR (with only a drop of ~8% in task accuracy when using 10% of all concept groups). This makes me a bit suspicious of all of the results in this table. Therefore, I was wondering the following:\n    1. When the paper says, for example, that 10% of concepts were used as annotations during training that means does that mean that the same group of 10% of training concepts were given for all samples of the training set **or** that, for any given sample, 10% of the concepts were selected at random and provided with annotations without regards of what concepts were provided to other training samples? That is, are concepts being subsampled at a \"global\" scale (where all samples are provided with the same subsample of concept annotations) or at a local scale (where each sample may have different concept annotations than other samples)? My guess is that it is the former (global scale subsampling) but I want to confirm this as it is unclear from the text.\n    2. If it is the former as I believe, then is the paper using the side channel version of “Hard AR” when evaluating CBM-AR? If so, this should be made explicitly clear somewhere as no details could be found.\n    3. **More importantly**, could you please clarify what is the intuition behind Coop-CBM having such a high accuracy when one uses a small number of training concepts (say 10%)? If the prediction used for the downstream task is that given by $g(f(x))$, then **I find this result extremely surprising**  given that Coop-CBM is still, just like its traditional CBM counterpart, severely constrained at the bottleneck when one uses a small number of concepts (and one would thus expect an even more severe drop in performance). One possible way I can see one can achieve this result is if the prediction used to compute the accuracy reported in Table 1 of the appendix is to use instead the immediate prediction made by $h$ (i.e., $h(f(x))$) as the output prediction of Coop-CBM. If that is the case, though, then this goes against how the model is originally defined in Section 3.2 and, more importantly, would mean that interventions are meaningless for this process as the prediction is bypassing the concept bottleneck. One convincing piece of evidence that your model is indeed working as expected (i.e., by predicting the label given by $g(f(x))$) is to show that it still positively reacts to interventions when the number of concepts is small (say 10%).\n2. How do the results reported in some of the experiments change if one selects the hyperparameters for competing methods using a simple hyperparameter search (say over 0.1, 1, 10) for the concept loss weight in Joint-CBMs, CEMs, and CBM-ARs?\n3. Could you please clarify the intervention concerns raised in the weaknesses? That is, how are interventions evaluated in Figure 1 and what is the relationship between these contributions and the rest of contributions? Are the contributions orthogonal to each other?\n4. In equation (5), it is still a bit unclear what $q_i$ is. Could you please clarify this?\n5. In section 5.1, why does 30% of the test set still have a spurious correlation? What is the rationale behind not removing the spurious correlation in its entirety to see how much the model truly depended on the shortcut for its predictions?\n\nRegarding suggestions for the presentation, the paper is very well-written (thank you!). Nevertheless, the following possible errors and typos could be addressed before a camera-ready version:\n\n- Typo in line 49: it should be “Mahinpei et al. [16] have shown” rather than “Mahinpei et al. [16] has shown”.\n- Typo in line 96: parenthesis is accidentally underscripted in $f(c|x)$.\n- Typo in line 127: missing “the” between “in” and “concept predictor”.\n- Typo in equation (3): I believe the last term should be $\\mathcal{L}_y(g(f(x)), y)$ rather than $\\mathcal{L}_y(g(c), y)$.\n- Typo in line 196: the sentence started in this line is a bit confusing (possibly missing a couple of words).\n- This came out earlier this year, so I think it is ok to miss the reference. However, for the sake of completeness, it is worth noting that recent work [2] does consider human error/uncertainty when performing interventions as opposed to the claim in lines 299-300. This is not a big deal as it happened within a few months before NeurIPS’ deadline, but it is worth noting for a future camera-ready version of the manuscript.\n- nit: ChatGPT and BARD are missing citations.\n- nit: “XAI” in line 26 is never explicitly defined.\n- The citation to “[1]” in the paper’s references of the main text is incomplete (the SENN paper reference).\n- I couldn’t find anything related to the “Oracle Impurity Score” in the original CEM paper as cited in the Appendix in line 160. A quick search for that term found that it is defined in a separate paper [3].\n- Section 3 could benefit from a diagram explaining the overall training process and architecture.\n- This is pure nitpicking on my end, but I am still unsure what the “coop” part of “Coop-CBM” stands for as it is never explained/defined.\n\n### References\n\n[1] Havasi et al. \"Addressing leakage in concept bottleneck models.\" *NeurIPS 2022*.\n\n[2] Collins et al. \"Human Uncertainty in Concept-Based AI Systems.\" *arXiv preprint arXiv:2303.12872* (2023).\n\n[3] Espinosa Zarlenga et al. \"Towards robust metrics for concept representation evaluation.\" AAAI 2023.", "limitations": "Although included in the appendix, this work provides a discussion of some of the limitations and potential impacts of its work. If possible, the manuscript could benefit from including such a discussion in the main body of the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Thank you for the very well-written and interesting paper. I thoroughly enjoy reading this work, and I believe it is both addressing an interesting problem while having the potential to inspire different methods elsewhere. After carefully reading the paper and the accompanying appendix, I believe that its main strengths are:\n\n1. Although the main idea in this paper is simple in nature, it is an interesting approach to alleviating some of the concerns that have been raised on the quality of learnt concept representations in CBMs, particularly regarding leakage.\n2. The evaluation across four real-world datasets, as well as variations of these datasets with spurious correlations for OOD evaluation, makes the experiment section of this paper very well-designed. In particular, I believe there is a lot of potential impactful work in better understanding these models in OOD scenarios.\n3. Given the need for interpretable architectures that are actually useful in practice, the core contributions of this work have the potential to be impactful if they stand thorough evaluation.\n4. The main idea is not entirely novel, as it shares similarities to ideas previously proposed in disentanglement learning and contrastive learning. Still, its application to this space, and its evaluation in this space, is novel and worthy of study.", "weaknesses": "While I believe the ideas in this work have future potential, I think there is significant room for improvement in how this work is presented and evaluated. Specifically, I believe the following are its main limitations:\n\n1.  I have some doubts about some of the claims made by the paper regarding the usability of its proposed method in concept-scarce (or concept-incomplete) settings. The results briefly discussed in passing in Section 5, and discussed in more detail in Section E of the Appendix, seem a bit suspicious to me without further clarification. This is because, intuitively,  one would not expect Coop-CBM to perform significantly better than CBM-AR or Joint-CBM when the number of training concepts is very limited (as the bottleneck's capacity will simply be too constrained!). Please see the question related to this point in the section below to see where my suspicion arises from. Please feel free to correct me if this is a misunderstanding on my end.\n2. Another serious weakness in this work, which I believe other reviewers may also point out, is the number of sheer hyperparameters the proposed loss has ($\\alpha, \\beta, \\gamma, \\lambda$ for the loss and even more for the intervention algorithm). This. together with the lack of an accompanying ablation study showing whether the results presented in this work would change if one fine-tunes these hyperparameters as well as those in competing methods, severely limits the robustness of the evaluation. The claim that “for fair comparison, the concept prediction weight (here $\\alpha$) in our baseline models is set to the same value”(lines 248-249) is in fact hard to take at face value. This is because it is unclear why all other hyperparameters are set to $0.01$ and whether this happens to be a value that greatly benefits the presented method. Therefore, more clarity on this end, including a potential ablation study and an appropriate hyperparameter selection for competing methods, would make the claims in this paper much stronger.\n3. A lot of the details on the intervention side of things are left undefined in the main body of the paper and moved to the appendix. Currently, most of the contributions this work makes on the side of interventions (e.g., the study of user uncertainty or the use of a new intervention policy for selecting concepts) seem completely orthogonal to the rest of the paper’s work/main motivation and therefore are hard to evaluate within the context of the same paper. Don’t get me wrong: I believe these are interesting results; it is just that they currently seem out of place with almost all details pushed to the appendix (see how much methodological content section F has) and with a very limited evaluation if this is a core contribution. For example, it is unclear whether Algorithm 1 is used to select concepts in the results of Figure 1 of the main paper. Based on the caption saying that concepts are randomly intervened, I believe this is not the case. If it is, then it is not a fair comparison to compare Coop-CBM with a non-trivial concept selection policy against baselines that select concepts at random. If it is not, then there is no evaluation of the proposed concept selection policy anywhere in the main body of the paper. Either way, further evaluation of the proposed policy or clarification of how it fits in with the rest of the work is required.\n4. I am a bit unsure about what the main takeaways of the experiments in Section 5.3 are that are not offered by previous experiments. Clarification on this end could help a lot.\n5. The codebase provided is incomplete (I could not find some of the baselines) and missing key comments or pseudo-documentation. I am not expecting production-quality documentation in any way but at least a README to guide readers through it would be very useful. \n6. Against traditional convention, and the authors' answer to the questionnaire question on error bars, Figure 1 and Tables 3, 4 and 5 are all missing error bars.\n7. Although in the questionnaire the authors indicate the compute resources and licenses are stated in the paper/supplementary material, I was unable to find either of these.\n8. Similarly, a lot of experimental/reproducibility details are missing from their codebase and their paper’s supplementary material, severely limiting the possibility of reproducing these results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for very detailed feedback. We have updated the readme.\n \n> 1. 1. Clarification on sparse concept experiment.\n\nTo provide clarity, when we mention \"10%\", it signifies that merely 10% of randomly chosen concepts were employed across all the images. Hence the same group of 10% of training concepts was given for all samples of the training set. This setup is designed to replicate scenarios involving sparse concept labels. This allows us to investigate the model's performance under conditions where concept annotations are limited, reflecting real-world scenarios of constrained concept availability.\n\n>1.2 Clarification on CBM-AR setup\n\nFrom Figure 3, Left image on CUB dataset, the red solid line Hard AR w/ side-channel is our baseline CBM-AR as the reviewer also alluded to. We shall make the setting of Hard AR more explicit in the camera-ready version.\n \n>1.3 Intuition behind Coop-CBM having such high accuracy in spare concept setting.\n\n* We would like to clarify that the task accuracy specified in the tables of the appendix and the main paper is g(f(x)) where the label prediction is bottlenecked on concepts only. We would like to build intuition using Figure 1 from the rebuttal pdf. Coop-CBM outperforms Joint-CBM due to its inherent design that introduces a \"collaboration\" between concept and supplemental task prediction at the same hierarchical level. In Coop-CBM, a task predictor stream is included alongside the concept predictor, enabling a synergistic relationship between the two. The concept features learned in Coop-CBM capture essential characteristics of the input data even when concept annotations are sparse. Through collaborative learning, Coop-CBM's concept predictor stream identifies and extracts meaningful relationships between the input features, labels and concepts. \n* In contrast, Joint-CBM despite being trained jointly follows a sequential approach where concepts are first predicted by the concept predictor stream and then utilized by the task predictor stream. This sequential approach can limit the capacity of the task predictor to capture subtle relationships in the data. Coop-CBM, by directly integrating the task predictor at the same hierarchical level, mitigates this limitation by combining both sets of information at an earlier stage, enabling better exploitation of the data.\n* We have plotted a graph for interventions for CUB in the rebuttal pdf.\n\n> How do the results reported in some of the experiments change if one selects the hyperparameters for competing methods [...]\n\nWe compare the classification accuracy for CUB and TIL dataset across all of the baselines and hyperparameters mentioned. Due to lack of space, we weren't able to add generalization results. We observe that CEM and coop-CBM are fairly stable to diff hyperparams.\n\n>Could you please clarify the intervention concerns raised in the weaknesses? That is, how are interventions evaluated in Figure 1 and[...]\n\nWe would like to clarify the reviewer's concerns by describing Figure 1 from the main paper. \n* L->R-The first two graphs show the importance of using human uncertainty(SCS) as a metric to select interventions as compared to concept uncertainty(CUS) and weightage(CWS). We show this in two ways:\n* First we plot for accuracy of the CBM model (as used by other baseline metrics [8,16]) when the annotator correctly intervenes on the concept. Intuitively, the accuracy should increase.\n* Second, consider that scenario is similar to the first except the annotator has changed. The annotator with ill intentions changes the concept values to incorrect concepts (instead of GT concepts). Now we plot all of the metrics and intuitively the accuracy would reduce.\n* L->R Last two graphs of Figure 1 compare the random interventions of different baseline models. Since SCS was present for CUB only, we consider random interventions for TIL and CUB. This also aligns with the evaluation method used in baseline (CBM-AR, CEM) papers.   We would also like to point out that this is a standard graph, also present in the baseline works. Hence we are not using any specific metric for coop-CBM, but we would like to emphasize that the concept representation of our model is enhanced due to COL. This enhancement is most evident in CUB. \n \n\n>In section 5.1, why does 30% of the test set still have a spurious correlation? What is the rationale behind not removing the spurious correlation in its entirety [...]\n\n5.1 Experiment was inspired by the invariant learning/shortcut learning literature [1]. Here we expose the model to an extremely strong correlation, so a model purely minimizing training error will tend to exploit the color. But models will fail at test time because the background color setting is changed. For a naive model, if we change the spurious correlation to 0%, we will not be able to evaluate the performance well since a naive model will fail completely (would be random). Hence the literature changes the direction of correlation$\\approx$1-strength during test time (usually between 20%-30%)\n\n>What the main takeaways of the experiments in Section 5.3 [...]\n\n5.1 and 5.2 experiments modify the input image space while 5.3 changes the concept space. In 5.3, we introduce direct spurious correlation between the image and concepts (as compared to the image-label correlation earlier).\n\n>Compute and error bars.\n\nWe have included compute in the rebuttal pdf. We shall include error bars from the missing tables in the camera-ready version. \n\ncoop-CBM $\\approx$ cooperative-CBM. We will clarify this in camera ready.\n\nWe thank the reviewer again for the insightful comments. We hope we covered all concerns. We are happy to answer more questions during the discussion period.\n\n[1] Arjovsky, Martin, et al. \"Invariant risk minimization.\" arXiv preprint arXiv:1907.02893 (2019).\n"}, {"review_id": "fXT7W3xerF", "reviewer": "Reviewer_VWed", "summary": "The paper proposes multiple improvements to current paradigm about learning, application and evaluation of concept bottleneck models (CBMs) for by-design interpretable classification. They propose to improve supervised learning of concepts in CBMs through a concept-orthogonality loss (COL) that encourages samples from different classes to have orthogonal feature representations used for concept prediction and samples with same concepts to have similar representation. Addtionally, to improve classification performance of the model, they propose to predict class labels additionally from this representation. In terms of usage of CBMs, the authors propose to incorporate supervisors confidence when making decisions about intervention. In terms of evauation they propose to also evaluate baselines under distribution shifts.", "questions": "Questions:\n\n(a) Loss weights: Among the loss weights to be set, $\\gamma$ and $\\lambda$ relating to COL seemed most crucial to me. The choice of $ \\lambda=0.05 $ seems a little arbitrary. Could you explain how did you arrive at this value whether it be through a heuristic or otherwise? Also, is the system behaviour sensitive to choice of $\\lambda$? How is the learning affected if its value changes? The experiment studying hyperparameters (appendix E) should also study role of $\\lambda$.\n\n(b) Statements surrounding COL: I got confused 2-3 times in understanding COL due to the statements made about it. Before going into the specific instances, I want to confirm that my understanding is accurate. Please correct me if I am mistaken in the following statements.\n1. $q$ is a single feature representation used to predict **all** the concept labels.\n2. In eq (5) $c_i, c_j$ denote concept labels for sample $i, j$ in a batch. These are binary vectors of size number of concepts. Consequently, COL encourages $q_i, q_j$ to be orthogonal if **ANY** concept label differs between samples $i, j$. COL also encourages $q_i, q_j$ to align if **ALL** concept labels among $i, j$ match.\n\nIf both these statements are correct, then these are my questions/concerns:\n1. Is differing of any concept label between sample $i, j$ equivalent to them belonging to different classes (line 188, Pg 5)? Although this is a completely reasonable proxy but is it possible that two samples from same class differ in just one concept label and thus COL regards them as dissimilar?\n2. Multiple times in line 182-186 in context of COL similarity loss, you state that it encourages similarity between feature representation for samples from \"same concept\". Isn't this an inaccurate phrasing? The accurate phrase should be for samples with \"same set of concept-labels\". \n3. In line 290-292, you also refer to COL loss as \"maximizing the inner product between the concept embeddings of different concepts\". Firstly, it should be 'minimizing'? Also there is only one concept embedding $q$, right? Reading this statement combined with ones in previous point was particularly confusing as it conveys a meaning of orthogonality differently imposed copmared to COL. Namely, one could encourage different feature maps in $q$ to be used in prediction of different concepts, even for a single sample. This would more accurately represent \"separating\" representations of different concepts. But that is not what COL proposes. \n4. Lastly, following the previous point, doesn't COL have some disadvantages in terms of imposing orthogonality for learning concepts? To be more specific, as an example consider two samples of similar classes. They are very likely to share many concepts (but not all). Isn't it sensible that the feature representation be similar as they share many concepts? The COL loss on the contrary encourages the feature representations to be completely dissimilar. Isn't it more sensible for dissimilarity loss to be applied **ONLY IF ALL** the concept labels present in two samples differ? If indeed this is the case, it needs to made very explicit.", "limitations": "The authors discuss the limitations in appendix. The content of the discussion covers some key points and is adequate for me.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths: \n(a) The paper is well written in large parts. The organization is fluid and comfortable to follow. The motivations for the proposals are also generally clear.\n\n(b) The experiments are solid and extensive with multiple different datasets, baselines and experimental settings.\n\n(c) The paper contains multiple novel contributions in terms of learning, usage or evaluation of CBMs.\n\nThe quality and clarity is high in most parts. Considering all the contributions as a whole, the paper is reasonably original and moderately significant in my opinion.", "weaknesses": "Weaknesses:\n(a) Each individual contribution is limited in either novelty or impact. For instance the concept-orthogonality loss (COL) is the most novel contribution but I wouldn't consider the improvement in concept/model accuracy (Tables 1-4) highly significant. On the other extreme while I'd consider evaluation of all baselines under distribution shifts useful for community, I won't regard it as highly original. This weakness sets an upper bound in terms of strength of this paper for me. That being said, considering all contributions together, this won't be a determining factor for me in terms of recommending acceptance or rejection.\n \n(b) The experiments studying impact of loss weights are lacking in certain ways. More details  about this in section for questions.  \n\n(c) Statements and reasoning around concept orthogonality-loss (COL) is sometimes confusing and potentially misleading to me. This is the most significant concern. Please refer to questions/concerns below for more details about this.\n\nOverall, I enjoyed reading the paper. Moreover, the paper as a whole should be useful for the ML community. However because of some major doubts regarding discussion around COL which is the most crucial idea of the paper, I can only favour the paper for borderline acceptance currently.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their feedback. \n> Loss weights: experiments around $\\lambda$\n\nWe experimented with different loss weights for $\\lambda$ in our experiments and the model+COL seemed to be fairly robust with different values of $\\lambda$. We have put those results in Table 5 of the rebuttal PDF on CUB and TIL datasets. While a fine-tuned value of $\\lambda$ might show good performance, we observe that regardless, the model is still able to beat the performances of other baselines. We observed that 0.05 set as a good tradeoff between performance and uncertainty across datasets.\n\n> Definition of COL confusion.\n\nUpon second look, we can see why the definition of COL could lead to confusion, we would like to make a small change to the equation and update it. Hopefully, this can alleviate some of the concerns. \n\n$d_1=\\sum_{\\substack{i,j\\in B,\\\\\\\\c_i^a = c_j^a\\\\\\\\a\\in A}}\\frac{q_i^T q_j}{||q_i|| ||q_j||};d_2=\\sum_{\\substack{i,j\\in B,\\\\\\\\ c_i^a\\neq c_j^a \\\\\\\\a\\in A}} \\frac{q_i^T q_j}{||q_i||||q_j||}$ where A is total concepts. \n\n\n\n> $c_i$ and $c_j$\n\nWith the clearer definition, we would rename $c_i$ and $c_j$ indeed represent the binary concept vectors across the batch $i$ and $j$, while $c_i^a$ and $c_j^a$ are individual concept values for a given concept across the batch.\n \n>Is differing of any concept label between sample $i$, $j$ equivalent to them belonging to different classes (line 188, Pg 5)? Although this is a completely reasonable proxy but is it possible that two samples from same class differ in just one concept label and thus COL regards them as dissimilar?\n\nWe understand that it might have been confusing if we look at Eqn 5, but looking at the revised version, we can see that it aligns with L188. Here the concept representation of two images from the same class is brought closer. It is also worth noting that datasets such as CUB and AwA2 have class-level concept representation.\n\n>Multiple times in line 182-186 in context of COL similarity loss, you state that it encourages similarity between feature representation for samples from \"same concept\". Isn't this an inaccurate phrasing? The accurate phrase should be for samples with \"same set of concept-labels\".\n\nThank you for pointing this out. We will rephrase it as \"concepts of the same class\".\n\n> L290-292 typo. This would more accurately represent \"separating\" representations of different concepts. But that is not what COL proposes.\n\nWe would like to apologize for the confusion. Indeed, L290-291 is a typo, it is minimizing not maximizing. Additionally, we would like to confirm that COL attempts to increase the separation between concepts by introducing the orthogonality constraint. \n\n> Isn't it more sensible for dissimilarity loss to be applied ONLY IF ALL the concept labels present in two samples differ? If indeed this is the case, it needs to made very explicit. \n\nFollowing your feedback, we have tried to make the COL definition more explicit mathematically. And dissimilarity loss, $d_2$ is applied in case of different image labels.\n\nWe thank the reviewer for constructive feedback. We ran the experiments and attempted to answer all of the concerns of the reviewer and would be happy to answer any further questions."}]

Author rebuttal:
Global rebuttal.

Meta-review: The reviewers and AC discussed this paper and read the authors’ response carefully. All the reviewers unanimously agreed to accept this paper. The authors are encouraged to address the concerns from the reviewers in the camera ready, such as

1.	Improve the discussion on related work

2.	Improve the writing of the paper

3.	Address the concern on the large number of hyperparameters.
Decision: Accept (poster)