Title: Auxiliary Losses for Learning Generalizable Concept-based Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Interpretability

concept bottleneck models

explainability
Abstract: The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts.  While beneficial, CBMs have been reported to often learn irrelevant concept representations that consecutively damage model performance. To overcome the performance trade-off, we propose a cooperative-Concept Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) to encourage the separation between the concept representations and to reduce the intra-concept distance. This paper presents extensive experiments on real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that our proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.

Auxiliary Losses for Learning Generalizable Concept-based Models

Abstract

The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts. While beneficial, CBMs have been reported to often learn irrelevant concept representations that consecutively damage model performance. To overcome the performance trade-off, we propose cooperative-Concept Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) to encourage the separation between the concept representations and to reduce the intra-concept distance. This paper presents extensive experiments on real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that our proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.

1 Introduction

The recent advances in deep learning have impacted various domains such as computer vision [21], natural language processing [53], and speech recognition [6]. Recently, the deployment of large models [8, 51] has led to various concerns regarding privacy and safety since machine learning models are often considered black boxes. With the increasing use of such deep learning models in daily human life and their wide deployment, it is essential to understand model behaviors beyond final prediction. Since neural networks are considered opaque decision-makers, inaccurate decisions by models in applications such as medicine [18] or autonomous driving [59] lead to catastrophe for humans. To understand the inner workings of such black-box neural networks, the field of XAI [32] has emerged in recent times. Concept Bottleneck Models (CBMs) [27] are a family of neural networks that enable human interpretable explanations.

Concept-based models introduce a bottleneck layer before the final prediction. This bottleneck layer consists of human-interpretable concept predictions. For example, in the context of images of animals, these concepts could be 'mane' in the case of a lion or 'black and white stripes' in the case of a zebra.

Concept bottleneck modelss (CBMs) map input images to interpretable concepts, which in turn are used to predict the label. The intermediary concept prediction allows a human supervisor to interpret and understand the concepts influencing the label prediction. In addition to explainability, CBMs offers an interesting paradigm that allows humans to interact with explanations. During inference, a supervisor can query for explanations for a corresponding label, and if it observes an incorrect concept-based explanation then the supervisor can provide feedback.

While CBMs present benefits with models' explainability, Mahinpei et al. [33] have shown that concept representations of CBM may result in information leakage that deteriorates predictive performance. It is also noted that CBM may not lead to semantically explainable concepts [34]. Such bottlenecks may result in ineffective predictions that could prevent the use of CBMs in the wild.

Along with model transparency, another challenge that modern neural networks face is robustness to distributional shifts [32]. Deep learning models fail to generalize in real-world applications where datasets are non-iid [17]. The absence of a comprehensive study examining the behavior of CBMs under distributional shifts is a significant limitation, potentially impeding their application in real-world scenarios.

In this work, we propose cooperative-CBM (coop-CBM) model aimed at addressing the performance gap between CBMs and standard black-box models. Coop-CBM uses an auxiliary loss that facilitates the learning of a rich and expressive concept representation for downstream task. To obtain orthogonal and disentangled concept representation, we also propose concept orthogonal loss (COL). COL can be applied during training for any concept-based model to improve their concept accuracy. Our main contributions are as follows:

We proposed a multi-task learning paradigm for Concept Bottleneck Models to introduce inductive bias in concept learning. Our proposed model coop-CBM improves the downstream task accuracy over black box standard models.

Using the concept orthogonal loss, we introduce orthogonality among concepts in the training of CBMs.

We perform an extensive evaluation of the generalisation capabilities of CBMs on three different distribution shifts.

We looked at using human uncertainty as a metric for interventions in CBMs during test-time.

2 Related Works

Concept-based Models Early concept-based models that involved the prediction of concepts prior to the classifier were widely used in few-shot learning settings [9, 62]. Other works propose to predict human-specified concepts with statistical modeling [28, 29]. Unsupervised concept learning methods use a concept encoder to extract the concepts and relevance network for final predictions [4, 43]. Although these methods are useful in the absence of pre-defined concepts, they do not enable effective interventions. Concept whitening [11] was introduced as a method to plug an intermediate layer in place of the batch normalization layer of a Convolutional Neural Network (CNN) to assist the model in concept extraction. CBM [27] extends the idea by decomposing the task into two stages: concept prediction through a neural network from inputs, and then target prediction from the concepts. Many works have proposed models built on CBMs to either improve the downstream task accuracy [44, 60, 26] or mitigate the concept leakage [33, 20]. There has been a line of work extending CBMs to real-world applications such as medical imaging [58, 13], autonomous driving [44] and reinforcement learning [14] CBMs require annotated concepts which poses a challenge for their applications to large-scale image datasets. Yuksekgonul et al. [58] propose using concept activation vectors [25] and Oikarinen et al. [38] used multimodal models such as CLIP [40] to annotate concepts for CBMs. Although these either require concept bank or suffer from pretrained model's biases [58].

Alternative losses The training of CBM and its variants typically involves the use of Cross Entropy (CE) loss. Several variants of the CE loss have been explored in the past to improve the discriminative power of learned feature representations of data [19, 46, 56, 39]. Ranasinghe et al. [41], Vorontsov et al. [54] introduce the use of orthogonality in feature space to encourage inter-class separation and intra-class clustering. Our work builds upon [41] by introducing orthogonality constraints in the concept feature space.

3 Method

3.1 Background

Consider a standard supervised learning setting for a classification task, where models are trained on a dataset D = { x i , y i } N i =1 with N data samples. Standard models aim to predict the true distribution p M ( y | x ) from an input x . Although such a setting has been proven effective on vision benchmarks, users are unaware of the detailed inner workings of the model. Therefore, CBMs introduces intermediate prediction of human-understandable concepts before the model prediction.

In the supervised concept-based model setting, the dataset uses additional labeled concept vectors c i ∈ { 0 , 1 } a where each element indicates the presence of one of a high-level concepts. This allows supervised concept learning in addition to target learning. Following a simplistic causal graph for data generation, y → c → x , CBMs consist of two models. The first model f X → C maps the input image x to concepts c , while the second model g C → Y maps the concepts c to the label y .

CBMscan be categorized by their method of training g C → Y from the obtained concept representations f ( c | x ) . This could be done in the following manner: jointly , where both f X → C and g C → Y are trained simultaneously end-to-end, sequentially , where f X → C is trained first, after which g C → Y is trained using p f ( c | x ) representations, and finally independently , where f X → C and g C → Y are trained individually and then combined.

Interventions Interventions are a core motivator of CBMs. The bottleneck model allows for interventions by editing the concept predictions. Since CBMs consider correcting the predicted concepts through interventions during test-time, the corrected concepts are not back-propagated through f X → C and g C → Y . During test-time intervention the predicted concepts can be modified by a supervisor to their ground truth values, leading to 'adjusted' concepts prediction. We represent the predicted concepts as ˆ c = p f ( c | x ) and the modified concepts as ¯ c . We consider test-time interventions as an important aspect of explainable models in safety-critical applications. We hypothesize that model-supervisor interaction must lead to the development of a symbiotic relationship between the model and the expert. Here, the expert learns about the potential causation between a concept and its corresponding label, and the model learns true concept values from the expert. We attempt to shine a light on these test-time interventions by simulating realistic scenarios by introducing human uncertainty.

3.2 Coop-CBM

While CBMs provide concept explanations behind a prediction, it has been observed that this can come at the expense of lower model accuracy compared to black-box standard models [33]. In this work, we propose a concept-based architecture, coopCBM to improve the performance of CBMs on downstream classification tasks.

Motivation The different training paradigms in CBMs introduced by Koh et al. [27] give rise to differences in their concept representations, p f ( c | x ) . Koh et al. [27] reports that joint CBMs have the highest task accuracy among the different CBM train-

ing procedures albeit still lower than standard models. Intuitively, this suggests that joint CBMs, which train both concept predictor and task predictor simultaneously, are able to encode the information about the task label y into concept labels c better than sequential and independent CBMs. In the case of joint CBMs, backpropagation of task loss through the concept predictor aids the overall model in giving more accurate predictions. In this work, we aim to leverage such 'soft' information

about the task to improve accuracy. Coop-CBM aims to leverage soft label information in concept predictors to better align the concept predictions to the corresponding label.

Model Coop-CBM introduces a multi-task setting before the final prediction. Along with predicting the concepts c , we introduce the prediction of task labels in the concept predictor. This allows the model to learn relevant signals and inductive biases of downstream tasks in the concept learning phase. In essence, we now have model f X → C that predicts concepts c from input x and new model h X → Y that predicts label y from input x . This enables the model to learn relevant knowledge about the task that could be absent in the bottleneck concepts c . Although this setting makes the model interpretable, since corresponding concepts to a label can be obtained, one loses the causal x → c → y property. Additionally, it does not allow test-time interventions, which is a key application of concept-based models that facilitates human-model interactions. Therefore, to maintain the original properties of CBM, coop-CBM uses a label predictor g which takes the predicted concepts c from f X → C as input and gives the final label y as output. Hence to avoid confusion, we call the label prediction from h immediate label , y ′ , and the final task label, y . It must be noted that f and h share all but the last linear layer. Therefore the concept predictor f and g parameters θ , ϕ are trained in the following manner:

Therefore coop-CBM is trained using a linear combination of three different CE losses: L C as concept loss, L Y ′ as immediate label loss and L Y as task prediction loss.

In summary, we argue that the introduction of an immediate label introduces the concept predictor to learn meaningful information about the task while still being interpretable. The h X → Y ′ model intuitively acts as a regularizer for meaningful concept prediction.

Mutual information perspective We hypothesize that by using coop-CBM, the concept predictor acquires better knowledge about y . In particular, this can be beneficial when fine-grained concept annotations are not available. We, therefore, suspect that the mutual information (MI) between the input image, concept representations, and the label becomes richer and more expressive as compared to CBM [27]. One way to quantify this is by visualising the MI planes throughout the training, similar to Zarlenga et al. [60].

3.3 Concept Orthogonal Loss

Following the current CBM literature, we use cross-entropy loss to train each of the models, f X → C , h X → Y ′ and g C → Y in coop-CBM. In model f , each concept is learned via independent and separate classifiers. Given their binary representation, it is intuitive to improve the embedding space of concepts by increasing separability. To do so, we introduce the concept orthogonal loss (COL). By incorporating COL into the training process, we aim to enhance the overall separability of the concept embeddings, leading to improved performance and interpretability of the coop-CBM model.

Motivation Due to the variations in training strategies employed in different CBM models, the resulting concept representations can exhibit varying levels of accuracy. The concept accuracy refers to how effectively the learned concept representations align with the ground truth or human-defined concepts. Coop-CBM was concerned with the predictive performance of the task, but here we focus on the concept label accuracy. Higher concept label accuracy signifies improved interpretability. As observed by Koh et al. [27], the concept accuracy of joint CBMs models is lower than other variants because the concepts learned are not completely independent of each other, also called leakage by [33]. Hence, increasing the inter-concept distance and intra-concept clustering throughout the concept vector for the entire dataset can allow the model to learn beyond co-dependent concept representations.

COL In addition to CE loss for learning concepts, we introduce novel concept orthogonal loss by conditioning orthogonality constraints on concept feature space. The disadvantage of CE loss is that it does not set a specific distance or separation between different concept representations in the feature space. Consider the CE loss for each concept prediction:

Using this traditional CE loss for each concept, in Equation 4 we are essentially minimizing the difference between the predicted probability distribution and the true probability distribution of the binary concepts. The model, f c i attempts to learn probability distribution for when a concept is active or inactive respectively. CE does not explicitly enforce separation between concepts.

With concept orthogonal loss (COL), we enforce the separation in the latent representation of the concepts. Our aim with COL is to group similar features together while ensuring that features belonging to different concept classes do not overlap with each other. COL L COL enforces interconcept orthogonality and intra-concept clustering. We define inter-concept separation and intraconcept similarity as d 1 and d 2 respectively. We enforce orthogonality constrain via cosine similarity. We define the L COL loss in the shared last layer, q of coop-CBM before concept and auxiliary label predictions. We enforce COL constraints within each batch, B .

̸

where || . || denotes the Frobenius norm.

Using the cosine distances, d 1 and d 2 , we simultaneously aim to increase the distance between different latent concept representations and decrease the distance between representations from the same concept. We can introduce a hyperparameter, λ to accordingly give weightage to either d 1 or d 2 . The similarity loss d 1 between the feature representation of two samples corresponding to the same concept aims to push the d 1 towards 1 which means that the feature representations of same concept should be as similar as possible. As for the dissimilarity loss, the goal is to push the loss towards 0 , which enforces that the feature representations of different class samples should be as dissimilar as possible. Therefore we consider the absolute value of d 2 .

It is important to note that CE loss is applied to each concept binary classification task, which measures the difference between the predicted class probabilities and the true labels. The introduction of COL encourages the network to learn features that are both discriminative and non-redundant among concepts at an intermediary network level. By combining the COL and CE losses, the network is trained to learn discriminative that separate each concept and useful features for classifying when a concept is active. A benefit of COL is it can be universally any concept-based model to encourage orthogonality between different concepts.

In this section, we introduce two auxiliary losses, one to improve the task accuracy using multi-task setting and the other to improve concept representation in latent space, leading to improved concept accuracy. The final loss is a linear combination ( α, β, γ are hyperparameters for weighting in Equation 7) of concept and task losses along with immediate and concept orthogonal losses.

3.4 Interventions

Koh et al. [27] demonstrated the potential of CBMs for facilitating human-model interaction and improving task performance during inference. But it can be time-consuming and costly to have domain experts go over each concept, hence some of the recent and concurrent works proposed to use uncertainty as a metric to select interventions.

We propose a lightweight approach that strategizes the supervisor-model interaction. Our method is intuitive and considers three aspects of intervention:

Uncertainty of concept prediction - CUS represents the confidence of the model to predict latent concepts, f X → C .

Supervisor confidence for concept correction - SCS represents the reliance on the supervisor to intervene and subsequently correct the concepts accurately.

Importance of concept for label prediction - CWS denotes the significance of each concept for the subsequent downstream task.

Chauhan et al. [10] propose to optimize interventions over a small validation set using CUS. In comparison, we consider access to the validation set unrealistic. Shin et al. [49], Sheth et al. [48] evaluated interventions more comprehensively and studies the behavior of CBMs during inference by selecting CUS and CWS metrics. We additionally take into account a supervisor's confidence in domain knowledge, SCS and their expertise in correcting the concepts. Concurrent work [12] also looked at human uncertainty for CBMs in depth. Unlike previous works that evaluate test-time interventions on the test splits of respective datasets, we also analyze test-time interventions in OOD setting in the Appendix E.

4 Experiments

For our evaluation we consider several image classification benchmark datasets. Our work performs an in-depth empirical analysis of the effectiveness of concept-based models in the presence of different distributional shifts simulating real-world scenarios where data is diverse.

Baselines We consider the models proposed by Koh et al. [27] as our baseline. Additionally, we compare our performance with recent concept-based models that are built on CBM [60, 20]. Due to biases introduced during automatic concept acquisition as mentioned by the authors of Yuksekgonul et al. [58], we consider it to be an unfair baseline to compare generalization properties. They also vary in the number of concepts considered which can also damage the performance and are limited by either the

presence of concept bank or application (CLIP will fail to generate concepts for TIL dataset), making an unfair comparison.

Datasets We use Caltech-UCSD Birds-200-2011 (CUB) [55] dataset for the task of bird identification. Every dataset image contains 312 binary (eg: beak color, wing color) concepts. We additionally use Animals with Attributes 2 (AwA2) [57] dataset for the task of animal classification. The dataset contains 85 binary concepts. We use all of the subsets of the Tumor-Infiltrating Lymphocytes (TIL) [42] dataset for cancer cell classification.

5 Results and Analysis

The primary metric used for the downstream classification task is accuracy. We use the same metric to evaluate the effectiveness of the intervention. We first evaluate the different model performances on the test data split of respective datasets and report the task accuracy g C → Y for coop-CBM model variants.

Coop-CBM improves task accuracy The evaluation of the performance of a model is based on the final prediction accuracy. In Table 1, we compare the performance of coop-CBM against other baseline models. We first observe that CBMs experienced a significant drop in performance compared

to the standard model that did not use concepts. Our proposed model, coop-CBM with immediate label prediction achieves state of art accuracy and statistically significant results on every dataset. We have observed a significant improvement in the performance of the CUB ( +1 . 8% increase from the standard model) and TIL( +3 . 1% increase from the standard model) datasets. This finding is important as it suggests that machine learning models can be designed to overcome the high accuracy vs interpretability tradeoff. Our performance can be further boosted by introducing orthogonality among different concepts. It must be noted that CUB is a fairly densely annotated dataset, which might not always be realistic, hence we also benchmark our model by training on a fraction of concept sets. We also observe a similar trend in results in concept-scarce settings (see Appendix D.2). This suggests that our method is robust to concept selection, which can be beneficial in scenarios where the number of available concepts is limited or expensive to obtain.

COL improves concept accuracy Previously in Table 1, we observed that adding concept orthogonal loss to coop-CBM improved its downstream accuracy, in Table 2, we study the impact of adding COL to baseline concept models. Our experiments show that adding COL improves the concept accuracy by a significant margin, especially in joint CBM, CEM, and CBM-AR settings. A known pitfall of CBMs is, concept leakage [33] could be potentially prevented by increasing the separation between their concept representations. By maximizing the inner product between the concept embeddings of different concepts, we can ensure that each concept is represented in a separate and distinct direction in the embed-

ding space. This helps in preventing the models from relying on irrelevant concepts. Further to intuitively understand the differences in the concept representation of our model, we compute the histogram for the predicted concept logits. From Figure 3 we see that Coop-CBM+COL minimizes the concept loss better (with help from the auxiliary loss which aids representation learning), which results in clearer separation of logits.

Clipping concept values to avoid concept leakage Further, we employ clipping of concept prediction proposed by Mahinpei et al. [33] to further mitigate information leakage in 3 throught two experiments. For the first experiment, we trained the model by clipping the predicted concept values to 'hard' labels. Second, we trained the model as we have described earlier in the paper (using soft labels) and evaluated the test set by clipping to 'hard' labels. From the above experiments, we conclude that the

model is able to learn a good representation of the concepts without necessarily leaking information.

Accounting for Human uncertainty for interventions As discussed earlier, higher concept accuracy also improves the test-time interventions as seen in Figure 2. While other works used concept weights and uncertainty as metrics to select the interventions, our work introduces a more realistic setting by introducing human uncertainty additionally. The previous works do not account for human error or certainty. Albeit human uncertainty is difficult to quantify since it is often subjective, we use the concept visibility data in the CUB dataset to quantify the confidence score, SCS. The

CUB

CUB

TIL

CUB

95

80

60

Random(CBM)

SOS

CWS

CUS,SCS,CWS

60

Seq-CBM

Joint-CBM

Coop-CBM

CEM

95

Joint-CBM

Seq-CBM

Coop-CBM

Accurac

2

90

CUS

Standard

90

Standard

40

55

85

85

Random(CBM)

CUS,SCS,CWS

CWS

SCS

20

50

80

80

0

100

200

300

0

100

200

300

0

50

100

150

0

100

200

300

Singleinterventions

Incorrectinterventions

Singleinterventions

Singleinterventions

annotators rank visibility from 1 -4 , 1 being 'not visible' and 4 being 'definitely visible'. Figure 2 presents an accuracy curve with an increasing number of interventions on the vanilla CBM model during test-time for proper and fair comparison to [49]. It suggests that considering SCS leads to the most meaningful interventions. On intervention comparison between different concept-based models, we observe that coop-CBM leads to significant interventions that improve downstream performance, especially in TIL dataset where both image descriptive and non-descriptive concepts are present.

In summary, the coop-CBM model and the addition of the concept orthogonal loss help to improve both task and concept accuracy without sacrificing interpretability, which was a common tradeoff in previous methods. This result demonstrates the potential for concept-based models to be more effective in human-AI interactions, especially in domains where expert intervention and interpretability are critical such as healthcare.

5.1 Performance under distribution shifts

5.1.1 Background spurious correlations

Shortcut-based biases [17] exist in many datasets, where deep learning can easily learn spurious features. In the presence of shortcuts, the model can learn to use spurious features to approximate the true distribution of the labels, as opposed to learning core features. It can be of particular interest to evaluate the performance of concept-based models in the presence of shortcuts. Furthermore, using explainable concepts to facilitate human-model interaction could help reduce the impact of these biases.

The shortcut we consider here is a spurious correlation to the background color in CUB dataset. Loosely following the experimental setup of [5], we correlate the background of a species of bird to its corresponding label. For the dataset, we segment the bird images and add a colored background to all of the images.

Each class here is correlated to a randomly generated color background with a probability of 80% for the train set. The in-domain test set contains images with similar color background probability as the train set while the correlation in the out-domain test set is reduced to 30% . We also consider the hair color-based shortcut induced in Large-scale CelebFaces Attributes (CelebA) dataset where we focus on gender classification between males and females. We aggregate and then construct a modified version of CelebA that learns the shortcut of blonde hair color with women similar to [3].

In Table 4 we report the in-domain and out-domain accuracies of our baseline models and proposed coop-CBM for CUB and CelebA datasets. Our results show the robustness of coop-CBM and COL to background spurious correlations achieving state-of-art results among the concept-based models. We observe an interesting trend for the CUB dataset. The concept-based models may have lower accuracy standard model in the in-domain setting but in the out-domain setting, all of the baseline models including ours have higher accuracy than the standard model. While we do not observe a similar definitive trend in CelebA dataset, it is evident that most of the models, CEM, joint CBM and coop-CBM are superior when the test data contains more images of men with blonde hair. This experiment suggests that the concept-based models are able to generalize better to unseen data outside the training distribution, which can be attributed to their ability to learn more invariant features through the explicit conditioning and learning of concepts. This is a promising result as it indicates that concept-based models, particularly coop-CBM may be more robust to extreme distribution shifts.

In Appendix E, we evaluate the influence of interventions in out-domain settings for the biased CUB dataset. Realistically, the user may fine-tune the model to distributional shift after deployment to improve the predictive performance, but in the circumstances where the labeled shifted dataset is absent, we suspect that interventions could greatly help if they are cheaper to obtain. In such cases, we especially argue that SCS score benefits the intervention selection.

5.1.2 Image corruptions

Distributional shift can occur due to various factors such as changes in the data collection process, changes in the environmental conditions under which the data is collected, or even changes in the underlying population that the data represents. The Corruptions dataset [22, 35] is a collection of image corruptions designed to evaluate the robustness of computer vision models. Some of the corruptions are realistic OOD settings such as snow, while others could be less likely in nature such as impulse noise. We introduce 7 such corruptions (Gaussian Noise, Blur, Zoom Blur, Snow, Fog, Brightness and Contrast) onto CUB, AwA2 and TIL datasets. We report detailed results on CUB and average accuracy for AwA and TIL datasets in Table 5.

Based on our evaluation of distributional shifts in Table 5, we found that incorporating an auxiliary loss in the form of a multi-task setting can help CBMs achieve competitive downstream accuracy overall. We notice that the standard black box models although do perform better in the presence of 'contrast' corruption in CUB dataset. Although coop-CBM outperforms other explanation models. In general, it is interesting to note that coop-CBM has better generalization performance in the presence of spurious correlations than in the presence of corruption. This may be because spurious correlations are introduced as a shortcut within the training data, and concept-based models are designed to learn invariant features that are robust to such shortcuts. Regardless, coop-CBM's superior generalization property across different corruptions suggests that the model is able to effectively filter out irrelevant information/noise in the data.

5.1.3 Noise concept correlation

In Section 5.1.1 and Section 5.1.2, we conducted experiments concerned with distribution shifts in image space, in this section, we introduce the evaluation of CBMs by simulating distribution shifts in the concept space. To investigate the potential risks of spurious correlations in concept models, we introduced Gaussian noise to the binary concepts. By altering the standard deviation ( σ ) of the Gaussian noise, we effectively correlated the shortcut (here noise level) with the image through the concept. To simulate a more realistic setting, instead of adding distinct noise σ for each class species, we aggregate random groups of species and add same σ to them. In our experiment for the CUB dataset, we add 10 different levels of noise (simulated by σ ) to groups of 20 species labels

(200 total classes). For AwA2, we create groups of 10 classes. This approach allowed us to simulate the possibility of introducing unintended correlations between the concepts and the images. By studying the effects of these correlations on the performance of the concept models, we gain insights into the robustness and reliability of the models in handling contaminated concepts. We observe that by introducing a separation between different concepts through COL, our model performs significantly better than the rest of the baselines.

6 Future work and Limitations

In this work, we introduced coop-CBM, a novel concept-driven method to balance AI model interpretability and accuracy. We utilized the Concept Orthogonal Loss (COL) to improve concept learning and applied coop-CBM to various datasets, achieving better generalization, robustness to spurious correlations, and improved accuracy-interpretability trade-offs.

However, our approach has limitations. It relies on labeled concept vectors, which can be challenging in domains with limited annotations and face biases in concept annotation methods. A potential future work could be to extend it to methods that do not assume concept label [58, 38]. Further, we used accuracy as a metric to evaluate concept leakage, in the future, it would be interesting to explore other metrics beyond the accuracy of concept prediction. A future extension of COL could be to evaluate which concepts should be explicitly orthogonalized. We recognize that our model has a few hyperparameters to be optimized. Furthermore, our model assumes that learned concepts align closely with human notions, but this alignment isn't always perfect, affecting comprehensibility. Future research could improve the accuracy of concept-based models by providing meaningful explanations and incorporating additional evaluation metrics. Another potential direction could be to assess the mutual information and therefore establish theoretical grounding to describe the superior performance of coop-CBM.

7 Discussion and Conclusion

In this work we proposed two significant contributions to the paradigm of concept-based models. First, we introduced a multi-task model that predicts an intermediary task label along with concept prediction. This is particularly helpful when dense and relevant concept annotation is absent, such as in TIL dataset. Second, we introduced orthogonality constrain in the concept representation space during training via concept orthogonal loss. This loss increases inter-concept separation and decreases intra-concept distance. For both of our proposed methods, we perform extensive experiments on diverse datasets and different distributional shifts. We observe that the bottleneck layer before the final prediction enables concept-based models to exhibit robustness to spurious correlations in the background. Coop-CBM along with COL achieves state-of-art performance for both task accuracy and concept accuracy. Our work indicates that coop-CBM and COL have a strong ability to adapt and generalize well across diverse datasets and real-world scenarios.

References

Julius Adebayo, Justin Gilmer, Ian J. Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. ArXiv , abs/1810.03307, 2018.

Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems , 31, 2018.

Mohammed Adnan, Yani Ioannou, Chuan-Yung Tsai, Angus Galloway, HR Tizhoosh, and Graham W Taylor. Monitoring shortcut learning using mutual information. arXiv preprint arXiv:2206.13034 , 2022.

David Alvarez-Melis. Self-explaining neural networks. 2018.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP) , pages 4945-4949. IEEE, 2016.

Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN , 2016.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. arXiv preprint arXiv:2007.07375 , 2020.

Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430 , 2022.

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nat. Mach. Intell. , 2:772-782, 2020.

Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , pages 869-889, 2023.

Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Advances in Neural Information Processing Systems , 35:18157-18167, 2022.

Devleena Das, Sonia Chernova, and Been Kim. State2explanation: Concept-based explanations to benefit agent learning and user understanding. arXiv preprint arXiv:2309.12482 , 2023.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning, 2015. URL https://arxiv.org/abs/1506.02142 .

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence , 2(11):665-673, 2020.

Marzyeh Ghassemi, Tristan Naumann, Peter Schulam, Andrew L Beam, Irene Y Chen, and Rajesh Ranganath. A review of challenges and opportunities in machine learning for health. AMIA Summits on Translational Science Proceedings , 2020:191, 2020.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , volume 2, pages 1735-1742. IEEE, 2006.

Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems , 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 , 2019.

Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference on learning representations , 2017.

Mark Ibrahim, Melissa Louie, Ceena Modarres, and John William Paisley. Global explanations of neural networks: Mapping the landscape of predictions. Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society , 2019.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574 , 2023.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. ArXiv , abs/2007.04612, 2020.

Neeraj Kumar, Alexander C Berg, Peter N Belhumeur, and Shree K Nayar. Attribute and simile classifiers for face verification. In 2009 IEEE 12th international conference on computer vision , pages 365-372. IEEE, 2009.

Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition , pages 951-958. IEEE, 2009.

Honglak Lee, Alexis Battle, Rajat Raina, and Andrew Ng. Efficient sparse coding algorithms. Advances in neural information processing systems , 19, 2006.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Luca Longo, Randy Goebel, Freddy Lecue, Peter Kieseberg, and Andreas Holzinger. Explainable artificial intelligence: Concepts, applications, research challenges and visions. In Machine Learning and Knowledge Extraction: 4th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2020, Dublin, Ireland, August 25-28, 2020, Proceedings , pages 1-16. Springer, 2020.

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. ArXiv , abs/2106.13314, 2021.

Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? ArXiv , abs/2105.04289, 2021.

Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv:1907.07484 , 2019.

Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and understanding deep neural networks. ArXiv , abs/1706.07979, 2018.

Yasunobu Nohara, Koutarou Matsumoto, Hidehisa Soejima, and Naoki Nakashima. Explanation of machine learning models using improved shapley additive explanation. In Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics , pages 546-546, 2019.

Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129 , 2023.

Ce Qi and Fei Su. Contrastive-center loss for deep neural networks. In 2017 IEEE international conference on image processing (ICIP) , pages 2851-2855. IEEE, 2017.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Kanchana Ranasinghe, Muzammal Naseer, Munawar Hayat, Salman Khan, and Fahad Shahbaz Khan. Orthogonal projection loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12333-12343, 2021.

J. Saltz, Rajarsi R. Gupta, Le Hou, Tahsin M. Kurç, Pankaj Kumar Singh, Vu Nguyen, Dimitris Samaras, Kenneth R Shroyer, Tianhao Zhao, Rebecca C. Batiste, John S. Van Arnam, Ilya Shmulevich, Arvind U. K. Rao, Alexander J. Lazar, Ashish Sharma, and Vésteinn Thorsson. Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell reports , 23 1:181-193.e7, 2018.

Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10286-10295, 2022.

Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access , 10:41758-41765, 2022.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 , 2013.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision , 128:336-359, 2017.

Ivaxi Sheth, Aamer Abdul Rahman, Laya Rafiee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou. Learning from uncertain concepts via test time interventions. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022 , 2022.

Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302.14260 , 2023.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2818-2826, 2016.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104.07167 , 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning , pages 3570-3578. PMLR, 2017.

Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge J. Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.

Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14 , pages 499-515. Springer, 2016.

Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence , 41:2251-2265, 2019.

Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480 , 2022.

Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision-based autonomous driving systems: Review and challenges. International Journal of Computer Vision , 130(10):2425-2452, 2022.

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056 , 2022.

Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv preprint arXiv:2301.10367 , 2023.

Yaohui Zhu, Weiqing Min, and Shuqiang Jiang. Attribute-guided feature learning for few-shot image recognition. IEEE Transactions on Multimedia , 23:1200-1209, 2020.

A More Related Works

Exlainability Post-hoc explanations aim to provide insight into why a particular prediction or decision was made by the model. These may be in the form of heatmaps, rule sets, or feature importance scores. Global explanations [24] aim to learn the overall generic features in order to explain black-box models by the use of explanators. These explanators are often simple Machine Learning (ML) models. Local explanation methods [1] exhibit explainability by exploring the intrinsic workings of the neural network. This can be executed by propagating layer-wise feature relevance [7]. Selvaraju et al. [47] propose a gradient-based saliency mapping technique that perturbs inputs by injecting noise. Deconstructing the nonlinear model to simpler sub-functions [36] was another proposed method to interpret model's predictions. Nohara et al. [37] propose a difference-to-reference approach to feature importance estimation. Popularly, saliency heatmaps for feature importance visualization have been used [2]. Post hoc explanations although do not address the fundamental issue of model transparency, as they are generated externally to the model and may not reflect the true reasoning of the model's internal mechanisms which can be addressed by concept-based models like ours.

Orthogonality Orthogonality in neural networks has been extensively studied in the literature, with various approaches proposed to enhance model performance and interpretability. X and Y have explored orthogonal regularization techniques, which impose orthogonality constraints on weight matrices or feature representations during training. These methods have been shown to improve generalization and reduce overfitting. Saxe et al. [45] have introduced orthogonal initialization methods, which initialize weight matrices using orthogonal transformations. This initialization strategy has been found to aid training convergence and stabilize the learning dynamics of deep networks. Trockman and Kolter [52] have proposed techniques to enforce orthogonality constraints specifically in convolutional filters of convolutional neural networks (CNNs). By imposing orthogonality on the filters, these methods enhance the representational power and robustness of CNNs. In our work, we introduce orthogonality to concept feature space.

B Experiments

The details regarding datasets and hyperparameters are mentioned below. Our codebase is available at https://github.com/ivaxi0s/coop-cbm and is built upon from open source repos [27, 41].

B.1 Dataset Details

CUB The CUB-200-2011 dataset is a collection of 11788 images that are used for fine-grained visual categorization. There are 312 concept attributes that are binarised following the Koh et al. [27] work. While most existing studies use a subset of these concepts, we have chosen to use the entire concept bank in our models and baselines, addressing the fairness issue of subgrouping concepts as highlighted by [49]. The primary task is to classify 200 different species of birds. We also utilize the meta-data of this dataset to obtain human uncertainty.

TIL [42] dataset contains Tumor-Infiltrating Lymphocytes Maps from TCGA HE Whole Slide Pathology Images. The dataset contains tumor maps from the most common cancer tumor types. We use all 13 subsets of the TCGA dataset, therefore constituting 13 cancer types. Although the popular task for such a dataset is necrosis classification, we modify the task to be a classification for different (here 13) tumor types. The advantage of medical images is that their meta-data is readily available from diagnosis. The metadata includes information such as the origin of the tumor, age, gender, and size of tumor cells, which are converted to concepts. We follow [48] for the dataset pre-processing. We wnd up with 185 binary concepts following the pre-processing.

AwA2 Animals with Attributes dataset contains over 37,000 images of 50 different animal species, each labeled with 85 distinctive attributes. These attributes can include various characteristics such as color, shape, or behavior, providing a rich source of information for concept bank.

B.2 Experimental setup

For CUB[55] dataset, we trained using 128 batch size with SGD optimizer with 0.9 momentum and learning rate of 10 -2 . The feature extractor was InceptionV3[50] as a concept encoder model.

For AwA2[57] dataset, we trained using 128 batch size with Adam optimizer with 0.9 momentum and learning rate of 5 × 10 -3 . The feature extractor was VIT[15] as a concept encoder model.

For the medical dataset, we create the task of classification divided on the basis of cancer types for TIL dataset[42]. We generate a concept attributes from the meta-data. We use a traditional 70% -10% -20% random split for training, validation, and testing datasets. Additionally, we trained using 64 batch size with SGD optimizer with 0.9 momentum and learning rate of 10 -2 . The feature extractor was InceptionV3[50] as a concept encoder model.

For m-CelebA[31] dataset, we train using 64 batch size with Adam optimizer with 0.9 momentum and learning rate of 5 × 10 -3 for 500 epochs. The feature extractor was InceptionV3[50] as a concept encoder model.

Across all of the models for tasks, we use weight decay of factor of 5 × 10 -5 and scale the learning rate by a factor of 0.1 if no improvement has been seen in validation loss for the last 15 epochs during training. We also train using an early stopping mechanism i.e. if the validation loss does not improve for 200 epochs, we stop training.

In this paper, we introduced two auxiliary losses, one to improve the task accuracy using multi-task settings and the other to improve concept representation in latent space, leading to improved concept accuracy. The final loss is a linear combination ( α, β, γ are hyperparameters for weighting in Equation 7) of concept and task losses along with immediate and concept orthogonal losses.

For the hyper-parameters of Equations 7, we use α and β values to 0 . 01 for all of the experiments.

CEMhyperparameters We would like to point out that we used the same concept α weightage hyperparameter for each of the model. In literature, for all [27, 60, 20] of the methods used the same concept weightage and we follow the same convention. Looking at our selected hyperparameter, the most divergent value is for CEM [60]. The original CEM paper selected the α = 5 . Since we could not find an ablation study around this hyperparameter in the original paper, we continued to use the same value as rest of the models. We acknowledge that our results might be skewed due to this reason.

B.3 Resources used

Our codebase was built upon the open codebase of [27]. We trained on Linux-based clusters mainly on V100 GPUs and partially on A100 GPU. The following table includes compute timing for each epoch for every baseline.

C Further experiments on COL

C.1 Disentanglement

Disentangled features allow for a more intuitive understanding of the underlying factors that influence the concepts. Several research efforts have explored the benefits and applications of disentangled representations [23]. One of the notable characteristics of Concept Orthogonal Loss (COL) is its ability to induce disentanglement in the concept space. By incorporating COL into the training process, the model learns to separate and represent concepts in a more distinct and independent manner. This disentanglement is achieved by enforcing an orthogonal relationship among different concept representations. To evaluate disentanglement, we use the Oracle Impurity Score as proposed by Zarlenga et al. [61]. The metric essentially aims to detect for impurities in soft representations of concepts. We use this metric to compare against Joint-CBM. The OIS score for Joint-CBM on the CUB dataset was 0 . 19 while on coop-CBM with COL 0 . 14 which shows better disentanglement in concept learning for coop-CBM + COL model showing the benefit of using COL on top of any model ad-hoc.

COL encourages the model to assign orthogonal directions to different concepts, thereby reducing the overlap and correlation between them. As a result, each concept becomes more independent and captures a specific aspect or attribute of the input data. This disentanglement in the concept space enables better interpretability and facilitates a clearer understanding of how different concepts contribute to the model's decision-making process.

Through disentanglement, COL enhances the separability and discriminative power of the learned concept representations. It allows the model to focus on relevant and informative aspects of the data while minimizing the influence of irrelevant or redundant features. This disentanglement not only improves the interpretability of the model but also contributes to its overall performance by reducing concept interference and enhancing the model's ability to generalize to new and unseen data.

By promoting disentanglement in the concept space, COL provides a valuable tool for understanding and analyzing the inner workings of concept-based models. It opens up opportunities for further research and exploration into how disentangled concept representations can be leveraged for various tasks, including transfer learning, domain adaptation, and model debugging.

C.2 When is concept orthogonality relevant

One obvious question and could be an interesting future work could be to devise what explicit concepts must be orthogonal. Our work assumes that every concept must be orthogonal but potentially there could be an application where it could be beneficial to include partial orthogonality. Devising an optimal point for when to use COL could be great future work building on our work. We attempt to provide justification using empirical analysis and some intuition in this section.

Duplicating concepts We consider a scenario where input concepts are intentionally duplicated to create a high degree of concept correlation. In our experiment from Table 9, we duplicated 10% , 25% , 50% and 100% of concepts and added them to the original concept bank. This is a worst-case representation of 'similar concepts'. From the table, we see that the duplication of concepts does not impact the concept or the task accuracy. Additionally, this experiment contributes to the broader understanding of how COL performs in various scenarios. We observe that the performance is not significantly impacted for concept duplication if we add COL.

Histogram of concept logits for joint-CBM and coop-CBM To gain further insight into the effect of COL, we computed histograms of the activations of the penultimate layer (to which the loss is applied) and saw that the histogram of CBM+COL is very sparse (with a large peak at 0 and a much smaller one at 1) in contrast to the vanilla CBM.

Histogram of concept logits for joint-CBM

coop-CBM+COL

joint-CBM

100

80

Frequency

60

40

20

0

0.0

0.2

0.4

0.6

0.8

Value

Intuition The dissimilarity loss d 2 encourages independent concept prediction. This is particularly important to reduce leakage in CBMs and improve the robustness of concept explanations. For the connection between entanglement of concept predictions and leakage, we refer the reader to the introduction section of Havasi et al. [20]. Motivated by this insight, we introduce d 2 , to encourage disentanglement of the penultimate layer of features for concept prediction. We agree that this may result in an overcomplete representation, induced by the opposing forces of d 1 and d 2 for samples with partially overlapping concepts, and multiple feature groups may contribute to the same concept. However, it might be difficult to achieve disentanglement otherwise, and as our experiments show, COL improves the concept representation and accuracy, including in the out-ofdomain settings. To gain further insight into the effect of COL, we computed histograms of the activations of the penultimate layer (Figure 3) and see that the histogram of CBM+COL is very sparse (with a large peak at 0 and a much smaller one at 1) in contrast to the vanilla CBM. This suggests that COL may encourage learning of an overcomplete sparse feature space, the elements of which encode various combinations of concepts, and the last layer of f learns to introduce invariance in the prediction of each concept c i with respect to specific combinations with other concepts c j by linear combination. Additionally, it must be noted that this regularization is only applied to the penultimate layer before concept prediction which means low-level features are still free to share weights. Essentially, we believe COL encourages learning a sparse over-complete dictionary of features with concepts still partially entangled in different combinations. The concept prediction layer in f then learns a linear combination of these specialized features to introduce invariance

with respect to the specific combinations. In fact, when studying histograms of activations of the penultimate layer, we observe that activations with COL are indeed very sparse in contrast to CBMs without COL. Given a sparse dictionary of combinations of concepts as induced by COL, the task of disentanglement of concept prediction would ideally reduce to a linear combination of dictionary elements. This is the intuition behind sparse coding (see [30]). In our case, the sparsity is induced indirectly as a result of the orthogonality-based loss formulation. Our experiments show that this approach significantly facilitates the overall optimization of CBMs, improving concept accuracy and downstream performance.

C.3 Effect of lambda on COL

We experimented with different loss weights for λ in our experiments and the model+COL seemed to be fairly robust with different values of λ . We have put those results in Table 5 of the rebuttal PDF on CUB and TIL datasets. While a fine-tuned value of λ might show good performance, we observe that regardless, the model is still able to beat the performances of other baselines. We observed that 0 . 05 set as a good tradeoff between performance and uncertainty across datasets.

D Further experiments

D.1 Comparison against models with automated concept acquisition

[38] and [58] used a pre-trained model - CLIP which was trained on a massive corpus of data to obtain concepts. This can potentially introduce inherent biases from pretraining into the concepts. This was also brought up in the Limitations and Conclusion section of [58]. Furthermore, the dissimilarity in the concepts employed in these works adds complexity to establish a fair and meaningful comparison. Moreover, we wish to emphasize that neither of these works directly compare with CBM variants in their main paper, except for [58] which appears in Appendix C. Also it is not possible to compare with realistic medical datasets as CLIP fails to generate meaningful concepts. Regardless, we have compared both the methods with our method on CUB+OOD datasets and our model outperforms the accuracy of [38] and [58], which is lower than the standard model. We believe it will be interesting future to include our model methodology and COL in the respective models.

D.2 Coop-CBM and COL in the presence of sparse concept labels

Concept labeling could be a labor-intensive task and hence it is important to understand the most optimal point of operation. We randomly select a subset of concepts and train baselines on the subset. Due to concept and task prediction at the same level, we observe coop-CBM provides inductive bias for the downstream task.

D.3 Detailed results with image corruptions

E Further about interventions

Concept Uncertainty Score

A significant part of the intervention selector constitutes Concept Uncertainty Score (CUS) which denotes the model uncertainty for the prediction of concepts. We calculate epistemic uncertainty that arises due to model parameters and lack of training samples. Realistically, labeled medical data is often scarce, encouraging the application of epistemic uncertainty quantification of the predicted concepts. We use Monte-Carlo dropout [16] to model epistemic uncertainty, with a random dropout rate of 0 . 2 . We apply the dropout before the prediction of concepts. For an image x i predicting concepts c i ...c K where K is the total number of concepts, we evaluate T softmax probabilities { p t } T t =1 for each concept prediction. We measure the uncertainty for each concept which we refer to as H ( · ) . Wecompute the entropy-based uncertainty for each concept as the measure of the expectation of the information inherited in the possible outcomes of a random variable. Using the uncertainty metric, we calculate the overall uncertainty concept vector H .

Concept Weightage Score

The second part of the intervention selector score is signified by Concept Weightage Score (CWS), which accounts for the importance of a concept in the final downstream prediction task. Using CWS, the intervention selector is able to prioritize the concept for intervention that is deemed to change the final prediction significantly. We define the weightage score as β . The f ( y | c ) is a linear one-layer Multi-Layer Perceptron (MLP) which helps in defining β .

Supervisor Confidence Score

Finally, in the intervention selector, we consider the reliability of the annotator via Supervisor Confidence Score (SCS). For example, while a histopathologist can identify diseases across human tissues and organs, they often have more specialized and nuanced areas of focus. It is therefore

beneficial for the model to request additional information from histopathologists from their expert knowledge. This in practice prevents ambiguous or inaccurate concept correction. The SCS is a variable across each annotator and is represented by γ .

Finally, bringing the three desiderata of concept selection for intervention, the final intervention selector score is a linear combination of all of the scores.

where k 1 , k 2 , k 3 are hyperparameters for importance weightage on each of the scores.

Intervention allows the model to query the most significant concepts. In our case, it is hypothesized that the most uncertain concepts will build a symbiotic relationship between the human and the model. The supervisor decides the threshold I th to correct the concept prediction. Realistically, in medical scenarios, due to the professional's limited availability, we would like to optimize the number of concepts to intervene. Setting a lower threshold is a trade-off decided by the user. In contrast to other works that perform group interventions [27] by intervening on a group of similar concepts, we perform single interventions. Group interventions require clustering of concepts on the basis of their similarity 2 , which is not realistic as such information is not always available. Therefore, single interventions are performed to minimize the dependence on human priors.

Algorithm 1 Intervention selector Pseudocode

F Example of OOD data

AccuracyvsinterventionsonCUB

Random(CBM)

CUS,SCS,CWS

SCS

76

CWS

CUS

74

2

72

Accurac

70

68

66

0

50

100

150

200

250

300

Number of single interventions

AccuracyvsinterventionsonCUB

Random(CBM)

84

CUS,SCS,CWS

SCS

CWS

82

CUS

2

80

ccura

78

76

74

0

50

100

150

200

250

300

Numberofsingleinterventions

G Limitations and Future Work

In this work, we proposed a novel concept-based approach, coop-CBM, to enhance the interpretability and accuracy trade-off in AI models. We introduced the Concept Orthogonal Loss (COL) to improve concept learning and employed Coop-CBM on various datasets and evaluation scenarios. Our results demonstrated superior generalization, robustness to spurious correlations, and advancements in the accuracy-interpretability trade-off.

While our proposed approach has shown promising results and made significant contributions, it is important to recognize certain limitations. Firstly, the reliance on labeled concept vectors poses challenges in domains where concept annotations are limited or costly to acquire. Furthermore, current concept annotation methods suffer from biases and a lack of domain knowledge, indicating the need for further improvements in this area. Our work does not address unsupervised concept acquisition methods, but instead focuses on a model architecture that can be applied regardless of the concept acquisition approach.

While our concept-based approach offers interpretability, it assumes that the learned concepts align closely with human-understandable notions. However, there is a possibility that the learned concepts may not always perfectly align with the intended interpretations, which can pose challenges in terms of comprehensibility and explainability. Future research could delve deeper into understanding explanations and their alignment with human understanding, thereby exploring ways to improve the fidelity of concept-based models in providing accurate and meaningful explanations. Future research could aim to incorporate additional evaluation metrics that assess the transparency, fairness, and robustness aspects of concept-based models.

Overall, while our approach shows promising results and addresses important concerns in the field of explainable AI, it is important to be aware of these limitations and continue advancing research to overcome them and further enhance the applicability and reliability of concept-based models.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes multiple improvements to current paradigm about learning, application and evaluation of concept bottleneck models (CBMs) for by-design interpretable classification. They propose to improve supervised learning of concepts in CBMs through a concept-orthogonality loss (COL) that encourages samples from different classes to have orthogonal feature representations used for concept prediction and samples with same concepts to have similar representation. Addtionally, to improve classification performance of the model, they propose to predict class labels additionally from this representation. In terms of usage of CBMs, the authors propose to incorporate supervisors confidence when making decisions about intervention. In terms of evauation they propose to also evaluate baselines under distribution shifts.\n\nSTRENGTHS:\nStrengths: \n(a) The paper is well written in large parts. The organization is fluid and comfortable to follow. The motivations for the proposals are also generally clear.\n\n(b) The experiments are solid and extensive with multiple different datasets, baselines and experimental settings.\n\n(c) The paper contains multiple novel contributions in terms of learning, usage or evaluation of CBMs.\n\nThe quality and clarity is high in most parts. Considering all the contributions as a whole, the paper is reasonably original and moderately significant in my opinion.\n\nWEAKNESSES:\nWeaknesses:\n(a) Each individual contribution is limited in either novelty or impact. For instance the concept-orthogonality loss (COL) is the most novel contribution but I wouldn't consider the improvement in concept/model accuracy (Tables 1-4) highly significant. On the other extreme while I'd consider evaluation of all baselines under distribution shifts useful for community, I won't regard it as highly original. This weakness sets an upper bound in terms of strength of this paper for me. That being said, considering all contributions together, this won't be a determining factor for me in terms of recommending acceptance or rejection.\n \n(b) The experiments studying impact of loss weights are lacking in certain ways. More details  about this in section for questions.  \n\n(c) Statements and reasoning around concept orthogonality-loss (COL) is sometimes confusing and potentially misleading to me. This is the most significant concern. Please refer to questions/concerns below for more details about this.\n\nOverall, I enjoyed reading the paper. Moreover, the paper as a whole should be useful for the ML community. However because of some major doubts regarding discussion around COL which is the most crucial idea of the paper, I can only favour the paper for borderline acceptance currently.\n\nQUESTIONS:\nQuestions:\n\n(a) Loss weights: Among the loss weights to be set, $\\gamma$ and $\\lambda$ relating to COL seemed most crucial to me. The choice of $ \\lambda=0.05 $ seems a little arbitrary. Could you explain how did you arrive at this value whether it be through a heuristic or otherwise? Also, is the system behaviour sensitive to choice of $\\lambda$? How is the learning affected if its value changes? The experiment studying hyperparameters (appendix E) should also study role of $\\lambda$.\n\n(b) Statements surrounding COL: I got confused 2-3 times in understanding COL due to the statements made about it. Before going into the specific instances, I want to confirm that my understanding is accurate. Please correct me if I am mistaken in the following statements.\n1. $q$ is a single feature representation used to predict **all** the concept labels.\n2. In eq (5) $c_i, c_j$ denote concept labels for sample $i, j$ in a batch. These are binary vectors of size number of concepts. Consequently, COL encourages $q_i, q_j$ to be orthogonal if **ANY** concept label differs between samples $i, j$. COL also encourages $q_i, q_j$ to align if **ALL** concept labels among $i, j$ match.\n\nIf both these statements are correct, then these are my questions/concerns:\n1. Is differing of any concept label between sample $i, j$ equivalent to them belonging to different classes (line 188, Pg 5)? Although this is a completely reasonable proxy but is it possible that two samples from same class differ in just one concept label and thus COL regards them as dissimilar?\n2. Multiple times in line 182-186 in context of COL similarity loss, you state that it encourages similarity between feature representation for samples from \"same concept\". Isn't this an inaccurate phrasing? The accurate phrase should be for samples with \"same set of concept-labels\". \n3. In line 290-292, you also refer to COL loss as \"maximizing the inner product between the concept embeddings of different concepts\". Firstly, it should be 'minimizing'? Also there is only one concept embedding $q$, right? Reading this statement combined with ones in previous point was particularly confusing as it conveys a meaning of orthogonality differently imposed copmared to COL. Namely, one could encourage different feature maps in $q$ to be used in prediction of different concepts, even for a single sample. This would more accurately represent \"separating\" representations of different concepts. But that is not what COL proposes. \n4. Lastly, following the previous point, doesn't COL have some disadvantages in terms of imposing orthogonality for learning concepts? To be more specific, as an example consider two samples of similar classes. They are very likely to share many concepts (but not all). Isn't it sensible that the feature representation be similar as they share many concepts? The COL loss on the contrary encourages the feature representations to be completely dissimilar. Isn't it more sensible for dissimilarity loss to be applied **ONLY IF ALL** the concept labels present in two samples differ? If indeed this is the case, it needs to made very explicit.\n\nLIMITATIONS:\nThe authors discuss the limitations in appendix. The content of the discussion covers some key points and is adequate for me.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Coop-CBM, a concept bottleneck model (CBM) trained using a novel multi-task loss and orthogonality regularizer. The proposed losses discourage undesired leakage in a CBM's concept representations while encouraging a better balance of their accuracy and interpretability. By incorporating these loss terms in their objective functions, *Coop-CBMs* are encouraged separate latent representations learnt for different concepts while reducing the intra-concept distance. This design enables effective concept interventions in these models while maintaining high concept and task accuracies. The authors perform an extensive evaluation of Coop-CBMs on four real-world datasets for image classification tasks, together with corrupted variants, and show that the proposed method achieves higher accuracy in all instances, compared to black-box models and previous CBM-based models, while remaining receptive to test-time concept interventions.\n\nSTRENGTHS:\nThank you for the very well-written and interesting paper. I thoroughly enjoy reading this work, and I believe it is both addressing an interesting problem while having the potential to inspire different methods elsewhere. After carefully reading the paper and the accompanying appendix, I believe that its main strengths are:\n\n1. Although the main idea in this paper is simple in nature, it is an interesting approach to alleviating some of the concerns that have been raised on the quality of learnt concept representations in CBMs, particularly regarding leakage.\n2. The evaluation across four real-world datasets, as well as variations of these datasets with spurious correlations for OOD evaluation, makes the experiment section of this paper very well-designed. In particular, I believe there is a lot of potential impactful work in better understanding these models in OOD scenarios.\n3. Given the need for interpretable architectures that are actually useful in practice, the core contributions of this work have the potential to be impactful if they stand thorough evaluation.\n4. The main idea is not entirely novel, as it shares similarities to ideas previously proposed in disentanglement learning and contrastive learning. Still, its application to this space, and its evaluation in this space, is novel and worthy of study.\n\nWEAKNESSES:\nWhile I believe the ideas in this work have future potential, I think there is significant room for improvement in how this work is presented and evaluated. Specifically, I believe the following are its main limitations:\n\n1.  I have some doubts about some of the claims made by the paper regarding the usability of its proposed method in concept-scarce (or concept-incomplete) settings. The results briefly discussed in passing in Section 5, and discussed in more detail in Section E of the Appendix, seem a bit suspicious to me without further clarification. This is because, intuitively,  one would not expect Coop-CBM to perform significantly better than CBM-AR or Joint-CBM when the number of training concepts is very limited (as the bottleneck's capacity will simply be too constrained!). Please see the question related to this point in the section below to see where my suspicion arises from. Please feel free to correct me if this is a misunderstanding on my end.\n2. Another serious weakness in this work, which I believe other reviewers may also point out, is the number of sheer hyperparameters the proposed loss has ($\\alpha, \\beta, \\gamma, \\lambda$ for the loss and even more for the intervention algorithm). This. together with the lack of an accompanying ablation study showing whether the results presented in this work would change if one fine-tunes these hyperparameters as well as those in competing methods, severely limits the robustness of the evaluation. The claim that “for fair comparison, the concept prediction weight (here $\\alpha$) in our baseline models is set to the same value”(lines 248-249) is in fact hard to take at face value. This is because it is unclear why all other hyperparameters are set to $0.01$ and whether this happens to be a value that greatly benefits the presented method. Therefore, more clarity on this end, including a potential ablation study and an appropriate hyperparameter selection for competing methods, would make the claims in this paper much stronger.\n3. A lot of the details on the intervention side of things are left undefined in the main body of the paper and moved to the appendix. Currently, most of the contributions this work makes on the side of interventions (e.g., the study of user uncertainty or the use of a new intervention policy for selecting concepts) seem completely orthogonal to the rest of the paper’s work/main motivation and therefore are hard to evaluate within the context of the same paper. Don’t get me wrong: I believe these are interesting results; it is just that they currently seem out of place with almost all details pushed to the appendix (see how much methodological content section F has) and with a very limited evaluation if this is a core contribution. For example, it is unclear whether Algorithm 1 is used to select concepts in the results of Figure 1 of the main paper. Based on the caption saying that concepts are randomly intervened, I believe this is not the case. If it is, then it is not a fair comparison to compare Coop-CBM with a non-trivial concept selection policy against baselines that select concepts at random. If it is not, then there is no evaluation of the proposed concept selection policy anywhere in the main body of the paper. Either way, further evaluation of the proposed policy or clarification of how it fits in with the rest of the work is required.\n4. I am a bit unsure about what the main takeaways of the experiments in Section 5.3 are that are not offered by previous experiments. Clarification on this end could help a lot.\n5. The codebase provided is incomplete (I could not find some of the baselines) and missing key comments or pseudo-documentation. I am not expecting production-quality documentation in any way but at least a README to guide readers through it would be very useful. \n6. Against traditional convention, and the authors' answer to the questionnaire question on error bars, Figure 1 and Tables 3, 4 and 5 are all missing error bars.\n7. Although in the questionnaire the authors indicate the compute resources and licenses are stated in the paper/supplementary material, I was unable to find either of these.\n8. Similarly, a lot of experimental/reproducibility details are missing from their codebase and their paper’s supplementary material, severely limiting the possibility of reproducing these results.\n\nQUESTIONS:\nAfter carefully reading the paper and the appendix, I had some trouble getting convinced by some of its results and proposed methods. My score is mostly influenced by this work's (1) lack of proper evaluation against existing baselines (specifically in how hyperparameters are selected), (2) its possibly counterintuitive results when the set of concept annotations is incomplete (and the lack thereof of an explanation for such unexpected results), and (3) the large amount of missing of details/explanations in some crucial components of this work that may lead to reproducibility issues. However,  I think my concerns may be addressed  by successfully addressing these questions:\n\n1. If you look at Figure 3 of [1], you can see an evaluation of interventions in the method I believe this work calls “CBM-AR” (called “Hard AR w/o side-channel” in [1]). In that figure, it is clear that the task accuracy of CBM-AR is significantly low when one uses a small number of concepts during training (as one would expect as the model’s bottleneck is extremely constrained). However, in the results shown in Table 1 of the Appendix, the same drastic change in accuracy is not observed for CBM-AR (with only a drop of ~8% in task accuracy when using 10% of all concept groups). This makes me a bit suspicious of all of the results in this table. Therefore, I was wondering the following:\n    1. When the paper says, for example, that 10% of concepts were used as annotations during training that means does that mean that the same group of 10% of training concepts were given for all samples of the training set **or** that, for any given sample, 10% of the concepts were selected at random and provided with annotations without regards of what concepts were provided to other training samples? That is, are concepts being subsampled at a \"global\" scale (where all samples are provided with the same subsample of concept annotations) or at a local scale (where each sample may have different concept annotations than other samples)? My guess is that it is the former (global scale subsampling) but I want to confirm this as it is unclear from the text.\n    2. If it is the former as I believe, then is the paper using the side channel version of “Hard AR” when evaluating CBM-AR? If so, this should be made explicitly clear somewhere as no details could be found.\n    3. **More importantly**, could you please clarify what is the intuition behind Coop-CBM having such a high accuracy when one uses a small number of training concepts (say 10%)? If the prediction used for the downstream task is that given by $g(f(x))$, then **I find this result extremely surprising**  given that Coop-CBM is still, just like its traditional CBM counterpart, severely constrained at the bottleneck when one uses a small number of concepts (and one would thus expect an even more severe drop in performance). One possible way I can see one can achieve this result is if the prediction used to compute the accuracy reported in Table 1 of the appendix is to use instead the immediate prediction made by $h$ (i.e., $h(f(x))$) as the output prediction of Coop-CBM. If that is the case, though, then this goes against how the model is originally defined in Section 3.2 and, more importantly, would mean that interventions are meaningless for this process as the prediction is bypassing the concept bottleneck. One convincing piece of evidence that your model is indeed working as expected (i.e., by predicting the label given by $g(f(x))$) is to show that it still positively reacts to interventions when the number of concepts is small (say 10%).\n2. How do the results reported in some of the experiments change if one selects the hyperparameters for competing methods using a simple hyperparameter search (say over 0.1, 1, 10) for the concept loss weight in Joint-CBMs, CEMs, and CBM-ARs?\n3. Could you please clarify the intervention concerns raised in the weaknesses? That is, how are interventions evaluated in Figure 1 and what is the relationship between these contributions and the rest of contributions? Are the contributions orthogonal to each other?\n4. In equation (5), it is still a bit unclear what $q_i$ is. Could you please clarify this?\n5. In section 5.1, why does 30% of the test set still have a spurious correlation? What is the rationale behind not removing the spurious correlation in its entirety to see how much the model truly depended on the shortcut for its predictions?\n\nRegarding suggestions for the presentation, the paper is very well-written (thank you!). Nevertheless, the following possible errors and typos could be addressed before a camera-ready version:\n\n- Typo in line 49: it should be “Mahinpei et al. [16] have shown” rather than “Mahinpei et al. [16] has shown”.\n- Typo in line 96: parenthesis is accidentally underscripted in $f(c|x)$.\n- Typo in line 127: missing “the” between “in” and “concept predictor”.\n- Typo in equation (3): I believe the last term should be $\\mathcal{L}_y(g(f(x)), y)$ rather than $\\mathcal{L}_y(g(c), y)$.\n- Typo in line 196: the sentence started in this line is a bit confusing (possibly missing a couple of words).\n- This came out earlier this year, so I think it is ok to miss the reference. However, for the sake of completeness, it is worth noting that recent work [2] does consider human error/uncertainty when performing interventions as opposed to the claim in lines 299-300. This is not a big deal as it happened within a few months before NeurIPS’ deadline, but it is worth noting for a future camera-ready version of the manuscript.\n- nit: ChatGPT and BARD are missing citations.\n- nit: “XAI” in line 26 is never explicitly defined.\n- The citation to “[1]” in the paper’s references of the main text is incomplete (the SENN paper reference).\n- I couldn’t find anything related to the “Oracle Impurity Score” in the original CEM paper as cited in the Appendix in line 160. A quick search for that term found that it is defined in a separate paper [3].\n- Section 3 could benefit from a diagram explaining the overall training process and architecture.\n- This is pure nitpicking on my end, but I am still unsure what the “coop” part of “Coop-CBM” stands for as it is never explained/defined.\n\n### References\n\n[1] Havasi et al. \"Addressing leakage in concept bottleneck models.\" *NeurIPS 2022*.\n\n[2] Collins et al. \"Human Uncertainty in Concept-Based AI Systems.\" *arXiv preprint arXiv:2303.12872* (2023).\n\n[3] Espinosa Zarlenga et al. \"Towards robust metrics for concept representation evaluation.\" AAAI 2023.\n\nLIMITATIONS:\nAlthough included in the appendix, this work provides a discussion of some of the limitations and potential impacts of its work. If possible, the manuscript could benefit from including such a discussion in the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce an auxiliary task in CBM training to incorporate more information regarding the downstream task in the input representation for predicting concepts. They also introduce Concept Orthogonal Loss (COL) to enforce orthogonalization between input representations of different concepts, and vice versa. Experiments show superiority of their method over existing CBM baselines on the performance of the downstream task, for standard training settings as well as settings with manually-injected distribution shifts.\n\nSTRENGTHS:\n* Simple and straightforward presentation of the proposed methods. Very easy to read.\n* Proposed method makes intuitive sense.\n* Solid experiments.\n* Experiments on extensive settings to demonstrate the superiority of CBMs over standard models w/o concepts.\n\nWEAKNESSES:\n* Risk of information leakage: Mahinpei et al. have shown that CBMs may leak information of the target task through the concepts, also mentioned in the paper (P2L50). Whether the performance gain of coop-CBM comes from information leakage is not exactly clear.\n* Assumption of all concepts should be orthogonalized: The proposed COL implicitly assumes that all concepts should be orthogonal. The information gain shown in the experiments that this assumption holds true in the tested settings. In general, user selected concepts may not be entirely orthogonal. This method may be limiting when a user does not know whether adopting COL is appropriate given the chosen concepts.\n\nQUESTIONS:\nI am willing to increase the score and accept the paper as I genuinely believe this work is a good solid step in terms of making concept-based models more viable if the following issues are resolved properly.\n\n### Risk of information leakage\nMy main concern is that coop-CBM improves downstream task performance because downstream task information is leaked through the concepts. This makes intuitive sense as the explanation of performance gain is the representation used to predict concepts contain more downstream task information. Actually I am not sure where the author stands on the topic of leaking information through the concepts. Does the authors believe that benign leakage of information is acceptable? \n\nPerhaps experiments could be repeated by clipping the concept predicted values in coop-CBM to hard labels (as suggested in Mahinpei et al's paper) to reduce the leakage and see if there is still performance gain (or maybe this is already the default behavior indicated somewhere in the work that I missed).\n\nThe risk of information leakage is that then the concept scores no longer purely represents the concept, but rather encodes the downstream task label in some implicit manner. Then the whole concept bottleneck idea breaks down. The bottleneck is no longer a bottleneck and explainability through concepts is no longer valid.\n\n### Assumption of all concepts should be orthogonalized\nThe authors extended the idea of concept whitening to CBMs through COL. However, orthogonalizing every concept may not be always beneficial (e.g. related concepts). Are the authors claiming orthogonalizing concepts is always beneficial? Is it possible to characterize when to adopt COL? Perhaps it is impossible to make definite statements about when COL would work but is it possible to at least provide a good heuristic that serves as a good-enough guideline?\n\nFor example, concepts in CUB are grouped into attributes of different parts of birds (e.g. body, head, wing). Does doing COL on inter-grouped concepts make sense? Perhaps something could be done with prior knowledge of which concept correlates more with others? Are there easy metrics that could serve as a proxies for concept correlation?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposes coop-CBMs + concept orthogonal loss (COL) to solve previous limitations in Concept Based Models with a focus to learn relevant concept representations that consecutively boost model performance.\n\nSTRENGTHS:\nIn general, the idea of coop-CBMs+COL is simple and effective. The method increases prediction accuracy on benchmark datasets and  seems to work well on noisy inputs.\n\nWEAKNESSES:\n* The overall structure of the paper is reasonable, but part of substructure in some sections are not well organized. In introduction section, the author first introduce definition of previous Concept bottleneck models, then pointing out its limitation.Then followed with the importance of studying the robustness of CBMs in various domains. After that, returns back to bottleneck of CBMs. Unless the author aims to describing totally different limitations of previous CBMs, it feels unnecessary.\n\n* In Section3, the author does not specify what does $\\hat{\\theta}$,  $\\hat{\\Phi}$ stands for,  it would be better to insert formula into the words such as a formula followed by it notations. Otherwise this description looks messy. \n\n* The author does not show experimental results for using CBM alone on classification Coop-CBM.  \n\n* It's confusing calling Coop-CBM both a component of the final strategy and the whole strategy incorporating it.There is a bit confusing that what is your final method. Coop-CBM or Coop-CBM + COL ?\n\nQUESTIONS:\n1. modify the structure/content arrangements in introduction section. \n2. do more literature review on section2 on the relationship between Concept-Based Model and explainability if applicable.\n3. please clarify the notations in method part.\n4. please clarify what is your final method, Coop-CBM or Coop-CBM + COL ?\n5. add classification results for only using Coop-CBM on CUB, AwA2 and TIL datasets.\n6. better to tell the significance of the proposed method, not just limited to some improvements in the benchmark dataset.\n\nLIMITATIONS:\nNo. \n\nit would better to discuss the limitations of proposed methods ( Coop-CBM along with COL) in `Discussion and Conclusion` section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes auxiliary losses to learn concept-bottleneck models: a coop task loss, and a COL (orthogonal concept loss). The experiments show that the proposed coop-cbm can perform 1-3% better than standard black-box models and outperform other CBM models on three datasets: CUB, AwA2 and TIL.  \n\n==================\npost-rebuttal: the authors have addressed my concerns and questions in the rebuttal. I would like to increase the rating from 4 to 6.\n\nSTRENGTHS:\n1. This topic is important\n2. The paper compared with 3 fairly new CBM baselines (CBM [12], CEM [35], CBM-AR [8]), but stronger baselines are not compared [33, 18]. The authors mentioned it's an unfair comparison to [33] in Line 228-229, but it's still not clear why, and why [18] is not compared either.\n\nWEAKNESSES:\n1. The proposed idea is interesting, though the two losses (COOP loss Eq 3, and COL in Eq 6) seem to be proposed in prior works? The authors mentioned that COL is from [21], and it feels like the COOP loss in Eq 3 is similar to the non-interpretable \"residual\" terms in Post-hoc CBM [33].  So the main contribution seems to be integrating them together? Please clarify and compare if there's some misunderstanding here. \n\n2. The description of some related work seems not accurate: Line 74-76: what does it mean that \"suffer from pretrained model's biases\"? Post-hoc CBM [33] with residual terms can improve the downstream task accuracy. It looks like COOP can perform better is due to the residual loss + join training (since post-hoc CBM only training the last 2 layers)? Besides, the result in Label-free CBM [18] show that it is comparable to original downstream task performance without the residual terms. Please compare with [33] and [18] with ablation study.\n\n3. Large scale datasets results are missing (ImageNet)\n\n4. The exposition of the technical part can be improved (sec 3). For example, to describe f, g, h, it'll be useful to visualize the structure. phi and theta are not defined. Also, since f and h actually differ in one linear layer, why both of them still use theta? it seems phi is the parameter of g. \n\n\n\n\n**minor**\n- an extra s in Line 277 \"modelss\"\n\nQUESTIONS:\n1. It's not clear why the concepts are necessary need to be orthogonal. Could the authors explain? \n\n2. In eq 5, how do you define intra-concepts? It looks like you are using c_i= c_j, which means the concepts are exactly the same? Please elaborate the details. \n\n3. Line 228-229, why is it unfair to compare with post-hoc CBM [33]? Do you mean that it is expected that [33] with joint training can outperform coop-cbm? please clarify\n\n4. Please report computation cost for baselines and the proposed method in Table 1.\n\n5. How is Fig 1 calculated? Is it correcting the mis-classified images in the test time through intervention? Please give more details on how Fig 1 is plotted.\n\nLIMITATIONS:\nI did not see a dedicated paragraph to discuss the limitation of the proposed work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}