SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ivaxi Sheth*
CISPA-Helmholtz Center for Information Security
ivaxi.sheth@cispa.de
Samira Ebrahimi Kahou
École de technologie supérieure, Mila, CIFAR AI Chair
samira.ebrahimi-ahou@etsmtl.ca
The increasing use of neural networks in various applications has lead to increasing apprehensions, underscoring the necessity to understand their operations beyond mere final predictions. As a solution to enhance model transparency, Concept Bottleneck Models (CBMs) have gained popularity since their introduction. CBMs essentially limit the latent space of a model to human-understandable high-level concepts. While beneficial, CBMs have been reported to often learn irrelevant concept representations that consecutively damage model performance. To overcome the performance trade-off, we propose cooperative-Concept Bottleneck Model (coop-CBM). The concept representation of our model is particularly meaningful when fine-grained concept labels are absent. Furthermore, we introduce the concept orthogonal loss (COL) to encourage the separation between the concept representations and to reduce the intra-concept distance. This paper presents extensive experiments on real-world datasets for image classification tasks, namely CUB, AwA2, CelebA and TIL. We also study the performance of coop-CBM models under various distributional shift settings. We show that our proposed method achieves higher accuracy in all distributional shift settings even compared to the black-box models with the highest concept accuracy.
The recent advances in deep learning have impacted various domains such as computer vision [21], natural language processing [53], and speech recognition [6]. Recently, the deployment of large models [8, 51] has led to various concerns regarding privacy and safety since machine learning models are often considered black boxes. With the increasing use of such deep learning models in daily human life and their wide deployment, it is essential to understand model behaviors beyond final prediction. Since neural networks are considered opaque decision-makers, inaccurate decisions by models in applications such as medicine [18] or autonomous driving [59] lead to catastrophe for humans. To understand the inner workings of such black-box neural networks, the field of XAI [32] has emerged in recent times. Concept Bottleneck Models (CBMs) [27] are a family of neural networks that enable human interpretable explanations.
Concept-based models introduce a bottleneck layer before the final prediction. This bottleneck layer consists of human-interpretable concept predictions. For example, in the context of images of animals, these concepts could be "mane" in the case of a lion or "black and white stripes" in the case of a zebra.
*Work done when the author was at Mila, École de technologie supérieure.
Concept bottleneck modelss (CBMs) map input images to interpretable concepts, which in turn are used to predict the label. The intermediary concept prediction allows a human supervisor to interpret and understand the concepts influencing the label prediction. In addition to explainability, CBMs offers an interesting paradigm that allows humans to interact with explanations. During inference, a supervisor can query for explanations for a corresponding label, and if it observes an incorrect concept-based explanation then the supervisor can provide feedback.
While CBMs present benefits with models' explainability, Mahinpei et al. [33] have shown that concept representations of CBM may result in information leakage that deteriorates predictive performance. It is also noted that CBM may not lead to semantically explainable concepts [34]. Such bottlenecks may result in ineffective predictions that could prevent the use of CBMs in the wild.
Along with model transparency, another challenge that modern neural networks face is robustness to distributional shifts [32]. Deep learning models fail to generalize in real-world applications where datasets are non-id [17]. The absence of a comprehensive study examining the behavior of CBMs under distributional shifts is a significant limitation, potentially impeding their application in real-world scenarios.
In this work, we propose cooperative-CBM (coop-CBM) model aimed at addressing the performance gap between CBMs and standard black-box models. Coop-CBM uses an auxiliary loss that facilitates the learning of a rich and expressive concept representation for downstream task. To obtain orthogonal and disentangled concept representation, we also propose concept orthogonal loss (COL). COL can be applied during training for any concept-based model to improve their concept accuracy. Our main contributions are as follows:
- We proposed a multi-task learning paradigm for Concept Bottleneck Models to introduce inductive bias in concept learning. Our proposed model coop-CBM improves the downstream task accuracy over black box standard models.
- Using the concept orthogonal loss, we introduce orthogonality among concepts in the training of CBMs.
- We perform an extensive evaluation of the generalisation capabilities of CBMs on three different distribution shifts.
- We looked at using human uncertainty as a metric for interventions in CBMs during test-time.
Concept-based Models Early concept-based models that involved the prediction of concepts prior to the classifier were widely used in few-shot learning settings [9, 62]. Other works propose to predict human-specified concepts with statistical modeling [28, 29]. Unsupervised concept learning methods use a concept encoder to extract the concepts and relevance network for final predictions [4, 43]. Although these methods are useful in the absence of pre-defined concepts, they do not enable effective interventions. Concept whitening [11] was introduced as a method to plug an intermediate layer in place of the batch normalization layer of a Convolutional Neural Network (CNN) to assist the model in concept extraction. CBM [27] extends the idea by decomposing the task into two stages: concept prediction through a neural network from inputs, and then target prediction from the concepts. Many works have proposed models built on CBMs to either improve the downstream task accuracy [44, 60, 26] or mitigate the concept leakage [33, 20]. There has been a line of work extending CBMs to real-world applications such as medical imaging [58, 13], autonomous driving [44] and reinforcement learning [14] CBMs require annotated concepts which poses a challenge for their applications to large-scale image datasets. Yuksekgonul et al. [58] propose using concept activation vectors [25] and Oikarinen et al. [38] used multimodal models such as CLIP [40] to annotate concepts for CBMs. Although these either require concept bank or suffer from pretrained model's biases [58].
Alternative losses The training of CBM and its variants typically involves the use of Cross Entropy (CE) loss. Several variants of the CE loss have been explored in the past to improve the discriminative power of learned feature representations of data [19, 46, 56, 39]. Ranasinghe et al. [41], Vorontsov et al. [54] introduce the use of orthogonality in feature space to encourage inter-class separation and intra-class clustering. Our work builds upon [41] by introducing orthogonality constraints in the concept feature space.
Consider a standard supervised learning setting for a classification task, where models are trained on a dataset  $\mathcal{D} = \{x_i,y_i\}_{i = 1}^N$  with  $N$  data samples. Standard models aim to predict the true distribution  $p_{\mathcal{M}}(y|x)$  from an input  $x$ . Although such a setting has been proven effective on vision benchmarks, users are unaware of the detailed inner workings of the model. Therefore, CBMs introduces intermediate prediction of human-understandable concepts before the model prediction.
In the supervised concept-based model setting, the dataset uses additional labeled concept vectors  $c_{i} \in \{0,1\}^{a}$  where each element indicates the presence of one of  $a$  high-level concepts. This allows supervised concept learning in addition to target learning. Following a simplistic causal graph for data generation,  $y \rightarrow c \rightarrow x$ , CBMs consist of two models. The first model  $f_{X \rightarrow C}$  maps the input image  $x$  to concepts  $c$ , while the second model  $g_{C \rightarrow Y}$  maps the concepts  $c$  to the label  $y$ .
CBMs can be categorized by their method of training  $g_{C \to Y}$  from the obtained concept representations  $f(c|x)$ . This could be done in the following manner: jointly, where both  $f_{X \to C}$  and  $g_{C \to Y}$  are trained simultaneously end-to-end, sequentially, where  $f_{X \to C}$  is trained first, after which  $g_{C \to Y}$  is trained using  $p_f(c|x)$  representations, and finally independently, where  $f_{X \to C}$  and  $g_{C \to Y}$  are trained individually and then combined.
Interventions Interventions are a core motivator of CBMs. The bottleneck model allows for interventions by editing the concept predictions. Since CBMs consider correcting the predicted concepts through interventions during test-time, the corrected concepts are not back-propagated through  $f_{X \to C}$  and  $g_{C \to Y}$ . During test-time intervention the predicted concepts can be modified by a supervisor to their ground truth values, leading to "adjusted" concepts prediction. We represent the predicted concepts as  $\hat{c} = p_f(c|x)$  and the modified concepts as  $\bar{c}$ . We consider test-time interventions as an important aspect of explainable models in safety-critical applications. We hypothesize that model-supervisor interaction must lead to the development of a symbiotic relationship between the model and the expert. Here, the expert learns about the potential causation between a concept and its corresponding label, and the model learns true concept values from the expert. We attempt to shine a light on these test-time interventions by simulating realistic scenarios by introducing human uncertainty.
3.2 Coop-CBM
While CBMs provide concept explanations behind a prediction, it has been observed that this can come at the expense of lower model accuracy compared to black-box standard models [33]. In this work, we propose a concept-based architecture, coopCBM to improve the performance of CBMs on downstream classification tasks.
Motivation The different training paradigms in CBMs introduced by Koh et al. [27] give rise to differences in their concept representations,  $p_f(c|x)$ . Koh et al. [27] reports that joint CBMs have the highest task accuracy among the different CBM train
Figure 1: Coop-CBM model that consists of an encoder, concept learner  $f$ , auxiliary label learner  $h$ , and task label learning  $g$ . The encoder transforms input data into a feature representation, which is used by  $f$  to predict high-level concepts and  $h$  to predict a supplemental auxiliary label, and finally  $g$  predicts the final task label conditioned on the concepts only.
ing procedures albeit still lower than standard models. Intuitively, this suggests that joint CBMs, which train both concept predictor and task predictor simultaneously, are able to encode the information about the task label  $y$  into concept labels  $c$  better than sequential and independent CBMs. In the case of joint CBMs, backpropagation of task loss through the concept predictor aids the overall model in giving more accurate predictions. In this work, we aim to leverage such "soft" information
about the task to improve accuracy. Coop-CBM aims to leverage soft label information in concept predictors to better align the concept predictions to the corresponding label.
Model Coop-CBM introduces a multi-task setting before the final prediction. Along with predicting the concepts  $c$ , we introduce the prediction of task labels in the concept predictor. This allows the model to learn relevant signals and inductive biases of downstream tasks in the concept learning phase. In essence, we now have model  $f_{X\rightarrow C}$  that predicts concepts  $c$  from input  $x$  and new model  $h_{X\rightarrow Y}$  that predicts label  $y$  from input  $x$ . This enables the model to learn relevant knowledge about the task that could be absent in the bottleneck concepts  $c$ . Although this setting makes the model interpretable, since corresponding concepts to a label can be obtained, one loses the causal  $x\rightarrow c\rightarrow y$  property. Additionally, it does not allow test-time interventions, which is a key application of concept-based models that facilitates human-model interactions. Therefore, to maintain the original properties of CBM, coop-CBM uses a label predictor  $g$  which takes the predicted concepts  $c$  from  $f_{X\rightarrow C}$  as input and gives the final label  $y$  as output. Hence to avoid confusion, we call the label prediction from  $h$  immediate label,  $y'$ , and the final task label,  $y$ . It must be noted that  $f$  and  $h$  share all but the last linear layer. Therefore the concept predictor  $f$  and  $g$  parameters  $\theta$ ,  $\phi$  are trained in the following manner:
$$
\hat {\theta} = \underset {\mathcal {D}} {\mathbb {E}} \left[ \operatorname {a r g m a x} _ {\theta} \left[ \log p \left(c, y ^ {\prime} \mid x; \theta\right) \right] \right] = \underset {\mathcal {D}} {\mathbb {E}} \left[ \operatorname {a r g m a x} _ {\theta} \left[ \log p _ {f} \left(c \mid x; \theta\right) + \log p _ {h} \left(y ^ {\prime} \mid x; \theta\right) \right] \right. \tag {1}
$$
$$
\hat {\phi} = \underset {\mathcal {D}} {\mathbb {E}} [ \operatorname {a r g m a x} _ {\phi} [ \log p (y | c; \phi) ] ] \tag {2}
$$
Therefore coop-CBM is trained using a linear combination of three different CE losses:  $\mathcal{L}_{\mathcal{C}}$  as concept loss,  $\mathcal{L}_{\mathcal{Y}'}$  as immediate label loss and  $\mathcal{L}_{\mathcal{Y}}$  as task prediction loss.
$$
\arg \min  \left[ \mathcal {L} _ {C} (f (x), c) + \mathcal {L} _ {y ^ {\prime}} (h (x), y) + \mathcal {L} _ {y} (g (f (x), y) \right] \tag {3}
$$
In summary, we argue that the introduction of an immediate label introduces the concept predictor to learn meaningful information about the task while still being interpretable. The  $h_{X \to Y'}$  model intuitively acts as a regularizer for meaningful concept prediction.
Mutual information perspective We hypothesize that by using coop-CBM, the concept predictor acquires better knowledge about  $y$ . In particular, this can be beneficial when fine-grained concept annotations are not available. We, therefore, suspect that the mutual information (MI) between the input image, concept representations, and the label becomes richer and more expressive as compared to CBM [27]. One way to quantify this is by visualising the MI planes throughout the training, similar to Zarlenga et al. [60].
3.3 Concept Orthogonal Loss
Following the current CBM literature, we use cross-entropy loss to train each of the models,  $f_{X \to C}$ ,  $h_{X \to Y'}$  and  $g_{C \to Y}$  in coop-CBM. In model  $f$ , each concept is learned via independent and separate classifiers. Given their binary representation, it is intuitive to improve the embedding space of concepts by increasing separability. To do so, we introduce the concept orthogonal loss (COL). By incorporating COL into the training process, we aim to enhance the overall separability of the concept embeddings, leading to improved performance and interpretability of the coop-CBM model.
Motivation Due to the variations in training strategies employed in different CBM models, the resulting concept representations can exhibit varying levels of accuracy. The concept accuracy refers to how effectively the learned concept representations align with the ground truth or human-defined concepts. Coop-CBM was concerned with the predictive performance of the task, but here we focus on the concept label accuracy. Higher concept label accuracy signifies improved interpretability. As observed by Koh et al. [27], the concept accuracy of joint CBMs models is lower than other variants because the concepts learned are not completely independent of each other, also called leakage by [33]. Hence, increasing the inter-concept distance and intra-concept clustering throughout the concept vector for the entire dataset can allow the model to learn beyond co-dependent concept representations.
COL In addition to CE loss for learning concepts, we introduce novel concept orthogonal loss by conditioning orthogonality constraints on concept feature space. The disadvantage of CE loss is that it does not set a specific distance or separation between different concept representations in the feature space. Consider the CE loss for each concept prediction:
$$
\mathcal {L} _ {C E} (c, \hat {c}) = \sum_ {c _ {i}} ^ {c _ {N}} - c _ {i} \log (\hat {c} _ {i}) - (1 - c _ {i}) \log (1 - \hat {c} _ {i}) \tag {4}
$$
Using this traditional CE loss for each concept, in Equation 4 we are essentially minimizing the difference between the predicted probability distribution and the true probability distribution of the binary concepts. The model,  $f_{c_i}$  attempts to learn probability distribution for when a concept is active or inactive respectively. CE does not explicitly enforce separation between concepts.
With concept orthogonal loss (COL), we enforce the separation in the latent representation of the concepts. Our aim with COL is to group similar features together while ensuring that features belonging to different concept classes do not overlap with each other. COL  $\mathcal{L}_{\mathcal{COL}}$  enforces inter-concept orthogonality and intra-concept clustering. We define inter-concept separation and intra-concept similarity as  $d_{1}$  and  $d_{2}$  respectively. We enforce orthogonality constrain via cosine similarity. We define the  $\mathcal{L}_{\mathcal{COL}}$  loss in the shared last layer,  $q$  of coop-CBM before concept and auxiliary label predictions. We enforce COL constraints within each batch,  $B$ .
$$
d _ {1} = \sum_ {\substack {i, j \in B, \\ c _ {i} ^ {a} = c _ {j} ^ {a} \\ a \in A}} \frac {q _ {i} ^ {T} q _ {j}}{\| q _ {i} \| \left\| q _ {j} \right\|}; d _ {2} = \sum_ {\substack {i, j \in B, \\ c _ {i} ^ {a} \neq c _ {j} ^ {a} \\ a \in A}} \frac {q _ {i} ^ {T} q _ {j}}{\| q _ {i} \| \left\| q _ {j} \right\|} \tag{5}
$$
where  $||.||$  denotes the Frobenius norm.
Using the cosine distances,  $d_{1}$  and  $d_{2}$ , we simultaneously aim to increase the distance between different latent concept representations and decrease the distance between representations from the same concept. We can introduce a hyperparameter,  $\lambda$  to accordingly give weightage to either  $d_{1}$  or  $d_{2}$ . The similarity loss  $d_{1}$  between the feature representation of two samples corresponding to the same concept aims to push the  $d_{1}$  towards 1 which means that the feature representations of same concept should be as similar as possible. As for the dissimilarity loss, the goal is to push the loss towards 0, which enforces that the feature representations of different class samples should be as dissimilar as possible. Therefore we consider the absolute value of  $d_{2}$ .
$$
\mathcal {L} _ {C O L} = \left(1 - d _ {1}\right) + \lambda | d _ {2} | \tag {6}
$$
It is important to note that CE loss is applied to each concept binary classification task, which measures the difference between the predicted class probabilities and the true labels. The introduction of COL encourages the network to learn features that are both discriminative and non-redundant among concepts at an intermediary network level. By combining the COL and CE losses, the network is trained to learn discriminative that separate each concept and useful features for classifying when a concept is active. A benefit of COL is it can be universally any concept-based model to encourage orthogonality between different concepts.
In this section, we introduce two auxiliary losses, one to improve the task accuracy using multi-task setting and the other to improve concept representation in latent space, leading to improved concept accuracy. The final loss is a linear combination  $(\alpha, \beta, \gamma)$  are hyperparameters for weighting in Equation 7) of concept and task losses along with immediate and concept orthogonal losses.
$$
\arg \min  \left[ \alpha \mathcal {L} _ {C} (f (x), c) + \beta \mathcal {L} _ {y ^ {\prime}} (h (x), y) + \mathcal {L} _ {y} (g (c), y) + \gamma \mathcal {L} _ {C O L} (q) \right] \tag {7}
$$
3.4 Interventions
Koh et al. [27] demonstrated the potential of CBMs for facilitating human-model interaction and improving task performance during inference. But it can be time-consuming and costly to have domain experts go over each concept, hence some of the recent and concurrent works proposed to use uncertainty as a metric to select interventions.
We propose a lightweight approach that strategizes the supervisor-model interaction. Our method is intuitive and considers three aspects of intervention:
1. Uncertainty of concept prediction - CUS represents the confidence of the model to predict latent concepts,  $f_{X \to C}$ .
2. Supervisor confidence for concept correction - SCS represents the reliance on the supervisor to intervene and subsequently correct the concepts accurately.
3. Importance of concept for label prediction - CWS denotes the significance of each concept for the subsequent downstream task.
Chauhan et al. [10] propose to optimize interventions over a small validation set using CUS. In comparison, we consider access to the validation set unrealistic. Shin et al. [49], Sheth et al. [48] evaluated interventions more comprehensively and studies the behavior of CBMs during inference by selecting CUS and CWS metrics. We additionally take into account a supervisor's confidence in domain knowledge, SCS and their expertise in correcting the concepts. Concurrent work [12] also looked at human uncertainty for CBMs in depth. Unlike previous works that evaluate test-time interventions on the test splits of respective datasets, we also analyze test-time interventions in OOD setting in the Appendix E.
For our evaluation we consider several image classification benchmark datasets. Our work performs an in-depth empirical analysis of the effectiveness of concept-based models in the presence of different distributional shifts simulating real-world scenarios where data is diverse.
Baselines We consider the models proposed by Koh et al. [27] as our baseline. Additionally, we compare our performance with recent concept-based models that are built on CBM [60, 20]. Due to biases introduced during automatic concept acquisition as mentioned by the authors of Yuksekgonul et al. [58], we consider it to be an unfair baseline to compare generalization properties. They also vary in the number of concepts considered which can also damage the performance and are limited by either the
presence of concept bank or application (CLIP will fail to generate concepts for TIL dataset), making an unfair comparison.
Model type	CUB	AwA2	TIL
Standard [No concepts]	82.3	96.2	51.1
±0.2	±0.1	±0.9
Independent CBM [27]	76.0	94.9	47.4
±0.4	±0.3	±1.0
Sequential CBM [27]	76.3	94.6	47.9
±0.2	±0.2	±0.9
Joint CBM [27]	80.1	95.4	49.6
±0.1	±0.1	±0.7
CEM [60]	82.5	96.2	51.3
±0.2	±0.1	±1.3
CBM-AR [20]	81.6	95.9	49.5
±0.4	±0.0	±1.0
Coop-CBM (ours)	83.6	96.6	53.4
±0.3	±0.1	±0.8
+ COL	84.1	97.0	54.2
±0.2	±0.1	±0.9
Table 1: Model accuracy on CUB, AwA2 and TIL datasets
Datasets We use Caltech-UCSD Birds-200-2011 (CUB) [55] dataset for the task of bird identification. Every dataset image contains 312 binary (eg: beak color, wing color) concepts. We additionally use Animals with Attributes 2 (AwA2) [57] dataset for the task of animal classification. The dataset contains 85 binary concepts. We use all of the subsets of the Tumor-Infiltrating Lymphocytes (TIL) [42] dataset for cancer cell classification.
The primary metric used for the downstream classification task is accuracy. We use the same metric to evaluate the effectiveness of the intervention. We first evaluate the different model performances on the test data split of respective datasets and report the task accuracy  $g_{C \to Y}$  for coop-CBM model variants.