SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Coop-CBM improves task accuracy The evaluation of the performance of a model is based on the final prediction accuracy. In Table 1, we compare the performance of coop-CBM against other baseline models. We first observe that CBMs experienced a significant drop in performance compared
to the standard model that did not use concepts. Our proposed model, coop-CBM with immediate label prediction achieves state of art accuracy and statistically significant results on every dataset. We have observed a significant improvement in the performance of the CUB  $(+1.8\%)$  increase from the standard model) and TIL  $(+3.1\%)$  increase from the standard model) datasets. This finding is important as it suggests that machine learning models can be designed to overcome the high accuracy vs interpretability tradeoff. Our performance can be further boosted by introducing orthogonality among different concepts. It must be noted that CUB is a fairly densely annotated dataset, which might not always be realistic, hence we also benchmark our model by training on a fraction of concept sets. We also observe a similar trend in results in concept-scarce settings (see Appendix D.2). This suggests that our method is robust to concept selection, which can be beneficial in scenarios where the number of available concepts is limited or expensive to obtain.
COL improves concept accuracy Previously in Table 1, we observed that adding concept orthogonal loss to coop-CBM improved its downstream accuracy, in Table 2, we study the impact of adding COL to baseline concept models. Our experiments show that adding COL improves the concept accuracy by a significant margin, especially in joint CBM, CEM, and CBM-AR settings. A known pitfall of CBMs is, concept leakage [33] could be potentially prevented by increasing the separation between their concept representations. By maximizing the inner product between the concept embeddings of different concepts, we can ensure that each concept is represented in a separate and distinct direction in the embed
ding space. This helps in preventing the models from relying on irrelevant concepts. Further to intuitively understand the differences in the concept representation of our model, we compute the histogram for the predicted concept logits. From Figure 3 we see that Coop-CBM+COL minimizes the concept loss better (with help from the auxiliary loss which aids representation learning), which results in clearer separation of logits.
Model type	w/o COL	w COL
Independent CBM [27]	96.6 ±0.0	97.2 ±0.1
Sequential CBM [27]	96.6 ±0.0	97.2 ±0.1
Joint CBM [27]	93.2 ±0.1	96.4 ±0.2
CEM [60]	94.8 ±0.2	97.0 ±0.1
CBM-AR [20]	94.2 ±0.1	96.8 ±0.1
Coop-CBM (ours)	93.9 ±0.2	97.3 ±0.2
Clipping concept values to avoid concept leakage Further, we employ clipping of concept prediction proposed by Mahinpei et al. [33] to further mitigate information leakage in 3 through two experiments. For the first experiment, we trained the model by clipping the predicted concept values to "hard" labels. Second, we trained the model as we have described earlier in the paper (using soft labels) and evaluated the test set by clipping to "hard" labels. From the above experiments, we conclude that the
model is able to learn a good representation of the concepts without necessarily leaking information.
Table 2: Concept prediction accuracy for each model before and after adding COL for CUB dataset
Model	CUB	TIL
Std	Exp1	Exp2	Std	Exp1	Exp2
Coop-CBM	83.6 ±0.3	82.1 ±0.2	83.0 ±0.5	53.4 ±0.8	52.6 ±0.9	52.8 ±1.1
+COL	84.1 ±0.2	83.2 ±0.2	84.0 ±0.6	54.2 ±0.9	53.5 ±0.9	54.1 ±0.8
Table 3: Testing for information leakage in our proposed model. Std - standard conditions when joint probabilities are learned to predict the final task, no clipping. Exp1 - During training, we clipped the predicted concept values to "hard" labels. Exp2 - During the evaluation, we clipped the predicted concept values to "hard" labels.
Accounting for Human uncertainty for interventions As discussed earlier, higher concept accuracy also improves the test-time interventions as seen in Figure 2. While other works used concept weights and uncertainty as metrics to select the interventions, our work introduces a more realistic setting by introducing human uncertainty additionally. The previous works do not account for human error or certainty. Albeit human uncertainty is difficult to quantify since it is often subjective, we use the concept visibility data in the CUB dataset to quantify the confidence score, SCS. The
Figure 2: L→R:1) Accuracy vs intervention graph using joint CBM while including supervisor uncertainty. 2) Accuracy vs intervention graph in presence of incorrect interventions by the supervisor using joint CBM. 3) Comparing different model's random interventions on the TIL dataset. 4) Comparing different model's random interventions on the CUB dataset.
annotators rank visibility from  $1 - 4$ , 1 being 'not visible' and 4 being 'definitely visible'. Figure 2 presents an accuracy curve with an increasing number of interventions on the vanilla CBM model during test-time for proper and fair comparison to [49]. It suggests that considering SCS leads to the most meaningful interventions. On intervention comparison between different concept-based models, we observe that coop-CBM leads to significant interventions that improve downstream performance, especially in TIL dataset where both image descriptive and non-descriptive concepts are present.
In summary, the coop-CBM model and the addition of the concept orthogonal loss help to improve both task and concept accuracy without sacrificing interpretability, which was a common tradeoff in previous methods. This result demonstrates the potential for concept-based models to be more effective in human-AI interactions, especially in domains where expert intervention and interpretability are critical such as healthcare.
5.1 Performance under distribution shifts
5.1.1 Background spurious correlations
Shortcut-based biases [17] exist in many datasets, where deep learning can easily learn spurious features. In the presence of shortcuts, the model can learn to use spurious features to approximate the true distribution of the labels, as opposed to learning core features. It can be of particular interest to evaluate the performance of concept-based models in the presence of shortcuts. Furthermore, using explainable concepts to facilitate human-model interaction could help reduce the impact of these biases.
The shortcut we consider here is a spurious correlation to the background color in CUB dataset. Loosely following the experimental setup of [5], we correlate the background of a species of bird to its corresponding label. For the dataset, we segment the bird images and add a colored background to all of the images.
Each class here is correlated to a randomly generated color background with a probability of  $80\%$  for the train set. The in-domain test set contains images with similar color background probability as the train set while the correlation in the out-domain test set is reduced to  $30\%$ . We also consider the hair color-based shortcut induced in Large-scale CelebFaces Attributes (CelebA) dataset where we focus on gender classification between males and females. We aggregate and then construct a modified version of CelebA that learns the shortcut of blonde hair color with women similar to [3].
Model	CUB	CelebA
In	Out	In	Out
Standard	86.9 ±0.3	27.7 ±2.8	96.5 ±0.5	76.3 ±1.8
Independent CBM [27]	81.1 ±0.5	30.9 ±2.3	93.8 ±0.6	76.2 ±1.9
Sequential CBM [27]	81.2 ±0.5	30.8 ±2.2	93.8 ±0.6	76.1 ±1.9
Joint CBM [27]	83.6 ±0.4	32.9 ±3.2	94.7 ±0.5	80.1 ±1.7
CEM [60]	84.0 ±0.4	34.0 ±2.9	95.2 ±0.4	81.0 ±1.7
CBM-AR [20]	82.6 ±0.6	33.6 ±2.5	95.9 ±0.5	79.7 ±2.0
Coop-CBM(ours)	84.9 ±0.3	35.4 ±2.5	95.4 ±0.6	81.3 ±1.9
+ COL	85.8 ±0.4	36.2 ±2.7	95.9 ±0.7	82.0 ±2.7
Table 4: Accuracy of different models under distributional shift - background spurious correlation.
In Table 4 we report the in-domain and out-domain accuracies of our baseline models and proposed coop-CBM for CUB and CelebA datasets. Our results show the robustness of coop-CBM and COL to background spurious correlations achieving state-of-art results among the concept-based models. We observe an interesting trend for the CUB dataset. The concept-based models may have lower accuracy standard model in the in-domain setting but in the out-domain setting, all of the baseline models including ours have higher accuracy than the standard model. While we do not observe a similar definitive trend in CelebA dataset, it is evident that most of the models, CEM, joint CBM and coop-CBM are superior when the test data contains more images of men with blonde hair. This experiment suggests that the concept-based models are able to generalize better to unseen data outside the training distribution, which can be attributed to their ability to learn more invariant features through the explicit conditioning and learning of concepts. This is a promising result as it indicates that concept-based models, particularly coop-CBM may be more robust to extreme distribution shifts.
In Appendix E, we evaluate the influence of interventions in out-domain settings for the biased CUB dataset. Realistically, the user may fine-tune the model to distributional shift after deployment to improve the predictive performance, but in the circumstances where the labeled shifted dataset is absent, we suspect that interventions could greatly help if they are cheaper to obtain. In such cases, we especially argue that SCS score benefits the intervention selection.
5.1.2 Image corruptions
Distributional shift can occur due to various factors such as changes in the data collection process, changes in the environmental conditions under which the data is collected, or even changes in the underlying population that the data represents. The Corruptions dataset [22, 35] is a collection of image corruptions designed to evaluate the robustness of computer vision models. Some of the corruptions are realistic OOD settings such as snow, while others could be less likely in nature such as impulse noise. We introduce 7 such corruptions (Gaussian Noise, Blur, Zoom Blur, Snow, Fog, Brightness and Contrast) onto CUB, AwA2 and TIL datasets. We report detailed results on CUB and average accuracy for AwA and TIL datasets in Table 5.
Model	CUB	AwA2	TIL
1	2	3	4	5	6	7	Avg
Standard	65.2±0.3	62.0±0.7	56.7±0.5	60.1±0.2	74.1±0.8	72.0±0.6	53.4±0.4	63.3	79.3	38.4
Independent CBM [27]	61.4±0.6	62.1±0.4	57.1±0.8	59.7±0.3	73.6±0.7	69.8±0.5	52.9±0.2	62.3	78.4	36.6
Sequential CBM[27]	60.3±0.5	61.9±0.2	56.5±0.6	58.5±0.4	72.7±0.8	71.2±0.3	52.0±0.7	61.8	78.2	36.3
Joint CBM[27]	63.1±0.4	64.5±0.8	57.4±0.3	60.6±0.7	73.8±0.5	72.3±0.2	51.8±0.6	63.4	79.1	37.1
CEM [60]	66.1±0.7	61.4±0.5	57.3±0.2	61.0±0.6	74.2±0.4	71.6±0.8	53.4±0.3	63.6	79.7	38.5
CBM-AR [20]	64.8±0.6	61.7±0.4	57.2±0.8	59.4±0.3	73.3±0.7	70.4±0.5	52.9±0.2	62.8	79.6	36.8
Coop-CBM (ours)	67.2±0.5	63.5±0.2	59.0±0.6	60.9±0.4	75.4±0.8	73.2±0.3	53.3±0.7	64.6	80.9	40.6
+ COL	67.8±0.4	63.9±0.8	58.7±0.3	61.5±0.7	75.8±0.5	73.4±0.2	53.3±0.6	64.9	81.5	40.9
Table 5: Comparison of concept-based models on image corruptions on CUB, AwA2 and TIL datasets.
Based on our evaluation of distributional shifts in Table 5, we found that incorporating an auxiliary loss in the form of a multi-task setting can help CBMs achieve competitive downstream accuracy overall. We notice that the standard black box models although do perform better in the presence of "contrast" corruption in CUB dataset. Although coop-CBM outperforms other explanation models. In general, it is interesting to note that coop-CBM has better generalization performance in the presence of spurious correlations than in the presence of corruption. This may be because spurious correlations are introduced as a shortcut within the training data, and concept-based models are designed to learn invariant features that are robust to such shortcuts. Regardless, coop-CBM's superior generalization property across different corruptions suggests that the model is able to effectively filter out irrelevant information/noise in the data.
5.1.3 Noise concept correlation
In Section 5.1.1 and Section 5.1.2, we conducted experiments concerned with distribution shifts in image space, in this section, we introduce the evaluation of CBMs by simulating distribution shifts in the concept space. To investigate the potential risks of spurious correlations in concept models, we introduced Gaussian noise to the binary concepts. By altering the standard deviation  $(\sigma)$  of the Gaussian noise, we effectively correlated the shortcut (here noise level) with the image through the concept. To simulate a more realistic setting, instead of adding distinct noise  $\sigma$  for each class species, we aggregate random groups of species and add same  $\sigma$  to them. In our experiment for the CUB dataset, we add 10 different levels of noise (simulated by  $\sigma$ ) to groups of 20 species labels
Model type	CUB	AwA2
Independent CBM [27]	69.7 ±0.6	80.1 ±0.4
Sequential CBM [27]	69.6 ±0.5	80.3 ±0.2
Joint CBM [27]	71.0 ±0.4	81.3 ±0.4
CEM [60]	71.2 ±0.6	81.9 ±0.3
CBM-AR [20]	71.1 ±0.4	81.5 ±0.3
Coop-CBM (ours)	71.9 ±0.6	82.5 ±0.2
+ COL	72.7 ±0.4	83.2 ±0.2
Table 6: Accuracy of different models under distributional shift-noise concept correlation
(200 total classes). For AwA2, we create groups of 10 classes. This approach allowed us to simulate the possibility of introducing unintended correlations between the concepts and the images. By studying the effects of these correlations on the performance of the concept models, we gain insights into the robustness and reliability of the models in handling contaminated concepts. We observe that by introducing a separation between different concepts through COL, our model performs significantly better than the rest of the baselines.
6 Future work and Limitations
In this work, we introduced coop-CBM, a novel concept-driven method to balance AI model interpretability and accuracy. We utilized the Concept Orthogonal Loss (COL) to improve concept learning and applied coop-CBM to various datasets, achieving better generalization, robustness to spurious correlations, and improved accuracy-interpretability trade-offs.
However, our approach has limitations. It relies on labeled concept vectors, which can be challenging in domains with limited annotations and face biases in concept annotation methods. A potential future work could be to extend it to methods that do not assume concept label [58, 38]. Further, we used accuracy as a metric to evaluate concept leakage, in the future, it would be interesting to explore other metrics beyond the accuracy of concept prediction. A future extension of COL could be to evaluate which concepts should be explicitly orthogonalized. We recognize that our model has a few hyperparameters to be optimized. Furthermore, our model assumes that learned concepts align closely with human notions, but this alignment isn't always perfect, affecting comprehensibility. Future research could improve the accuracy of concept-based models by providing meaningful explanations and incorporating additional evaluation metrics. Another potential direction could be to assess the mutual information and therefore establish theoretical grounding to describe the superior performance of coop-CBM.
In this work we proposed two significant contributions to the paradigm of concept-based models. First, we introduced a multi-task model that predicts an intermediary task label along with concept prediction. This is particularly helpful when dense and relevant concept annotation is absent, such as in TIL dataset. Second, we introduced orthogonality constrain in the concept representation space during training via concept orthogonal loss. This loss increases inter-concept separation and decreases intra-concept distance. For both of our proposed methods, we perform extensive experiments on diverse datasets and different distributional shifts. We observe that the bottleneck layer before the final prediction enables concept-based models to exhibit robustness to spurious correlations in the background. Coop-CBM along with COL achieves state-of-art performance for both task accuracy and concept accuracy. Our work indicates that coop-CBM and COL have a strong ability to adapt and generalize well across diverse datasets and real-world scenarios.
We would like to thank Vincent Michalski and the reviewers for engaging in discussions on the earlier version of the paper. The authors would like to thank Google, CIFAR (Canadian Institute for Advanced Research) and NSERC (Natural Sciences and Engineering Research Council of Canada) for supporting and funding the research and Digital Research Alliance of Canada for the compute support.
[1] Julius Adebayo, Justin Gilmer, Ian J. Goodfellow, and Been Kim. Local explanation methods for deep neural networks lack sensitivity to parameter values. ArXiv, abs/1810.03307, 2018.
[2] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian Goodfellow, Moritz Hardt, and Been Kim. Sanity checks for saliency maps. Advances in neural information processing systems, 31, 2018.
[3] Mohammed Adnan, Yani Ioannou, Chuan-Yung Tsai, Angus Galloway, HR Tizhoosh, and Graham W Taylor. Monitoring shortcut learning using mutual information. arXiv preprint arXiv:2206.13034, 2022.
[4] David Alvarez-Melis. Self-explaining neural networks. 2018.
[5] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[6] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. End-to-end attention-based large vocabulary speech recognition. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 4945-4949. IEEE, 2016.
[7] Alexander Binder, Grégoire Montavon, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Layer-wise relevance propagation for neural networks with local renormalization layers. In ICANN, 2016.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[9] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. arXiv preprint arXiv:2007.07375, 2020.
[10] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430, 2022.
[11] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nat. Mach. Intell., 2:772-782, 2020.
[12] Katherine Maeve Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 869–889, 2023.
[13] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto Novoa, and James Y Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. Advances in Neural Information Processing Systems, 35:18157-18167, 2022.
[14] Devleena Das, Sonia Chernova, and Been Kim. State2explanation: Concept-based explanations to benefit agent learning and user understanding. arXiv preprint arXiv:2309.12482, 2023.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
[50] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818-2826, 2016.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
[52] Asher Trockman and J Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104.07167, 2021.
[53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[54] Eugene Vorontsov, Chiheb Trabelsi, Samuel Kadoury, and Chris Pal. On orthogonality and learning recurrent networks with long term dependencies. In International Conference on Machine Learning, pages 3570-3578. PMLR, 2017.
[55] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge J. Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
[56] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pages 499-515. Springer, 2016.
[57] Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41:2251-2265, 2019.
[58] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. arXiv preprint arXiv:2205.15480, 2022.
[59] Éloi Zablocki, Hédi Ben-Younes, Patrick Pérez, and Matthieu Cord. Explainability of deep vision-based autonomous driving systems: Review and challenges. International Journal of Computer Vision, 130(10):2425-2452, 2022.
[60] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056, 2022.
[61] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv preprint arXiv:2301.10367, 2023.
[62] Yaohui Zhu, Weiqing Min, and Shuqiang Jiang. Attribute-guided feature learning for few-shot image recognition. IEEE Transactions on Multimedia, 23:1200-1209, 2020.
A More Related Works
Explainability Post-hoc explanations aim to provide insight into why a particular prediction or decision was made by the model. These may be in the form of heatmaps, rule sets, or feature importance scores. Global explanations [24] aim to learn the overall generic features in order to explain black-box models by the use of explainators. These explainators are often simple Machine Learning (ML) models. Local explanation methods [1] exhibit explainability by exploring the intrinsic workings of the neural network. This can be executed by propagating layer-wise feature relevance [7]. Selvaraju et al. [47] propose a gradient-based saliency mapping technique that perturbs inputs by injecting noise. Deconstructing the nonlinear model to simpler sub-functions [36] was another proposed method to interpret model's predictions. Nohara et al. [37] propose a difference-to-reference approach to feature importance estimation. Popularly, saliency heatmaps for feature importance visualization have been used [2]. Post hoc explanations although do not address the fundamental issue of model transparency, as they are generated externally to the model and may not reflect the true reasoning of the model's internal mechanisms which can be addressed by concept-based models like ours.