SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Luke Gusukuma, Austin Cory Bart, Dennis Kafura, and Jeremy Ernst. Misconception-driven feedback: Results from an experimental study. In Proceedings of the 2018 ACM Conference on International Computing Education Research, pages 160-168, 2018.
[2] Noreen M Webb. Group collaboration in assessment: Multiple objectives, processes, and outcomes. Educational Evaluation and Policy Analysis, 17(2):239-261, 1995.
[3] AnHai Doan, Raghu Ramakrishnan, and Alon Y Halevy. Mass collaboration systems on the world-wide web. Communications of the ACM, 54(4):86-96, 2010.
[4] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338-5348. PMLR, 2020.
[5] Mateo Espinosa Zarlenga, Barbiero Pietro, Ciravegna Gabriele, Marra Giuseppe, Francesco Giannini, Michelangelo Diligenti, Shams Zohreh, Precioso Frederic, Stefano Melacci, Weller Adrian, et al. Concept embedding models: Beyond the accuracy-explainability trade-off. In Advances in Neural Information Processing Systems, volume 35, pages 21400-21413. Curran Associates, Inc., 2022.
[6] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems, 2022.
[7] Lucie Charlotte Magister, Pietro Barbiero, Dmitry Kazhdan, Federico Siciliano, Gabriele Ciravegna, Fabrizio Silvestri, Mateja Jamnik, and Pietro Lio. Encoding concepts in graph neural networks. arXiv preprint arXiv:2207.13586, 2022.
[8] Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302.14260, 2023.
[9] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430, 2022.
[10] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv e-prints, pages arXiv-2301, 2023.
[11] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6541-6549, 2017.
[12] Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8730-8738, 2018.
[13] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability Beyond Feature Attribution: Quantitative Testing With Concept Activation Vectors (TCAV). In International conference on machine learning, pages 2668-2677. PMLR, 2018.
[33] Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. Advances in Neural Information Processing Systems, 35:21212-21227, 2022.
[34] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. arXiv preprint arXiv:1806.07538, 2018.
[35] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[36] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021.
[37] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIR2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2023.
[38] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
[39] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574, 2023.
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
[41] Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In Osdi, volume 16, pages 265-283. Savannah, GA, USA, 2016.
[42] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[43] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. pmlr, 2015.
[44] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, Georgia, USA, 2013.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063, 2(417):1, 2012.
A.1 Software and Hardware Used
Software For this work, we evaluate all baselines using the MIT-licensed implementations of both CBMs and CEMs made public in [5]. Our implementation of IntCEM is built on top of that repository using PyTorch 1.12 [40], an open-source deep learning library with a BSD license. We incorporated CooP into this library by basing ourselves on the code released under an MIT license by Chauhan et al. [9] as part of their original publication; their code was written in TensorFlow, and as such, we converted it to PyTorch. All numerical plots and graphics have been generated using Matplotlib 3.5, a Python-based plotting library with a BSD license, while all conceptual figures were generated using draw.io, a free drawing software distributed under an Apache 2.0 license. Furthermore, when building our experiment tasks, we use TensorFlow's distribution [41] of MNIST [30] and CelebA [32] and use the processing scripts made public by Koh et al. [4] to generate our concept annotated CUB-tasks. All of our code, including config and scripts to recreate results shown in this paper, has been released as part of CEM's official public repository found at https://github.com/mateoespinosa/cem.
Resources All our non-ablation experiments were run in a shared GPU cluster with four Nvidia Titan Xp GPUs and 40 Intel(R) Xeon(R) E5-2630 v4 CPUs (at 2.20GHz) with 125GB of RAM. In contrast, all of our ablation experiments were run on a separate GPU cluster with 4x Nvidia A100-SXM-80GB GPUs, where each GPU is allocated 32 CPUs. We estimate that to complete all of our experiments, including prototyping and initial explorations, we required approximately 350 to 370 GPU hours.
A.2 Additional Details on the Gumbel-Softmax Sampler
As introduced in Section 3, expressing interventions using Equation (1) allows us to differentiate the output embedding for concept  $c_{i}$  with respect to the intervention mask  $\mu$  after an intervention has been performed. Therefore, if the output of the sampler  $p(\eta \mid \psi (\hat{\mathbf{c}}^{(t - 1)},\mu^{(t - 1)}))$  used to generate intervention  $\eta$  is differentiable with respect to its parameters  $\psi (\hat{\mathbf{c}}^{(t - 1)},\mu^{(t - 1)})$ , then we can use the gradients from  $\nabla \mathrm{CE}\big(y,f(\tilde{g} (\mathbf{x},\mu^{(T)},\mathbf{c}))\big)$ , one of the terms in  $\mathcal{L}_{\mathrm{pred}}$ , to update the parameters of  $\psi$  through standard back-propagation. This is because the intervention mask used at the end of the training trajectory is given by  $\mu^{(T)} = \mu^{(0)} + \sum_{t = 1}^{T}\eta^{(t)}$ , allowing the gradient of  $\tilde{g} (\mathbf{x},\mu^{(T)},\mathbf{c})$  to be back-propagated into the generating functions of each of the interventions  $\{\eta^{(t)}\}_{t = 1}^T$ . Therefore, when parameterising our learnable policy  $\psi$  as a DNN with weights  $\theta_{\psi}$ , this policy can better update  $\theta_{\psi}$  during training so that its interventions decrease both the prediction loss  $\mathcal{L}_{\mathrm{pred}}$  at the end of any intervention trajectory as well as its mean rollout loss  $\mathcal{L}_{\mathrm{roll}}$ .
In practice, we construct a differentiable categorical sampler  $p(\eta \mid \psi (\hat{\mathbf{c}}^{(t - 1)},\mu^{(t - 1)}))$  using a Gumbel-Softmax distribution [28]: a differentiable and continuous relaxation of the categorical distribution. To use this distribution, we begin by interpreting  $\psi (\hat{\mathbf{c}},\mu) = \omega \in (-\infty ,0)^k$  as the log-probability that each concept will be selected for the next intervention. Then, we proceed by sampling a noise vector  $\mathbf{h}\in \mathbb{R}^k$  from a Gumbel distribution such that each dimension  $h_i$  of this vector is independently sampled from  $h_i\sim \mathrm{Gumbel}(0,1)$ . This noise allows us to introduce non-determinism in our system, akin to the use of Gaussian noise in a Variational Autoencoder's reparameterisation trick [42]. Finally, the crux of the Gumbel-Softmax trick is to use this noise to shift the log-probabilities predicted by  $\psi$  and generating a sharp continuous representation of the sample through a softmax activation function:
$$
\operatorname {G u m b e l S o f t m a x} (\omega ; \tau) = \operatorname {S o f t m a x} ((\omega + \mathbf {h}) / \tau)
$$
Here,  $\tau \in [0,\infty)$  is a temperature hyperparameter indicating the sharpness of the output continuous representation of the sample (smaller values leading to samples that are closer to a one-hot representation of a vector) and  $\operatorname{Softmax}(\omega)$  is the softmax function whose  $i$ -th output is given by
$$
\operatorname {S o f t m a x} (\omega) _ {i} = \frac {e ^ {\omega_ {i}}}{\sum_ {j = 1} ^ {k} e ^ {\omega_ {j}}}
$$
In this process, one can easily sample the Gumbel noise variables  $h_i$  from  $\mathrm{Gumbel}(0,1)$  by first sampling  $u_i \sim \mathrm{Uniform}(0,1)$  and then computing  $h_i = -\log (-\log (u_i))$ . In this work, we always use a temperature of  $\tau = 1$  and make the output of the Gumbel Softmax discrete (i.e., into a one-hot
representation) using the Straight-Through estimator proposed by Jang et al. [28] which still enables continuous gradients to be back-propagated. Note, this procedure permits one to accidentally sample a previously intervened concept from  $p(\eta | \psi(\hat{\mathbf{c}}^{(t-1)}, \mu^{(t-1)}))$  (as there may be a non-zero probability assigned to a concept  $c_i$  where  $\mu_i^{(t-1)} = 1$ ); to guard against this possibility, after sampling an intervention  $\eta^{(t)}$  from the Gumbel Softmax distribution, we clamp all values of the new intervention mask  $\mu^{(t)} = \mu^{(t-1)} + \eta^{(t)}$  to be between 0 and 1 for all  $t$ .
A.3 Dataset Details
In this Appendix, we describe the datasets and tasks used in Section 4. A summary of each task's properties can be found in Table A.1 and a detailed discussion of each task can be found below.
Table A.1: Details of all tasks used in this paper. For each task, we show the number of testing samples and training samples. Furthermore, we include the shape of each input, the number of concepts provided (as well as how many groups of mutually exclusive concepts there are), and the number of output task labels. In all experiments, we subsample  $20\%$  of the training set to make our validation set.
Task	Training Samples	Testing Samples	Input Size	Concepts (k)	Groups	Output Labels
MNIST-Add	12,000	10,000	[12, 28, 28]	72	12	1
MNIST-Add-Incomp	12,000	10,000	[12, 28, 28]	54	8	1
CUB	5,994	5,794	[3, 299, 299]	112	28	200
CUB-Incomp	5,994	5,794	[3, 299, 299]	22	7	200
CelebA	13,507	3,376	[3, 64, 64]	6	6	256
MNIST-Add MNIST-Add is a visual arithmetic task based on the MNIST [30] dataset. Each sample  $\mathbf{x} \in [0,1]^{12 \times 28 \times 28}$  in this task is formed by 12 digit "operands", where the  $i$ -th channel  $\mathbf{x}_{i,::}$  of  $\mathbf{x}$  contains a grayscale zero-to-nine digit representing the  $i$ -th operand. We construct this dataset such that some operands are inherently harder to predict. To do so, we (i) constrain each of the first four operands to be at most 2, (ii) constrain each of the four middle operands to at most 4, and (iii) constrain each of the last four operands to be at most 9.
Each sample  $\mathbf{x}$  in this task is annotated with a binary label  $y$  corresponding to whether the sum of the digits in all 12 operands is at least half the maximum attainable (i.e., at least 30). We construct concept annotations  $\mathbf{c} \in \{0,1\}^{72}$  for  $\mathbf{x}$  by providing the one-hot encoding representations of each operand in  $\mathbf{x}$  as an annotation. Because the first four operands can be at most 2, and the next 4 and 8 operands can be at most 4 and 9, respectively, this results in a total of  $4 \times 3 + 4 \times 5 + 4 \times 10 = 72$  concepts organised into groups of 12 mutually exclusive concepts. To construct MNIST-Add's training set, we generate 12,000 training samples by randomly selecting all 12 operands of each sample from MNIST's training set. Similarly, to construct MNIST-Add's test set, we generate 10,000 test samples by randomly selecting all 12 operands of each sample from MNIST's test set.
MNIST-Add-Incomp MNIST-Add-Incomp is a visual arithmetic task that modifies MNIST-Add so that its set of concept annotations is incomplete with respect to the task label. This is done by randomly selecting 8 out of the 12 groups of concepts in MNIST-Add and providing the concepts corresponding to those 8 groups to all models at train-time (making sure the same group of 8 groups is selected across all baselines).
CUB CUB is a visual bird-recognition task where each sample  $\mathbf{x} \in [0,1]^{3 \times 299 \times 299}$  represents an RGB image of a bird obtained from the CUB dataset [31]. Each of these images comes with a set of attribute annotations - e.g., "beak_type" and "bird_size" - as well as a downstream label  $y$  representing the classification of one of 200 different bird types. In this work, we construct concept annotations  $\mathbf{c}$  for this dataset using the same  $k = 112$  bird attributes selected by Koh et al. [4] in their CBM work. These 112 concepts can be grouped into 28 groups of mutually exclusive concepts (e.g., "white wing", "black wing", etc). Following [4], we preprocess all of the images in this dataset by normalising, randomly flipping, and randomly cropping each image during training. We treat this dataset as a representative real-world task where one has access to a complete set of concept annotations at train-time.
CUB-Incomp CUB-Incomp is a concept-incomplete variant of the CUB task constructed by randomly selecting  $25\%$  of all concept groups in CUB and providing train-time annotations only for those concepts. For fairness, we ensure that the same groups are selected when training all baselines, resulting in a dataset with 7 groups of mutually exclusive concepts and 22 concepts.
CelebA The CelebA task is generated from samples in the Celebrity Attributes Dataset [32] using the task and concept annotations defined in [5]. For each image  $\mathbf{x}$  in CelebA, we first select the 8 most balanced attributes  $[a_1, \dots, a_8]$  from the 40 binary attributes available; we then build a vector of binary concept annotations  $\mathbf{c}$  using only the 6 most balanced attributes  $[a_1, \dots, a_6]$ , meaning that attributes  $a_7$  and  $a_8$  are not provided as concept annotations. To simulate a concept-incomplete task, we construct a task label  $y$  via the base-10 representation of the number formed by  $[a_1, \dots, a_8]$ . This results in a total of  $l = 2^8 = 256$  classes. As in [5], we downsample every image to  $(3,64,64)$  and randomly subsample images in CelebA's training set by selecting every  $12^{\mathrm{th}}$  sample.
A.4 Model Architecture and Baselines Details
In this section, we describe the architectures and training hyperparameters used in our experiments.
A.4.1 Model Architectures and Hyperparameters
To ensure a fair comparison across methods, we strive to use the exact same architectures for the concept encoder  $g$  and label predictor  $f$  across all baselines. Moreover, when possible, we follow the architectural choices selected in [5] to ensure that we are comparing our own IntCEM architecture against highly fine-tuned CEMs. The only tasks where this is not possible, as they were not a part of the original CEM evaluation, are the MNIST-based tasks. Below, we describe our architectural choices for  $f$  and  $g$  for each of our tasks. Moreover, for all tasks and models we include the choices of loss weights  $\lambda_{\mathrm{roll}}$  and  $\lambda_{\mathrm{concept}}$  where, for notational simplicity, we use  $\lambda_{\mathrm{concept}}$  to indicate the strength of the concept loss in both CEM-based models as well as in jointly trained CBMs. Finally, as in [5], for both CEMs and IntCEMs we use an embedding size of  $m = 16$ , a RandInt probability  $p_{\mathrm{int}} = 0.25$  and a single linear layer for all concept embedding generators  $\{(\phi_i^-,\phi_i^+)\}_{i = 1}^k$  and the shared scoring model  $s$ .
MNIST-Based Tasks For all MNIST-based tasks, the concept encoder  $g$  begins by processing its input using a convolutional neural network backbone with 5 convolutional layers, each with sixteen  $3 \times 3$  filters and a batch normalisation layer [43] before their nonlinear activation (a leaky-ReLU [44]), followed by a linear layer with  $g_{\mathrm{out}}$  outputs. When working with CEM-based models, the output of this backbone will be a  $g_{\mathrm{out}} = 128$ , from which the concept embedding generators  $\{(\phi_i^{-},\phi_i^{+})\}_{i = 1}^k$  can produce embeddings  $\{(\hat{\mathbf{c}}_i^{-},\hat{\mathbf{c}}_i^{+})\}_{i = 1}^k$  and from which we can construct  $g$ . Otherwise, when working with CBM-based models, the output of this backbone will be the number of training concepts  $g_{\mathrm{out}} = k$ , meaning that  $g$  will be given by the backbone's output. For all models, we use as a label predictor  $f$  a Multi-Layer Perceptron (MLP) with hidden layers  $\{128,128\}$  and leaky ReLU activations in between them. We use an MLP for  $f$  rather than a simple linear layer, as it is commonly done, because we found that CBM-based models were unable to achieve high accuracy on these tasks unless a nonlinear model was used for  $f$  (the same was not observed in CEM-based model as they have higher bottleneck capacity).
For CEMs, IntCEMs, and Joint CBM-Logit models, we use  $\lambda_{\mathrm{concept}} = 10$  across both MNIST-based tasks. This is only different for Joint CBM-Sigmoid were we noticed instabilities when setting this value to 10 (leading to large variances) and reduced it to  $\lambda_{\mathrm{concept}} = 5$  (reducing it lower than this value seem to decrease validation performance). All models are trained using a batch size of  $B = 2048$  for a maximum of  $E = 500$  epochs. To avoid models learning a majority class classifier, we weight each concept's cross entropy loss by its frequency and we use a weighted binary cross entropy loss for the task loss. Finally, for both MNIST tasks, the best performing IntCEM, determined by the model with the highest area under the intervention vs validation accuracy curve when  $\lambda_{\mathrm{roll}}$  varies through  $\{5,1,0.1\}$ , was obtained when  $\lambda_{\mathrm{roll}} = 1$ .
CUB-Based Tasks As with our MNIST-based tasks, for all CUB-based tasks we construct a concept encoder  $g$  by passing our input through a backbone model formed by a ResNet-34 [45] whose output linear layer has been modified to have  $g_{\mathrm{out}}$  activations. The output of this backbone is set to  $g_{\mathrm{out}} = k$
for all models and it is used to construct  $g$  in the same way as described for the MNIST-based tasks. For all models, we use as a label predictor  $f$  a linear layer.
For CEMs, IntCEMs, and Joint CBM-Logit models, we use  $\lambda_{\mathrm{concept}} = 5$  across both CUB-based tasks. The only exception is in CUB-Incomp, where we noticed a high variance in Joint CBM-Sigmoid when setting this value to 5 and reduced it to  $\lambda_{\mathrm{concept}} = 1$  as a response. All models are trained using a batch size of  $B = 256$  for a maximum of  $E = 300$  epochs and using a weight decay of  $4\times 10^{-5}$  as in [4]. As in the MNIST-based models, we scale each concept's binary cross entropy loss by a factor inversely proportional to its frequency to help the model learn the more imbalanced concepts. Finally, for both CUB tasks, the best performing IntCEM was obtained when  $\lambda_{\mathrm{roll}} = 1$ .
CelebA We use the same architecture and training configuration as in the CUB-based tasks for all CelebA models. The only differences between models trained in the CUB-tasks and those trained in CelebA, is that we set the batch size to  $B = 512$  and train for a maximum of  $E = 200$  epochs while setting the concept loss weight  $\lambda_{\mathrm{concept}} = 1$  for all joint CBMs and CEM-based models. When iterating over different values of  $\lambda_{\mathrm{roll}}$  for IntCEM, we found that  $\lambda_{\mathrm{roll}} = 5$  works best for this task.
A.4.2 Training Hyperparameters
In our experiments, we endeavour to provide all models with a fair computational budget at train-time. To that end, we set a large maximum number of training epochs and stop training early based on each model's own validation loss. Here, we stop early if a model's validation loss does not decrease for 15 epochs. We monitor the validation loss by sampling  $20\%$  of the training set to make our validation set for each task. This validation set is also used to select the best-performing models whose results we report in Section 4. We always train with stochastic gradient descent with batch size  $B$  (selected to well-utilise our hardware), initial learning rate  $r_{\mathrm{initial}}$ , and momentum 0.9. Following the hyperparameter selection in reported in [5], we use  $r_{\mathrm{initial}} = 0.01$  for all tasks with the exception of the Ce1ebA task where this is set to  $r_{\mathrm{initial}} = 0.005$ . Finally, to help find better local minima in our loss function during training, we decrease the learning rate by a factor of 0.1 whenever the training loss plateaus for 10 epochs.
A.4.3 IntCEM-Specific Parameters
As discussed in Section 3, IntCEM requires one to select a prior for its initial trajectory length and intervention mask, as well as an architecture to be used for the intervention policy model  $\psi$ . Below we describe who we set these values for all of our experiments. Later, we discuss the effect of some of these hyperparameters on our results in our ablation studies in Appendix A.6.
Prior Selection In our experiments, to allow our prior to imitate how we intervene on IntCEMs at test-time, we construct  $p(\mu)$  by partitioning  $\mathbf{c}$  into sets of mutually exclusive concepts and selecting concepts belonging to the same group with probability  $p_{\mathrm{int}} = 0.25$ . Complementing this prior, we construct the intervention trajectory horizon's prior  $p(T)$  using a discrete uniform prior  $\mathrm{Unif}(1, T_{\max})$  where  $T_{\max}$  is annealed within [2, 6] growing by a factor of 1.005 every training step. This was done to help the model stabilise its training by first learning meaningful concept representations before employing significant resources in improving its learnable policy  $\psi$ .