SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intervention Policy Architecture For all tasks, we parameterise IntCEM's learnable policy  $\psi$  using a leaky-ReLU MLP with hidden layers \{128, 128, 64, 64\} and Batch Normalisation layers in between them. When groups of mutually exclusive concepts are known at train-time (as in CUB-based tasks and MNIST-based tasks), we let  $\psi$ 's output distribution operate over groups of mutually exclusive concepts rather than individual concepts. This allows for a more efficient training process that better aligns with how these concepts will be intervened on at test-time. Finally, when training IntCEMs we use global gradient clipping to bound the global norm of gradients to 100. This helps us avoid exploding gradients [46] as we backpropagate gradients through our intervention trajectory.
A.5 Computational Resource Usage
As mentioned in Sections 4 and 5, IntCEMs bear an additional overhead in their training times compared to traditional CEMs due to their need to sample intervention trajectories during training. In this section, we first study how training times vary across all of our baselines. Then we show that
IntCEM's training time overhead is amortised in practice if one considers the efficiency of IntCEM's learnt policy  $\psi$  over existing state-of-the-art intervention policies.
Table A.2: Average wall-clock training times per epoch, epochs until convergence, and the number of parameters for baselines in the CUB task. All of these results were obtained by training 5 differently-Initialised models for each baseline using the same hardware as specified in Appendix A.1.
Model	Epoch Training Time (s)	Epochs To Convergence	Number of Parameters
Joint CBM-Sigmoid	37.75 ± 1.99	225.67 ± 27.18	2.14 × 107
Joint CBM-Logit	36.76 ± 0.70	210.67 ± 22.48	2.14 × 107
Sequential CBM	31.08 ± 1.17	284.67 ± 28.67	2.14 × 107
Independent CBM	31.07 ± 1.16	284.67 ± 28.67	2.14 × 107
CEM	40.77 ± 0.48	174.00 ± 34.88	2.57 × 107
IntCEM	64.89 ± 0.44	224.00 ± 26.77	2.60 × 107
In Table A.2 we show the wall-clock times of a single training epoch for each of our baselines as well as the number of epochs it took the model to converge (as determined by our early stopping procedure defined in Appendix A.4). Our results indicate that, compared to traditional CEMs, IntCEMs take approximately  $60\%$  longer than CEMs per training epoch while also taking approximately  $\sim 20\%$  longer to converge (in terms of epochs). Nevertheless, we underscore three main observations from these results. First, the results in Table A.2 are very highly implementation-and-hardware-dependent, meaning that they should be taken with a grain of salt as they could vary widely across different implementations, batch sizes, and machines. Second, although on average we observe that IntCEMs take more epochs to converge than traditional CEMs, the variances we observe across different random initialisations are large. This means that these differences are not likely to be statistically significant; in fact, we observe some runs where IntCEMs converged faster than CEMs. Finally, as seen in Table A.3, these extra training costs may be amortised in practice when one considers that IntCEM's learnt policy is much faster than an equivalent competitive intervention policy like CooP. This implies that if these models are expected to be intervened frequently in practice, the overhead spanning from IntCEM's training procedure may result in better amortised computational resource usage over CEMs intervened on with the CooP policy.
Table A.3: Average wall-clock times taken to query the dynamic policies for trained IntCEMs across different tasks. Notice that all values are shown in microseconds (ms) and that these results can be highly hardware dependent, as they can be heavily affected if one runs a policy in a machine with GPU access. Nevertheless, all of these results were obtained by running all intervention policies using the same hardware as specified in Appendix A.1 and averaging over the entire test set.
Dataset	Random (ms)	Learnt Policy ψ (ms)	CooP (ms)	UCP (ms)	Skyline (ms)
MNIST-Add	0.38 ± 0.02	0.37 ± 0.02	0.56 ± 0.03	0.40 ± 0.04	0.44 ± 0.03
MNIST-Add-Incomp	0.33 ± 0.02	0.32 ± 0.01	0.43 ± 0.04	0.31 ± 0.00	0.35 ± 0.01
CUB	2.91 ± 0.13	2.90 ± 0.20	4.77 ± 0.10	2.92 ± 0.09	4.54 ± 0.09
CUB-Incomp	3.19 ± 0.39	3.10 ± 0.35	5.38 ± 0.85	3.08 ± 0.51	4.73 ± 0.27
CelebA	0.70 ± 0.01	0.76 ± 0.04	0.73 ± 0.02	0.68 ± 0.06	0.76 ± 0.06
A.6 Exploring IntCEM's Hyperparameter Sensitivity
In this section, we explore how our results reported in Section 4 change as we vary several of IntCEM's hyperparameters, specifically,  $\lambda_{\mathrm{roll}}$ ,  $\lambda_{\mathrm{concept}}$ ,  $\gamma$ , and  $p_{\mathrm{int}}$  and  $T_{\mathrm{max}}$ .
We begin by exploring its sensitivity to its two loss weights,  $\lambda_{\mathrm{roll}}$  and  $\lambda_{\mathrm{concept}}$ , and show that, although such parameters have an impact mostly on the unintervened accuracy of IntCEMs, our model is able to outperform CEMs once interventions are introduced throughout multiple different values for such weights. Then, we investigate how sensitive IntCEM is to its intervened prediction loss scaling factor  $\gamma$  and show that its performance varies very little as  $\gamma$  is modified. Finally, we explore how the hyperparameters we selected for IntCEM's prior ( $p_{\mathrm{int}}$  and  $T_{\mathrm{max}}$ ) affect its performance and describe similar results to those observed when varying IntCEM's loss weight hyperparameters.
For all the results we describe below, we train IntCEMs on the CUB task while fixing  $\gamma = 1.1$ ,  $\lambda_{\mathrm{concept}} = 5$ ,  $\lambda_{\mathrm{roll}} = 1$ ,  $p_{\mathrm{int}} = 0.25$ , and  $T_{\mathrm{max}} = 6$  unless any of these hyperparameters is the target
(a)  $\lambda_{\mathrm{concept}}$  ablation study in CUB.
(b)  $\lambda_{\mathrm{roll}}$  ablation study in CUB.
Figure A.1: Ablation studies in CUB for IntCEM's  $\lambda_{\mathrm{roll}}$  and  $\lambda_{\mathrm{concept}}$  loss weights. Notice that although initial unintervened performance may vary across different hyperparameters, in all instances we observe that IntCEMs outperform traditional CEMs (pink diamonds) after being intervened on.
of our ablation. Furthermore, we include as a reference the accuracy of a CEM trained in this same task. Finally, when showing intervention performance, all interventions in IntCEMs are made following their learnt policy  $\psi$  while interventions in CEMs are made following the CooP policy (the best-performing policy for CEMs based on our experiments in Section 4).
A.6.1 Rollout and Concept Loss Weights Ablations
In Figure A.1 we show the results of varying  $\lambda_{\mathrm{roll}}$  and  $\lambda_{\mathrm{concept}}$  in  $\{0, 0.1, 1, 5, 10\}$  in the CUB task. Our ablation studies show that IntCEM's initial unintervened performance can vary as these two parameters are changed, particularly when  $\lambda_{\mathrm{concept}}$  is small or  $\lambda_{\mathrm{roll}}$  is large. Similarly, we observe that the average concept AUC of IntCEM decreases as  $\lambda_{\mathrm{concept}}$  is very small, as one would expect by  $\lambda_{\mathrm{concept}}$  's definition, and is affected as we increase the value of  $\lambda_{\mathrm{roll}}$ , as described in Section 4.1. Nevertheless, and perhaps more importantly, we notice that for all hyperparameter configurations, IntCEM outperforms a traditional CEM's intervened task accuracy within five interventions. This is observed even when  $\lambda_{\mathrm{roll}} = 0$  or  $\lambda_{\mathrm{concept}} = 0$  and suggests that our model's preconditioning to intervention receptiveness is robust to changes in its hyperparameters. Moreover, we highlight that IntCEM's sensitivity to  $\lambda_{\mathrm{concept}}$  is not inherent to our architecture but rather inherent to any jointly trained CBMs and CEMs, as shown by Koh et al. [4] in their original CBM evaluation.
Looking at our  $\lambda_{\mathrm{roll}}$  ablation study, we observe that IntCEM's concept accuracy and unintervened task accuracy seem to decrease as  $\lambda_{\mathrm{roll}}$  increases. We hypothesise that happens as our model has a greater incentive to depend on interventions, and therefore can afford to make more concept and task mispredictions in the absence of interventions. Furthermore, we observe that the variance in IntCEM's unintervened task performance and concept performance as  $\lambda_{\mathrm{roll}}$  changes are less significant than that observed when  $\lambda_{\mathrm{concept}}$  changes, in fact achieving competitive accuracies for all values except when  $\lambda_{\mathrm{roll}} = 10$ . This suggests that when fine-tuning IntCEM's loss hyperparameters using a validation set, as done in our experiments, it is recommended to leave  $\lambda_{\mathrm{roll}}$  within a small value (we found
Figure A.2: Ablation study for the loss scaling factor  $\gamma$ . Our results show that IntCEM's task and concept accuracies are robust to changes of  $\gamma$  within a reasonable range [0.5, 2].
success with  $\lambda_{\mathrm{roll}} \in [0.1, 5]$ ) while spending more efforts fine-tuning  $\lambda_{\mathrm{concept}}$ , a hyperparameter that unfortunately needs fine tuning for all jointly trained CBM-based architectures.
A.6.2 Task Loss Scaling Factor
In Section 3, we defined  $\gamma \in [1,\infty)$  as a hyperparameter that determines the penalty for mispredicting the task label  $y$  with interventions  $\mu^{(0)}$  versus mispredicting  $y$  with interventions  $\mu^{(T)}$  at the end of the trajectory. In all experiments reported in Section 4, we fixed  $\gamma = 1.1$ . In our ablation study for  $\gamma$  (see Figure A.2), we show that IntCEM's receptiveness to interventions, and it mean concept AUC, is robust and not significantly affected as  $\gamma$  varies within a reasonable range [0.5, 2]. In particular, and as expected, we observe that for a higher value of  $\gamma$ , the model has slightly lower unintervened accuracy while attaining slightly higher intervened accuracy after a few interventions. Nevertheless, these differences are small and suggest that our method is robust to minor variations in this hyperparameter.
A.6.3 Prior Hyperparameters Ablation
IntCEM uses two prior distributions to sample trajectories at train-time: (i)  $p(\mu_i) = \text{Bernoulli}(p_{\text{int}})$  on the initially sampled masks  $\mu^{(0)} \in \{0,1\}^k$  and (ii)  $p(T) = \text{Unif}(1,T_{\text{max}})$  on the horizon, or length, of the intervention trajectory to be sampled. Our ablation studies on both  $T_{\text{max}}$  and  $p_{\text{int}}$  show that IntCEMs outperform CEMs when the number of interventions increases, regardless of the parameters used for  $p_{\text{int}}$  and  $T_{\text{max}}$ . Nevertheless, we observe that when  $T_{\text{max}}$  is small, e.g.  $T_{\text{max}} = 1$ , or  $p_{\text{int}}$  is high, e.g.  $p_{\text{int}} = 0.75$ , IntCEM has a slower increase in performance as it is intervened. We believe this is a consequence of our model not having long-enough trajectories at train-time, therefore failing to condition the model to be receptive to long chains of interventions. This is because as  $p_{\text{int}}$  is larger, the number of concepts initially intervened will be higher on expectation, leading to potentially less impactful concepts to select from at train-time. Similarly, as  $T_{\text{max}}$  is lower, IntCEM's trajectories will be shorter and therefore it will not be able to explore the intervention space enough to learn better long-term trajectories. Therefore, based on these results our recommendation for these values, leveraging performance and train efficiency, is to set  $T_{\text{max}}$  to a small value in [5, 10] (we use  $T_{\text{max}} = 6$  in our experiments) and  $p_{\text{int}}$  to a value within [0, 0.5] (we use  $p_{\text{int}} = 0.25$  in our experiments).
A.7 Additional Results and Details on Intervention Policy Evaluation
In this section, we provide further details on the set-up of our intervention policy experiments and include additional results on the impact of intervening on IntCEMs and our baselines.
(a)  $p_{\mathrm{int}}$  ablation study in CUB.
(b)  $T_{\mathrm{max}}$  ablation study in CUB.
Figure A.3: Ablation studies in CUB for IntCEM's priors' hyperparameters. Notice that, as in previous ablations, although initial unintervened performance may vary across different hyperparameters, regardless of hyperparameter selection IntCEM outperforms traditional CEMs (shown in pink diamonds above) within  $\sim 12$  concept groups being intervened.
A.7.1 Intervention Policy Details
Intervening on Logit CBMs Because Joint CBM-Logits use log-probabilities in their bottleneck rather than sigmoidal zero-one probabilities, intervening on their bottlenecks is slightly different than intervening on Joint CBM-Sigmoidal models. As in [4], we avoid setting intermediate concepts to out-of-distribution values by setting  $\hat{c}_i$  to its empirical 95-th percentile value (computed over the training distribution) when we intervene on concept  $c_i$  and want to indicate that its ground truth value is  $\tilde{c}_i = 1$ . Similarly, when we intervene on concept  $c_i$  and want to indicate that its ground truth value is  $\tilde{c}_i = 0$ , we set it to its empirical 5-th percentile value (computed over its training distribution). This allows us to extend the intervention formulation in Equation (1) to CBMs with logit bottlenecks by letting  $\hat{\mathbf{c}}_i^+$  be the 95-th percentile value of the empirical training distribution of  $\hat{c}_i$  and letting  $\hat{\mathbf{c}}_i^-$  be the 5-th percentile value of the same distribution.
CooP CooP selects the best next concept to intervene on by considering a weighted sum of (i) an uncertainty measurement of each concept's predicted probability (weighted by hyperparameter  $\alpha$ ), (ii) the expected change in the probability of the currently predicted class (weighted by hyperparameter  $\beta$ ), and (iii) the cost of acquiring each concept (weighted by hyperparameter  $\gamma_{\mathrm{coop}}$ ). For simplicity, in this work, we treat all concepts as having the same acquisition cost; i.e., we set  $\gamma_{\mathrm{coop}} = 0$  and only fine-tune  $\alpha$  and  $\beta$ . We leave the exploration of variants of our method which consider different acquisition costs for future work.
Figure A.4: Task accuracy when intervening with different policies (colours) on different methods (columns) and datasets (rows). We observe that, across all tasks and datasets, IntCEMs outperform all other baselines when intervened on. This difference is particularly sharp on the more complicated datasets such as CUB-Incomp and CelebA.
When using CooP for any of our baselines, we select  $\alpha$  and  $\beta$  by performing a grid search over  $\alpha \in \{0.1, 1, 10, 100\}$  and  $\beta \in \{0.1, 1, 10, 100\}$  and selecting the pair of hyperparameters that give us the best area under the validation intervention curve. Given that this process can be costly (in our runs it may take up to  $\sim 45$  minutes to run), we estimate the area under the validation intervention curve by checking CooP performance after intervening with  $1\%$ ,  $5\%$ ,  $25\%$ , and  $50\%$  of the available concept groups for a given task.
Behavioural Cloning Policy To learn a behavioural cloning policy which imitates Skyline through demonstrations of the latter policy, we train a leaky-ReLU MLP with hidden layers  $\{256, 128\}$  that maps a previous bottleneck  $\hat{\mathbf{c}}$  and a mask of previously intervened concepts  $\mu$  to  $k$  outputs indicating the log-probabilities of selecting each concept as the next intervention. This model is trained by generating 5,000 demonstrations  $(\langle \hat{\mathbf{c}}, \mu \rangle, \eta_{\mathrm{sky}})$  of "Skyline".
Each demonstration is formed by (1) selecting a training sample  $\mathbf{x}$  from the task's training set uniformly at random, (2) generating an initial intervention mask  $\mu$  by selecting, uniformly at random,  $l\sim \mathrm{Unif}(0,k)$  concepts to intervene on, (3) producing an initial bottleneck  $\hat{\mathbf{c}} = \tilde{g} (\mathbf{x},\mu ,\mathbf{c})$  using concept encoder  $g$  of the model we will intervene on and  $\mathbf{x}$  's corresponding concept labels  $\mathbf{c}$ , and (4) generating a target concept intervention  $\eta_{\mathrm{sky}}$  by calling the Skyline policy with inputs  $(\hat{\mathbf{c}},\mu)$ . This BC model is then trained to minimise the cross entropy loss between its predicted probability distribution
Figure A.5: Test accuracy for traditional CBMs trained with IntCEM's loss functions (IntCBMs) on the CUB-Incomp dataset. Our results suggest that although there may be merit to incorporating IntCEM's loss into traditional CBMs, their use of scalar concept representation severely limits their stability and unintervened performance.
and  $\eta_{\mathrm{sky}}$  for 100 epochs using stochastic gradient descent with learning rate 0.01, batch size 256, weight decay  $4\times 10^{-5}$ , and momentum 0.9.
We highlight that when selecting concepts from both the BC policy and the learnt policy  $\psi$  at test-time, we deterministically select the concept with the maximum log-probability that has not yet been selected by  $\mu$ .
As discussed in Section 4, in Figure A.4 we observe that IntCEMs achieve significantly higher task accuracies across all tasks under interventions, regardless of the intervention policy used. We notice, however, that the gap between the optimal intervention policy and IntCEM's best-performing policy tends to be larger in the simpler datasets. Further work can investigate the origins of this gap as well as mechanisms to bridge it.
A.8 Extending IntCEM's Loss to Traditional CBMs
Because IntCEM's loss function only assumes that one can intervene on a given model at train-time, this loss function is general enough to be applicable to traditional Joint CBMs. In Figure A.5 we show the results of intervening on joint CBMs whose training objectives have been modified to include IntCEM's  $\mathcal{L}_{\mathrm{roll}}$  and  $\mathcal{L}_{\mathrm{pred}}$  terms (IntCBMs). These results show that IntCBMs, in particular when using a logit bottleneck, can achieve a higher test accuracy than their traditional CBM counterparts when presented with a large number of test-time interventions. Nevertheless, we note the following limitations: (i) we observe that for IntCBMs with sigmoidal bottlenecks, there is no improvement regardless of the use of interventions, and in fact, adding  $\mathcal{L}_{\mathrm{pred}}$  and  $\mathcal{L}_{\mathrm{roll}}$  to traditional CBMs can in fact lead to worse performance; (ii) even in IntCBMs with logit bottlenecks whose intervened performance outperforms their CBM counterparts, we observe a high variance, especially when the number of intervened concepts is small; and (iii) in comparison to IntCEMs, the improvements of our training loss in IntCBMs with logit activations are still significantly underperforming when receiving little-to-none interventions. We believe that these observations arise for two core reasons: first, using concept embeddings allows for a better flow of information between the concept encoder and the label predictor, leading to embedding-based models being more robust to concept incompleteness than their scalar-based counterparts (especially with respect to sigmoidal bottleneck). Second, using embeddings rather than scalar concept representations enable gradients to flow from the label predictor into the concept encoder even after interventions are made, something that is not possible with traditional CBMs as setting a bottleneck activation to a fixed value (e.g., 0 or the 5-th percentile of the activation) is a gradient-blocking operation. This also leads to significantly more stable gradient propagation throughout an intervention trajectory, explaining the high variance in IntCBMs. These results suggest that using high-dimensional embedding representations for concepts may be a crucial
Figure A.6: Task accuracy for IntCEMs and CEMs as we adversarially intervene on an increasingly large number of concept groups (by intentionally selecting the wrong concept label). For IntCEMs we show interventions following its learnt intervention policy  $\psi$  as well as a Random policy. For CEMs, all interventions are done following a Random intervention policy.
component for these models to be receptive to interventions, and further, simply including a higher train-time penalty when mispredicting a label after an intervention trajectory is not enough for a model to become competitive with and without interventions.
A.9 Effect of Adversarial Interventions
In their evaluation, Espinosa Zarlenga et al. [5] observed that CEMs were somewhat more robust to adversarial interventions (i.e., interventions that are intentionally incorrect) compared to existing CBM architectures – where robustness was defined as being able to withstand a certain number of wrong interventions before significantly dropping in performance. This suggests that CEMs may have a form of error correction in their embeddings that allow the label predictor to correct intervention mistakes. In contrast, as posited in our discussion in Section 5, by encouraging IntCEMs to be receptive to test-time interventions, we are also incentivising them to be receptive to all forms of interventions, even if those interventions are adversarial in nature. This is because our model's explicit train-time intervention loss, which encourages IntCEMs to positively respond to test-time interventions, has the underlying assumption that interventions, when provided at train-time or test-time, are correct. Hence, we hypothesise that the same error correction observed in CEMs will not be seen in IntCEMs, leading our models' performance to significantly drop when interventions are wrong. We find evidence supporting this hypothesis when performing adversarial interventions across all our baselines on the CUB, CUB-Incomp, and CelebA tasks. Specifically, as seen in Figure A.6, we observe that an IntCEM's performance drastically drops when receiving only a handful of adversarial interventions, with this result being even more damaging if we follow its learnt intervention policy  $\psi$ . These results suggest that a very interesting and impactful direction for future work can be to explore mechanisms to introduce error correction in IntCEM-like models.
A.10 Leakage in IntCEM's Learnt Representations
Concept leakage [36, 10] is a known issue in CBMs where a CBM's concept encoder learns to encode unnecessary information in its learnt concept representations in order to bypass information from the input to its label predictor. It has been empirically shown that leakage may lead to less interpretable concept representations [36] and to detrimental interventions on CBMs [10]. The latter is a consequence of a CBM's label predictor learning to rely on information that accidentally leaked into the concept representations, information that is necessarily destroyed at intervention-time when an expert overwrites a bottleneck activation with its ground truth value. Nevertheless, the same is not true in embedding-based architectures, such as CEMs and IntCEMs, where interventions simply involve "swapping" an embedding with one that can still carry leaked information into the label predictor (see Equation 1). Hence, in this section we hypothesize that higher leakage may, in fact, be a contributor to IntCEMs' better performance when being intervened on as the model may learn to take better advantage of this leakage to learn to be more receptive to interventions at test-time.
Table A.4: Oracle Impurity Score (OIS) for all jointly-trained baselines across all tasks. Higher OIS values indicate higher leakage in a model's learnt concept representations.
Dataset	IntCEM	CEM	Joint CBM-Sigmoid	Joint CBM-Logit
OIS (%)	MNIST-Add	30.97 ± 0.29	28.25 ± 0.44	13.14 ± 0.27	20.96 ± 0.02
MNIST-Add-Incomp	36.73 ± 0.23	34.09 ± 0.47	13.28 ± 0.27	26.12 ± 0.08
CUB	45.86 ± 0.29	42.54 ± 2.30	21.01 ± 0.58	41.66 ± 0.49
CUB-Incomp	38.79 ± 1.41	42.13 ± 3.48	27.81 ± 2.02	30.32 ± 3.09
CelebA	50.87 ± 4.07	40.63 ± 4.83	29.65 ± 16.51	24.11 ± 09.30