Learning to Receive Help: Intervention-Aware Concept Embedding Models
Authors: Mateo Espinosa Zarlenga
Katherine M. Collins
Krishnamurthy Dj Dvijotham
Adrian Weller
Zohreh Shams
Mateja Jamnik
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Explainable Artificial Intelligence
Concept Bottleneck Models
Concept-based Explainability
Interpretability
XAI
Concept Interventions
TL;DR: We propose a novel training loss that significantly boosts the performance of a Concept Embedding Model when expert concept interventions are available.

Abstract: Concept Bottleneck Models (CBMs) tackle the opacity of neural architectures by constructing and explaining their predictions using a set of high-level concepts. A special property of these models is that they permit concept interventions, wherein users can correct mispredicted concepts and thus improve the model's performance. Recent work, however, has shown that intervention efficacy can be highly dependent on the order in which concepts are intervened on and on the model's architecture and training hyperparameters. We argue that this is rooted in a CBM's lack of train-time incentives for the model to be appropriately receptive to concept interventions. To address this, we propose Intervention-aware Concept Embedding models (IntCEMs), a novel CBM-based architecture and training paradigm that improves a model's receptiveness to test-time interventions. Our model learns a concept intervention policy in an end-to-end fashion from where it can sample meaningful intervention trajectories at train-time. This conditions IntCEMs to effectively select and receive concept interventions when deployed at test-time. Our experiments show that IntCEMs significantly outperform state-of-the-art concept-interpretable models when provided with test-time concept interventions, demonstrating the effectiveness of our approach.

Learning to Receive Help:
Intervention-Aware Concept Embedding Models
Mateo Espinosa Zarlenga University of Cambridge me4660cam. ac.
Katherine M. Collins University of Cambridge kmc61 10am00cam... ac.
Krishnamurthy (Dj) Dvijotham Google DeepMind dvij@google. com
Adrian Weller University of Cambridge Alan Turing Institute aw6650cam.. ac uk
Zohreh Shams University of Cambridge zs3158cam ac uk
Mateja Jamnik University of Cambridge mateja. jamnik@cl. cam ac
Abstract
Concept Bottleneck Models (CBMs) tackle the opacity of neural architectures by constructing and explaining their predictions using set of high-level concepts. special property of these models is that they permit concept interventions, wherein users can correct mispredicted concepts and thus improve the model's performance. Recent work, however, has shown that intervention efficacy can be highly dependent on the order in which concepts are intervened on and on the model's s architecture and training hyperparameters. We argue that this is rooted in CBM's lack of train-time incentives for the model to be appropriately receptive to concept interven- tions. To address this, propose Intervention-awan Concept Embedding models (InCCEMs), a novel CBM-based architecture and training paradigm that improves model's receptiveness to test-time interventions. Our model learns concept intervention policy in an end-to-end fashion from where can sample meaningful intervention trajectories at train-time. This conditions IntCEMs to effectively select and receive concept interventions when deployed at test-time. Our experiments show that IntCEMs significantly outperform state-of-the-an concept- inneppt-iteerpretbble models when provided with test-time concept interventions, demonstrating the effectiveness of our approach.
Introduction
It is important know how to ask for help, but also important to know how to receive help. Knowing how to react to feedback positively allows for bias correction I and efficient learning [2] while being instrumental for mass collaborations (3. Nevertheless, although the uptake of feedback is ubiquitous in real-world Decision-making the same cannot said about modern artificial intelligence (AI) systems, where deployment tends to be in isolation from experts whose feedback could be queried.
Progress in this aspect has recently come from Explainable AI, where interpretable Deep Neural Networks (DNNs) that can benefit from expert feedback at test-time have been proposed 477 In particular, Concept Bottleneck Models (CBMs) 4.. a family of interpretable DNNs, enable expert-model interactions by generating, part of their inference process, explanations for their predictions using high-level "concepts". Such explanations allow human experts to better understand CBM's prediction based on interpretable units of information (e.g., input is "cat" because it has "paws" and "whiskers") rather than low-level features (e.g.. input pixels). This design enables humans to provide feedback the CBM test-time via concept interventions [4] (Figure [ process specific to CBM-like architectures in which an expert analyses a CBM's inferred concepts
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

X
Concept Encoder
Label Predictor
Lage
Hot dog
g(x)
f(()
Ketchup
Not Hot Dog
Sesame Sords
Olherrve Eaplanation
Obbarve Sample 1
Intervene
Legs
Ban
f(c)
Hot dog Not Hot Dog
Ketchup
Human Expert
Sesame Seeds
Figure 1: When intervening on a CBM, a human expert analyses the predicted concepts and corrects mispredicted values (e.g., the mispredicted concept "legs"), allowing the CBM to update its prediction.
and corrects mispredictions. Concept interventions allow CBM update its final prediction after incorporating new expert feedback, and have been shown to lead to significant performance Nevertheless, and in contrast these results, recent works have found that concept interventions can potentially increase CBM's test-time error depending on its choice of architecture [0] and its set of train-time annotations [5]. This discrepancy suggests that CBM's receptiveness to interventions is neither guaranteed nor fully understood.
boosts g
In this paper, posit that the sensitivity of CBM's receptiveness interventions is artefact of its training objective function. Specifically, we argue that CBMs lack an explicit incentive to be receptive interventions they are neither exposed to interventions at train-time optimised to perform well under interventions. To address this, we propose a novel intervention-aware objective function and architecture based on Concept Embedding Models (CEMs) [5]. generalisation of CBMs that represent concepts as high-dimensional embeddings. Our architecture, which we call Intervention-on aware Concept Embedding Model (InCCEM), has two key distinctive features. First, learns an intervention policy end-to-en which imposes a prior over which concepts an expert could be queried intervene on next to maximise intervention receptiveness. Second, its loss function includes an explicit regulariser that penalises InCCEM for mispredicting the task label after series of train-time interventions sampled from its learned policy. We highlight the following contributions:
We introduce IntCEM, the first ionerpt-iteerpretaale neural architecture that learns not only explain its predictions using concepts but also learns an intervention policy dictating which concepts an expert should intervene on next to maximise test performance.
We show that IntCEM significantly outperforms all baselines in the presence of interventions while maintaining competitiveness when deployed without any interventions. We demonstrate that, by rewriting concept interventions as differentiable operation. IntCEM learns an efficient dynamic intervention policy that selects performance-boosinn concepts to intervene on at test-time.
2 Background and Previous Work
Concept Bottleneck Models Methods in Concept-based Explainable AI | advocate for explaining DNNs by constructing explanations for their predictions using high-level concepts captured within their latent space. Within these methods, Concept Bottleneck Models (CBMs) provide a framework for developing cancept-based interpretable DNNs.
Given a training set D := ((x(), c(.) (0))))))))))))) ) where each sample E R (e.g., an image) is annotated with a task label y E {1,.............. -L (e.g., "cat" or "dog") and k binary concepts E {0. 1}a (e.g.. "has whiskers"), a CBM is a pair of functions () whose composition [((()) predicts a probability over output classes given x. The first function g R. - [0, 1] called the concept encoder, learns a mapping between features x and concept activations é = g(x) [0, 1] where ideally è is close to 1 when concept is active in x, and 0 otherwise. The second function I: (0, 1] (0, (0, 1]4, called the label predictor, learns a mapping from the predicted concepts è to a distribution ý == =()) over L task classes. When implemented as DNNs, y's and /'s parameters can be learnt (i) jointly, by minimising a combination of the concept predictive loss Luoncept and the task cross-entropy loss Lusk. (ii) sequentially, by first training minimise Leoncopt and then training

---

of to minimise Lusk from g's outputs, or (iii) independently, where g and f are independently trained to minimise their respective losses. Because at prediction-time f has access only to the "bottleneck" of concepts activations é, the composition (f o g) yields a concept-interpretable model where é; can be interpreted as the probability ias that concept o is active in x.
Concept Embedding Models Because operates only on the concept scores predicted by g. if the set of training concepts is not predictive of the downstream task, then a CBM will be forced to choose between being highly accurate at predicting concepts or at predicting task labels. Concept Embedding Models (CEMs) [S] are a generalisation of CBMs that address this trade-off using nigh-dimensional concept representations in their bottlenecks. This design allows information of concepts not provided during training to flow into the label predictor via two m-dimensional vector representations (i.e., embeddings) for each training concept c;: cf E R=, an embedding representing e; when it is active (i.e., C = 1), and C E I me. an embedding representing G when is inactive (i.e., C = 0).
When processing sample x, for each training concept G; CEM constructs oj and cj by feeding x into two learnable models of" : 07 R R implemented DNNs with shared preprocessing module. These embeddings are then passed to a learnable scoring model $ R 2 - [0. 11, shared across all concepts, that predicts the probability م. = s((č, c of concept c, being active. With these probabilities, one can define a CEM's concept encoder gxx) = ė == [č Cx. E Rim by mixing the positive and negative embeddings of each concept to generate final concept embedding C : , + (1 P,) Finally, a CEM can predict a label y for sample x by passing é (i.e., its "bottleneck") to learnable label predictor f(é) whose output can be explained with the concept probabilities p == [...... 'x1"" This generalisation of CBMs has been empirically shown to outperform CBMs especially when the set of concept annotations is incomplete [5].
Concept Interventions A key property of both CBMs and CEMs is that they allow concept interventions, whereby experts can correct mispredicted concept probabilities P at test-time. By updating the predicted probability of concept (i.e., pt) so that it matches the ground truth concept label (i.e., setting P. == ct), these models can update their predicted bottleneck è and propagate that change into their label predictor (()), leading to potential update in the model's output prediction.
Let E 13 be a mask with μı; = if we will intervene on concept C. and μı; = 0 otherwise. Here, assume that we given the ground truth values è of all the concepts we will intervene on. For notational simplicity, and without loss of generality, we can extend è to be in [0. 1]at by setting & = C; μ4, 1, and à = 0.5 otherwise (where 0.5 expresses full uncertainty but, as it will become clear next, the specific value is of no importance). Thus, this vector contains the expert-provided ground truth concept labels for concepts we are intervening on, while it assigns an arbitrary value (e.g., 0.5) to all other concepts. We define an intervention as a process where, for all concepts c,, the activation(s) é; corresponding to concept o in a bottleneck are updated as follows:
é = (u,ā + (1 (. + (1 - (mi + (1 ee
(1)
where éit and é, are 's positive and negative concept embeddings (for CBMs èt [1] and e, [0]). This update forces the mixing coefficient between the positive and negative concept embeddings to be the ground truth concept value e; when we intervene on concept e; (i.e., (4; = 1), while maintain- ing it as f. if we are not intervening on concept c;. The latter fact holds as (J40 + (1 p1))),) ) P; when /4; 0, regardless of é. Because the predicted positive and negative concept embeddings, as well as the predicted concept probabilities, are all functions of x, for simplicity we use ğ(x. μ. é) to represent the updated bottleneck of CBM (g. f) when intervening with mask μ and concept values č.
We note that although we call the operation defined in Equation a "concept intervention" to follow the term used in the concept learning literature, it should be distinguished from interventions in causal models [18 20). Instead, concept interventions in this context are connected, yet not identical, to previous work in active feature acquisition in which one has access to expert at test-time to request the ground truth value of a small number of input features 21
In practice, may not be sensible to ask a human to intervene on every concept (24]: instead, we may want preferential ordering of which concepts to query first (e.g., some concepts may be highly informative of the output task, yet hard for the model to predict). "Intervention policies" have been developed to produce sensible ordering for concept interventions [8] 25 9.
Intervention Policies
3

---

These orderings can be produced independently of a particular instance (i.e., static policy), or selected on a per-instance basis dynamic policy). The latter takes as an input mask of previously intervened concepts μ and a set of predicted concept probabilities p. and determines which concept should be requested next from an expert, where the retrieved concept label is assumed to be "ground truth". Two such policies are Cooperative Prediction (CooP) [9] and Expected Change in Targer Prediction (ECTP) [8) which prioritise concept c; if intervening on G, next leads to the largest change in the probability of the currently predicted class in expectation (with the expectation taken over the distribution given by p(c; = 1) = p.). These policies, however, may become costly as the number of concepts increases and share the limitation that, by maximising the expected change in probability in the predicted class, they do not guarantee that the probability mass shifts towards the correct class.
3 Intervention-Aware Concept Embedding Models
Although there has been significant work on understanding how CBMs react to interventions [ 9 (,225, the fact that these models are trained without any expert intervention - yet are expected to be highly receptive to expert feedback at test-time- is often overlooked. To the best of our knowledge, the only exception comes from Espinosa Zarlenga et al. [5], wherein the authors introduced RandInt, a procedure that randomly intervenes on the model during train-time. However, RandInt assumes that model should be equally penalised for mispredicting y regardless of how many interventions have been performed. This leads to CEMs lacking the incentive to perform better when given more feedback. Moreover, RandInt assumes that all concepts are equally likely to be intervened on. This inadequately predisposes model to assume that concepts will be randomly intervened on at test-time.
To address these limitations, we propose Intervention-Aware Concept Embedding Models (InCCEMs), a new CEM- basd- architecture and training framework designed for inducing receptiveness to interventions. IntCEMs are composed of two core components: 1) end-to-end learnable concept intervention policy (ê, 1i), and 2) a novel objective function that penalises InCCEM it mispredicts its output label after following an intervention trajectory sampled from & (with heavier penalty the trajectory is longer). By learning the intervention policy v/ in conjunction with the concept encoder and label predictor, IntCEM can simulate dynamic intervention steps at train-time while it learns a policy which enables it to "ask" for help about specific concepts at test-time.
3.1 Architecture Description and Inference Procedure
An IntCEM is tuple of parametric functions (g, () where: (i) g is concept encoder, mapping features x to a bottleneck è, (ii) f is a label predictor mapping è to a distribution over L classes, and (iii) v/= : Rkm x 10, + [0, 1] is a concept intervention policy mapping è and a mask of previously intervened concepts μ to probability distribution over which concepts to intervene on next. IntCEM's concept encoder g works by (1) passing an input x to a learnable "backbone" {(x) == (((e)" é, ) D P). which predicts pair of embeddings (e," e, ) and a concept probability P; for all concepts, and (2) constructs bottleneck è by concatenating all the mixed vectors (prct possible for Ç. in this
(1 - i1 .......... c
+ (1 px,C }. While multiple instantiations work we parameterise it using the same backbone as in a CEM. This implies that Ç is formed by concept embedding generating models {(ot o, } and scoring model This yields model which, at inference, is almost identical to a CEM, except that IntEEM also outputs a probability distribution (ú 14) placing high density on concepts that may yield significant intervention boosts. At train-time, these models differ significantly, as we discuss next.
3.2 IntCEM's Training Procedure
The crux of IntCEM's training procedure, shown in Figure 2 lies in inducing receptiveness to test-time interventions by exposing our model to dynamic train-time interventions. Specifically, at the beginning of each training step, we sample an initial mask of intervened concepts (1))) from prior distribution p(μ) as well as the number T E {1..... .k) of interventions perform on top of ()) from prior T (TT). Next, we generate a trajectory of T new interventions (n()) E {0, p(n; ; (0)), by sampling the tth intervention nf(*) p(n: ; w()) (represented as not encoding) from categorical distribution with parameters w(t) A key component of this
work is that parameters (9) are predicted using IntCEM's policy &h (e((t1)) (((t=1),((t-l))) (((-))) evaluated on the

---

Input
CRM Backbone
Intervention Trajectory with T ((T) and a p(a)
Legend
Trainable Model
C
p
2
[(B0-))
/
Loss Computation
Intervention
CB(c (xc
w))
Fill
Sumpling Operation
m
1000
7Bly,)1(11+))
6
becameys
Figure 2: Given concept embeddings and probabilities ((x) = (((C) c, ) (=))) P). an IntCEM training step (1) samples an intervention mask y,()) ~ p(p) and horizon T ~ p(T), (2) generates an intervention trajectory ((((t11, ( ))(= from the learnable intervention policy 8/, (3) and predicts the task label at the start and end of the trajectory. Our loss incentivises (i) good initial concept intervention trajectories (Crat)), (iii) and low task loss before and after interventions (Cprea), with a heavier penalty 7 > 1 for mispredicting at the end of the trajectory. In this figure, dashed orange arrows indicate recursive steps in our sampling process.
predictions (Cconcept) (ii) performance-boostinn
previous bottleneck c(t- := ((x,xx(t f((-1),, c) and intervention mask (,(=1),, where we recursively define ( as ( := (1(t-1)+++ (0) for t > O. This allows InCCEM to calibrate its train-time trajectory of interventions reinforce supervision on the embeddings corresponding currently mispredicted concepts, leading to IntCEM learning more robust representations inherently harder concepts.
We design our objective function such that it incentivises learning performance-boose intervention trajectories and achieving high task performance as more interventions are provided:
L(x, yy T) = Arnn Xx, c. y, 7) Freat (x, c. y, (0) (x( >econcep (c. p)
where T = ((((t11), ) is the intervention trajectory while Aroil and Xconeope are user-defined hyperparameters. Aro11 encodes user's tradeoff for number of interventions versus task performance gains (e.g.. high Aro11 encourages rapid task performance gains from few interventions, whereas low Aroll is more suitable in settings where many expert queries can be made). Aconcopt expresses user's preference between accurate explanations and accurate task predictions before any interventions.
Rollout Loss (Cron) The purpose of Lioll is to learn an intervention policy that prioritises the selection of concepts for which the associated concept embeddings fails to represent the ground truth concept labels, i.e., select concepts which benefit from human intervention. We achieve this through a train-time "behavioural cloning" 22 27 approach where is trained to learn to predict the action taken by an optimal greedy policy at each step. For this, we take advantage of the presence of ground truth concepts and task labels during training and incentivise u to mimic "Skyline", an oracle optimal policy proposed by Chauhan et al. (9. Given bottleneck è and ground-truth labels (y. c).
the Skyline policy selects (x, μt C, y) == arg magi ff [ğ(x, V 1,, c)), y as its next concept, where [(), is the probability of class y predicted by I and µ V 11 represents the action of adding an intervention on concept i to p. Intuitively, e, corresponds to the concept whose ground-truth label and intervention would yield the highest probability over the ground-truth class y. We use such a demonstration to provide feedback to & through the following loss:
T
(x c, y, T)
CE(c, ( 1) ,,, v), (0(x,, (-11 (,, cc)
T
1=1
where CE(p, p) is the cross-entropy loss between ground truth distribution p and predicted distribu- tion P. penalising &b for not selecting the same concept as Skyline throughout its trajectory.
Task Prediction Loss (Cpred) We penalise IntCEM for mispredicting y. both before and after intervention trajectory, by imposing a higher penalty when it mispredicts at the end of the trajectory:
(x, y, H ( (T)) = CE(y. fğ(x, ()) c))) + + TCE(E (y. f0((x, ((T),) (c)))
5

---

Here E [1, oc) is scaling factor that penalises InCCEM more heavily for mispredicting at the end of the trajectory, by factor of y > than for mispredicting y the start of the trajectory. In practice, we observe that value in E [1.1, 1.5) works well; we use unless specified otherwise. For an ablation study showing that InCCEM's s intervention performance surpasses that of existing methods for wide array of scaling factors 7, see Appendix AA.6]
1.1 in our experiments
key realisation is that expressing interventions as in Equation 1 yields an operator which is differentiable with respect to the concept intervention mask f.. This trick allows us to backpropagate
gradients from peed into sb when the sampling operation r7! ~ p((] ( (a0t-")),______ (((=)))) is differen-
tiable with respect its parameters. In this work, this is achieved by relaxing our trajectory sampling using a differentiable Gufbel-Softmax
(8 categorical sampler (see Appendix A.2 for details).
Concept Loss (Cconcept) The last term in our loss incentivises accurate concept explanations for IntCEM's predictions using the cross-entrope loss averaged across all training concepts:
A: (- log ps -1 (1-c;)
1 Σ CE(c, (.) k
Leoncepi (c.
c2) log (1 -p:))
Putting everything together (L)
We learn InICEM's parameters 0 by minimising the following:
arg min E (x.c.w)D [F (ppp(l)) T~p(T) [A(x, c, TDI]
0- 8
(2)
The outer expectation be optimised via stochastic gradient descent, while the inner expectation can be estimated using Monte Carlo samples from the user-selected priors (p(n). For the sake of simplicity, we estimate the inner expectation using single Monte Carlo sample per training opt to use Bernoulli prior for p(u), where each concept is selected with probability Piot = 0.25, and a discrete uniform prior Unif({1,- ,
step for both the initial mask (0)) and the trajectory horizon T. In our experiments,
mmax)) }) for p(Z"), where Tmax is annealed within [2, 6j. Although
domain-specific knowledge can be incorporated into these priors, we leave this for future work and show in Appendix (A.633 (....
that InCCMMs are receptive interventions
vary Pint and T Tmax-
4 Experiments
In this section, we evaluate IntCEMs by exploring the following research questions:
Unintervened Performance (Q1): In the absence of interventions, how does InICEM's task and concept predictive performance compare to CEMs's ' and CBM's? Does InCCEM's updated loss function have detrimental effects on its uninterverned performance? Intervention Performance (Q2A): Are IntCEMs more receptive to test-time interventions than state-of-the-as CBM variants?
Effects of Policies during Deployment (Q2B): What is the impact on IntCEM's perfor-
mance when employing different intervention policies at test-time?
Benefits of End-to-Ema Learning of 1b (Q3): Is it beneficial to learn our intervention policy sb at train-time?
Datasets and Tasks We consider five vision tasks: (1) MNIST-Add, task inspired by the UMNIST dataset [29] where one is provided with 12 MNIST 30 images containing digits in ... 9), as well their digit labels concept annotations, and needs to predict if their sum is at least half of the maximum attainable, (2) MNIST-Ades Add-Incomp. a concept-incomppete [17] extension of the MNIST-Add task where only 8/12 operands are provided concept annotations, (3) CUB (31, bird classification task whose set of concepts cover 112 bird features, e.g., "wing shape", selected by Koh al. 14. (4) Incomp, a concept-incomplees extension of CUB where we provide only 25% of all concept groups, and (5) CelebA, task on the Celebrity Attributes Dataset [32] whose task and
concept labels follow those selected in [5] For more details refer Appendix A.3
We evaluate IntCEMs against CEMs with the exact same architecture, averaging all metrics of interest over five different random initialisations. We also include CBMs that are trained jointly (Joint CBM), sequentially (Seq. CBM) and independently (Ind. CBM) part of our evaluation to
Models

---

Table Task predictive performance (aceuracy or AUC in %) and mean concept AUC (%) across all tasks and baselines. Values within one standard deviation from the highest baseline are emphasised. CelebA's task accuracies are relatively low across all baselines given its inherent difficulty (256 highly
imbalanced classes). Nevertheless, we highlight these results match previous observations S
Datsser
InCEMM Aur - 5)
InCCMM) 1)
InKEM (1.1)
CEM
Joar CBM Signoid
Join CIM Lops
Ind. CBM
Seq. CBM
4dd
30.76:166 1,66
92.07
12.05:0.25
90.13 0.44
17.66
=
Thok
HMTST HMIST-Astt-Te Aen--r
99.41 IL16
39.43 00.39
RIA3 6,13
66.11 0.7.5
1661 1:15
B4.0% 0.87
56.40 4647
76:20 (.95
78.03 0..5
17.59 0
79,00 0.61
14.95 0.56
78,16 8 0.24
6025 3:30
5341
CUB Incomp
70:39 = 1:11
7466 9,50
19.73 4,00
68.42 5: 7.91
4:332
Ckiiba
38,09 : 0.26
32,89 0.886
30,66 1.0.1
$1.04200.00
24.15 1.443
23.57 8: 1.25
26.25
34.33
Colcept
MATBT Encomp
HS Lsd
00.81 x
N7.07 1:04
$5.25 0.:22
1.58
65.03.20.27 0.227
K7:36 + 1.44
K3.31 3.3.5
M--8 --0.1
59.73 0.15
v1.15 + 9.45
89.21 8.71
WL.8 1.28
12.2910.66
57.58.00.32
20.59:
67.38.00.37 +-9.7
59.63 0.43
0.008
0.222
94,66 0.04
93.76.20.16
0.77
Inccup
02.10
94.43
9
9465 0.10
93.58
82.57
93.65
92.65 +.2.5
be able to compare InICEM's policy against state-of-art intervention policies developed for CBMs. Since previous work shows that interventions in joint CBMs are sensitive to its bottleneck activation function [0], we include joint CBMs with a sigmoidal (Joint CBM- (BM-Sigmoid) and logit-based (Joins CBM-Logit) bottleneck. We note that we do not include other noteworthy concept- based models such as leakage-free CBMs 6 Nets 33 Self-explaining Neural Networks (SENNs) (34], Concept
Whitening (CW) R5 Concept Model Extraction (CME) [6] as these either lack well-defined mechanism to be intervened on (e.g., as in SENNS and CW) or represent more constrained versions of other baselines such as CEMs (c.g.. leakage-free CBMs, GlanceNets, and CME).
Throughout our experiments, we strive for a fair comparison by using the same architectures for the concept encoder and label predictor f of all baselines. Unless specified otherwise, for each task, we tune IntCEM's Aroli byperparameter by varying it over {5. 1, 0.1} and selecting the model with the highest area under the curve defined by plotting the validation task accuracy vs number of interventions. For the CUB- based tasks and CelebA, we use the same values of Aconcept selected in (5). Otherwise, we select this value from (10,5, }} using CEM's validation error. Following (5), we use embeddings with m = 16 activations and a value of Pim = 0.25 for CEM's RandInt. For Appendix A.4 A.4
further details on model and training hyperparameters,
4.1 Task and Concept Performance in the Absence of Interventions (Q1)
First, we examine how InCEM'ss loss function affects its unintervened test performance compared to existing baselines as we vary Acel-- We summarise our results in Table where, for all tasks and baselines, we show the downstream task predictive test accuracy, or the AUC when the task is binary (i.e.. WNMST--baaee tasks), and the concept performance. We measure the latter via the mean AUC of predicting ground-truth test concept e, from its corresponding predicted concept probability pa-
In the absence of concept interventions, IntCEM is accurate as state-of-the-aat baselines. Evaluating test task accuracy reveals that InCEMs are more accurate (e.g., by ~7% in CelebA) or just as accurate other baselines across most tasks. Only in CUB do we observe small drop in performance (less than ~1%) for the hest-performing IntCEM variant. Notice that three of the five tasks use, namely MNIST- -ee---Add-nneee Incomp, CUB-Income and CelebA, lack complete set of training concept annotations. Therefore, our results suggest that IntCEM maintains a high task performance even with concept incompleteness Finally, the performance improvements of IntCEMs over CEMs in MNIST, MNIST- Incomp, and CelebA suggest that our loss function may lead more informative concept representations that enable more accurate label predictors to be trained.
highly desired property for real-world deployment.
IntCEM's concept performance is competitive with respect to traditional CEMs. The bottom half of Table [ shows that IntCEM's s mean concept AUC, particularly for low values of Aroll is competitive against that of CEMs. The only exception in the MNIST-based task, where Joint CBMs outperform CEMs and InCCEMs. Nevertheless, this drop is overshadowed by the poor task accuracy obtained by joint CBMs in these two tasks, highlighting that although CBMs may have better concept predictive accuracy than CEMs, their task performance is generally bounded by that of a CEM. These results demonstrate that IntCEM's explanations are as accurate as those in CEMs without sacrificing task accuracy, suggesting our proposed objective loss does not significantly affect IntCEM's performance in the absence of interventions. Finally, we observe a trade-off between incentivising the InCEM to be highly accurate after small number of interventions (i.c., high A
7

---

Task Accuracy (86
MNIST-Add
MNIST-Add Incoom
CUB
CUB-I Incomp
CelebA
100
100
90
80
60
80
80
40
20
12
6 Groups Intervened
2
0
12 16 20 24 28
2
4 Groups Intervened
2
Groups Intervened
Groups Intervened
Groups Intervened
IMCEEM
CEM
Joint CBM-Logit
X
Joint CBM-Sigmood
Sequential CBM
Independent CBM
Figure 3: Task accuracy of all baseline models after receiving a varying number of randomly selected interventions. For our binary MNIST-based tasks, we show task AUC rather than accuracy. Here and elsewhere, we show the means and standard deviations (can be insignificant) over five random seeds.
and incentivising the IntCEM to generate more accurate concept explanations (i.e., low Awst). which can be calibrated by setting Avol after considering how the IntCEM will be deployed.
4.2 Intervention Performance (Q2)
Given that core appeal of CBM-like models is their receptiveness to interventions, we explore how test-time interventions affect IntCEM's task performance compared to other models. We first study whether IntCEM's training procedure preconditions it be more receptive test-time inter- ventions by evaluating the task accuracy of all baselines after receiving random concept interventions. Then, we investigate whether InICEM's competitive advantage holds when exposing our baselines to state-of-the intervention policies. Below, we report our findings by showing each model's ' task accuracy as intervene, following 44) on increasing number of groups of mutually exclusive concepts (e.g.. "white wing" and "black wing").
IntCEM significantly more receptive to test-time interventions. Figure shows that InCCEM's task accuracy after receiving concept interventions is significantly better than that of competing methods across all tasks, illuminating the benefits of IntCEM preconditioning for receptiveness. We observe this for both concept-complete tasks (i.e., CUB and (rIST-AAdd) well as concept- incomplete tasks (i.e., rest). In particular, we see significant gains in CUB and CelebA; InCCEMs attain large performance improvements with only a handful of random interventions (~10% with 255 of concepts intervened) and surpass all baselines.
IntCEM's performance when intervened on with randomly selected set of concepts can be better than the theoretically-prover optimal intervention performance achievable by CEMs alone. We explore InCCEM's receptiveness to different test-time intervention policies by computing its test performance while intervening following: (I) a Random intervention policy, where the next concept selected, uniformly at random, from the of unknown concepts, (2) the Uncertainty of Concept Prediction (UCP) [8] policy, where the next concept is selected by choosing the concept e, whose predicted probability P. has the highest uncertainty (measured by 1/l0; 0.5)), (3) the Cooperative Policy (CooP) [9] where we select the concept that maximises a linear combination of its predicted uncertainty (akin to UCP) and the expected change in the predicted label's probability when intervening on that concept, (4) Concept Validation Accurucy (CVA) static policy, where concepts are selected using a fixed order starting with those whose validation errors are the highest as done by Koh et al. [4, (5) the Concept Validation Improvement (CVT) (9] static policy, where concepts are selected using a fixed order starting with those concepts that, on average, lead to the biggest improvements in validation accuracy when intervened on, and finally (6) oracle Skyline policy, which sclects c, on every step, indicating an upper bound for all greedy policies. For details on each policy bascline, including how their hyperparameters are selected, see Appendix A.7
We demonstrate in Figure IntCEM consistently outperforms CEMs regardless of the test-time intervention policy used, with CooP and UCP consistently yielding superior results. Further, we uncover that intervening on InCCEM with Random policy at test-time outperforms the theoretical best performance aliuinable by CEM (Skyline) under many interventions, while its best-performing policy outperforms CEM's Skyline after very few interventions. These results suggest that IntCEM's Main-time conditioning not only results in InCCMMs being more receptive to test-time interventions
8

---

CUB
CelebA
IntCEM
CEM
IntCEM
CEM
Task Accuracy (%)
69
69
94
59
59
89
89
49
49
84
79
39
39
74
12 16 24 26
4 16 28
29
0
29
2
4
Groups Intervened
Groups Intervened
Groups Intervened
Groups Intervened
Skyline (Oracle)
Random
UCP
CooP
CVA
Figure 4: Task accuracy of IntCEMs and CEMs on CUB and CelebA when intervening with different test-time policies. We show similar improvements of IntCEMs over CBMs in Appendix A.7
Task Accuraty (%)
MNIST-Add
MNIST-Add-Incoms
CUB
CUB-Incomp
CelebA
100
97.5
95.0
80
80
12
0
2
4
8 12 16202428
2
4
6
0
4
Groups Intervened
Groups Intervened
Groups Intervened
Groups Intervened
Groups Intervened
I
IntCEM (Random Policy)
InCCEM (BC-Skyline Policy)
+ IntCEM no y (Random Policy)
InICEM (CooP Palicy)
InCEM (Learnt Policy 9)
Figure 5: Task performance when intervening on InCCESs following test-time policies vb, CooP, Ran- dom, and BC-Skyline. Our baseline "IntCEM no & is an IntCEM whose test and train interventions sampled from Random policy rather than from y (i.e., policy is not learnt in this baseline).
but may lead significant changes in an IntCEM's theoreticalooptimal intervention performance. We include analyses on our remaining datasets, revealing similar trends, in Appendix A.7
4.3 Studying IntCEM's Intervention Policy (Q3)
Finally, we explore IntCEM's 's intervention policy by evaluating test-time interventions predicted by vb. To understand whether there is a benefit of learning & end-to-end (i.e., in conjunction with the rest of model training), rather than learning it post-hoc after training IntCEM, compare it against Behavioural Cloning (BC) 26 policy "BC-Skyline" trained on demonstrations Skyline applied to a trained IntCEM. Furthermore, explore how v trajectories train-time help improve IntCEM's re- ceptiveness to test-time interventions, we study an IntCEM trained by sampling trajectories uniformly random ("InCEEM no "²" Our results in Figure 5 suggest that: (1) IntCEM's learnt policy i leads to test-time interventions that are as good or better than CooP's, yet avoid CooP's computational cost (up ~xx faster seen in Appendix (A.5). (2) learning vo end-to-end during training yields a better policy than one learnt through BC after the IntCEM is trained, as seen in the BC policy's 's lower performance; and (3) using a learnable intervention policy at train-time results in significant boost of test time performance. This suggests that part of IntCEM's receptiveness to interventions, even when using a random test-time policy, can be partially attributed to learning and using á during training. We show some qualitative examples of rollouts of zb in the CUB dataset in Figure 6
Discussion and Conclusion
Concept Leakage Within IntCEMs Previous work (36] 6 10 has shown that CBMs are prone to encoding unnecessary information in their learnt concept representations. This phenomenon, called concept leakage, may lead to less interpretable concept representations (36) and detrimental concept interventions in CBMs I0) Given that interventions in IntCEMs involve swapping concept's predicted embeddings rather than overwriting an activation the bottleneck, such interventions are distinct from those in CBMs they enable leaked information to be exploited after intervention

---

Ale Lnvrnetia 41
Abte kgodee
#3 Ahe vasulapo
Aby anecynatio BSl leeded cake
Boker lomee
"pperaats puke
Sakel he the Trekited
Scier lamaieer
dape
Ale luaaaee
tead
Altr Stcretii th
the poeee poi you Sogeit
Stew chaisse mi Aoooe the Mereegeeee
Jonr doniie thie in aogee
on the Co
Figure 6: Examples of interventions on an InCCEM following its policy eb in CUB. We show the task label distribution p(glé) for the most likely classes after each intervention. We highlight the correct label's probability using orange bars and show the selected concept by vp above each panel.
performed. This means our loss function may incentivise IntCEM exploit this mechanism to improve model's receptiveness to interventions. In Appendix A010 we explore this hypothesis and find that we can detect more leakage in InCCEM's concept representations than in those learnt by CBMs and CEMs. This suggests that, contrary to common assumptions, leakage may be healthy byproduct of more expressive concept representations in models that accommodate such expressivity. Nevertheless, we believe further work is needed to understand the consequences of this leakage.
Limitations and Future Work To the best of our knowledge, we are the first to frame learning to receive concept interventions as a joint optimisation problem where we simultaneously learn concepts, downstream labels, and policies. IntCEMs offer a principled way to prepare concept-based models for interventions at test-time, without sacrificing performance in the absence of interventions. Here, we focused on CEMs as our base model. However, our method can be extended to traditional CBMs (see Appendix A.8 and future work may consider extending it to more recent concept-based models such as post-hoc CBMs (37), label-free CBMs (38), and probabilistic CBMs [3).
Furthermore, we note some limitations. First, IntCEM requires hyperparameter tuning for A and Asoncepe While our ablations in Appendix suggest that InCCEM's s performance gains extend across multiple values of Anoll, users still need to tune such parameters for maximising IntCEM's utility. These challenges are exacerbated by the computational costs of training IntCEMs due to their train-tine time trajectory sampling (see Appendix A.5). However, such overhead gets amortised over time given v efficiency over competing policies. Further, we recognise our training procedure renders IntCEMs more receptive to any form of intervention, including adversarial interactions (see Appendix A.9p. This raises potential societal concern when deciding how our proposed model is deployed. Future work may explore such issues by incorporating error-correcting mechanisms or by considering intervention-time human uncertainty [29].
Finally, our evaluation was limited to two real-world datasets and one synthetic dataset, all of which have medium cediium-to-small to-small training set sizes. Therefore, future work may explore how to apply our proposed architecture to larger real-world datasets and may explore how to best deploy IniCEM's policy in practice via large user studies.
Conclusion A core feature of concept-basee models is that they permit experts to interpret predic- tions in terms of high-level concepts and intervene on mispredicted concepts hoping to improve task performance. Counter-intuitive work, we introduce a novel pmncept-interpreaaae Aware Concept Embedding Models (InCCEMs) designed explicitly for intervention receptiveness. IntCEMs simulate interventions during train-time, continually preparing the model for interventions they may receive when deployed, while learning an efficient intervention policy. Given the cost of querying experts for help, our work addresses the lack of a mechanism leverage help when received, and demonstrates the value of studying models which, by design, know how to utilise help.
such models are rarely trained with interventions in mind. In this
architecture and training paradigm Intervention-
10

---

Acknowledgments and Disclosure of Funding
The authors would like to thank Naveen Raman, Andrei Margeloiu, and the NeurIPS 2023 reviewers for their insightful and thorough comments earlier versions of this manuscript. MEZ acknowledges support from the Gates Cambridge Trust via Gates Cambridge Scholarship. KMC acknowledges support from the Marshall Commission and the Cambridge Trust. AW acknowledges support from Turing AI Fellowship under grant EP/VO25279/1, The Alan Turing Institute, and the Leverhulme Trust via CFL. GM is funded by the Research Foundation Flanders 1239422N). MJ supported by the EPSRC grant EP/T019603/1.
GA No
References
[1] Luke Gusukuma, Austin Cory Bart, Dennis Kafura, and Jeremy Ernst. Misconceetiion-drienn feedback: Results from an experimental study. In Proceedings of the 2018 ACM Conference on International Computing Education Research, pages 160-168, 2018.
[2] Noreen M Webb. Group collaboration in assessment: Multiple objectives, processes, and outcomes, Educational Evaluation and Policy Analysis, 17(2):239-261, 1995.
[3] AnHai Doan, Raghu Ramakrishnan, and Alon Y Halevy. Mass collaboration systems on the world-wide web. Communications of the ACM, 54(4):86-96, 2010.
[4] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338 5348. PMLR, 2020.
[5] Mateo Espinosa Zarlenga, Barbiero Pietro, Ciravegna Gabriele, Marra Giuseppe, Francesco Giannini, Michelangelo Diligenti, Shams Zohreh, Precioso Frederic, Stefano Melacci, Weller Adrian, et al. Concept embedding models: Beyond the accuracy -currac-explainaabiiyy trade-off. In Advances Neural Information Processing Systems, volume 35, pages 11400-21413 Curran Associates, Inc., 2022.
[6] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems, 2022.
[7] Lucie Charlotte Magister, Pietro Barbiero, Dmitry Kazhdan, Federico Siciliano, Gabriele Ciravegna, Fabrizio Silvestri, Mateja Jamnik, and Pietro Lio. Encoding concepts in graph neural networks. arXiv preprint arXiv:2207. 13586, 2022.
[8] Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXhv:2302.142660 2023.
[9] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.074400, 2022.
[10] Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv e-prints, pages arXiv-2301, 2023.
[11] David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissec- tion: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[12] Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8738,8738, 2018.
[13] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability Beyond Feature Auribution: Quantitative Testing With Concept Activation Vectors (TCAV). International conference on machine learning, pages 2668-200 2677. PMLR, 2018.
11

---

[14] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (CaCE), arXiv preprint arXiv:1907.07165. 2019.
[15] Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint uuXiv:/902.03/22) 2019.
[16] Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Lio, and Adrian Weller. Now you see me (cme): concept-based model extraction. arXiv preprint arXiv:2010.1322 2020.
[17] Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554-2008 20565, 2020.
[18] Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: Cambridgeenivveriiyyyresss 19(2):3, 2000.
[19] Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv: 066,, 2019.
[20] Matthew O' S'Shaugneessy. Gregory Canal, Marissa Connor, Christopher Rozell, and Mark Davenport. Generative causal explanations of black-box classifiers. Advances in neural information processing systems, 33:5453-5467, 2020.
[21] Hajin Shim, Sung Ju Hwang, and Eunho Yang. Joint active feature acquisition and classification with variable-size set encoding. Advances neural information processing systems, 31, 2018.
[22] Yang Li and Junier Oliva. Active feature acquisition with generative surrogate models. In International Conference on Machine Learning, pages 6450-6459, PMLR, 2021.
[23] Ryan Strauss and Junier B Oliva. Posterior matching for arbitrary conditioning. Advances in Neural Information Processing Systems, 35:18088-1800 18099, 2022.
[24] Matthew Barker, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, and Umang Bhatt. Selective concept models: Permitting stakeholder customisation at test-time. AAAI HCOMP, 2023.
[25] Ivaxi Sheth, Aamer Abdul Rahman, Laya Rafiee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou. Learning from uncertain concepts via test time interventions. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurlPS 2022, 2022.
[26] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelli- gence 15, pages 103-129, 1995.
[27] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[28] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 1.01 1.01 2016.
[29] Katherine M. Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. AIES, 2023.
[30] Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 296)):141-14,, 2012.
[31] C. Wah, S. Branson, P. Welinder, Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report NNNS-TR-2011-00 California Institute of Technology, 2011.
[32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
12

---

[33] Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models, Advances in Neural Information Processing Systems, 35:21212-21227, 2022.
[34] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. arXiv preprint arXiv:1806.07338, 2018.
[35] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
[36] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint htti:2106.133114 2021.
[37] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIR2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2023.
[38] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui- Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations. 2023.
[39] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprini arXiv:2306.01 01574, 2023.
[40] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, -ereforeffomeece deep learning library. arXiv preprint arXiv:1912.01703. 2019.
[41] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In Osdi, volume 16, pages 265-283. Savannah, GA, USA, 2016.
[42] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.66 14, 2013.
[43] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference machine learning, pages 448-456. pmlr, 2015.
[44] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Atlanta, Georgia, USA, 2013.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[46] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/12/ 2(417):1, 2012.
13

---

A Appendix
A.1 Software and Hardware Used
Software For this work, we evaluate all baselines using the MIT-licensed implementations of both CBMs and CEMS made public in [5]. Our implementation of InCCEM is built on top of that repository using PyTorch 1.12 [40, an open-source deep learning library with a BSD license. We incorporated CooP into this library by basing ourselves on the code released under an MIT license by Chauhan et al. 9 as part of their original publication; their code was written in TensorFlow, and as such, we converted it to PyTorch. All numerical plots and graphics have been generated using Matplotlib 3.5, a Python-based plotting library with BSD license, while all conceptual figures were generated using draw.io a free drawing software distributed under an Apache 2.0 license. Furthermore, when building our experiment tasks, we use TensorFlow' 's distribution [4] of MNIST [30 and CelebA [2) and use the processing scripts made public by Koh et al. 4 to generate our concept annotated CUB-tasks. All of our code, including configs and scripts to recreate results shown in this paper, has been released as part of CEM's official public repository found at httpps//gitthu..co. http://hhhhu.cgoommmmsppsppsssss hatp////aatess////ccoo
Resources All our non-ablation experiments were run in shared GPU cluster with four Nvidia Titan Xp GPUs and 40 Intel(R) Xeon(R) E5-2630 v4 CPUs (at 2.20GHz) with 125GB of RAM. In contrast, all of our ablation experiments were run separate GPU cluster with 4x Nvidia A100-SXM-80GE GPUs, where each GPU is allocated 32 CPUs. We estimate that to complete all of our experiments, including prototyping and initial explorations, we required approximately 350 to 370 GPU hours.
A.2 Additional Details on the Gumbel-Softman Sampler
As introduced in Section 3 expressing interventions using Equation T allows us to differentiate the output embedding for concept e, with respect to the intervention mask μ after an intervention has been performed. Therefore, if the output of the sampler p(7) ((é(t-1), 1(t-1)) used to generate intervention 1) is differentiable with respect to its parameters (e(t-1)) (()-))), then we can use the gradients from VCE f (ā pu(T), e))). one of the terms in Spred. to update the parameters of th through standard back- propaaropagttion This is because the intervention mask used at the end of the training trajectory is given by y²²²) = ( + ΣT= 1)"), allowing the gradient of ğ(x, (T) c) to be back-propagated into the generating functions of each of the interventions (1)()){= Therefore, when parameterising our learnable policy &b as a DNN with weights 8s. this policy can better update Og during training so that its interventions decrease both the prediction loss Lpred at the end of any intervention trajectory as well as its mean rollout loss Lroll
)f=1
In practice, we construct differentiable categorical sampler p(n) (e(t-11) ((0((-)))(((-1)))) using a Gumbel- Softmax distribution 28 : a differentiable and continuous relaxation of the categorical distribution. To use this distribution, we begin by interpreting (,, 1)) = w (-oo, 0)* as the probability that each concept will be selected for the next intervention. Then, we proceed by sampling noise vector E &&* from a Gumbel distribution such that each dimension h; of this vector is independently sampled from h4 (0, 1). This noise allows us to introduce non-determinism in our system, akin to the use of Gaussian noise in a Variational Autoencoder's 's reparameterisation trick [42). Finally, the crux of the -oabel--oftmax trick is and generating a sharp continuous representation of the sample through a softmax activation function:
this noise shift the -roobbbbabsl predicted by vi
GumbelSof tmax(w: 7) Softmax (w+ + h)/r)
Here, E [0, o0) is temperature hyperparameter indicating the sharpness of the output contin- uous representation of the sample (smaller values leading to samples that are closer to a one-hot representation of vector) and Softmax(w) is the softmax function whose i-th output is given by
Softmax(w),
ビj=1 6
In this process, one can easily sample the Gumbel noise variables hy from Gumbel 1) by first sampling uy ~ Uniform(O, 1) and then computing h; = - log())) In this work, we always use temperature of 7 = 1 and make the output of the Gumbel Softmax discrete (i.e., into one-hot
14

---

representation) using the Straight- Through estimator proposed by Jang et al. [28] which still enables continuous gradients to propaaaee...... Note, this procedure permits one accidentally sample a previously intervened concept from p(n) ((n)/b((((-1 n 1)) (as there may be a non-zero probability assigned to a concept where / ((-1) = 1); to guard against this possibility, after sampling an intervention (/)) from the Gumbel Softmax distribution, we clamp all values of the new intervention mask 1 (1( + -y(1) to be between 0 and for all I.
A.3 Dataset Details
In this Appendix, describe the datasets and tasks used in Section A summary of each task's properties can be found in Table A.1 and detailed discussion of each task can be found below.
Table A.1: Details of all tasks used in this paper. For each task, we show the number of testing samples and training samples. Furthermore, include the shape of cach input, the number of concepts provided (as well as how many groups of mutually exclusive concepts there are), and the number of output task labels. In all experiments, we subsample 20% of the training
make
validation set.
Task
Training Samples
Testing Samples
Input Size
Concepts (R)
Groups
Output Labels
WNIST-Add SWIST-Add-Inoom
12,000
10,000
[12, 28, 28)
72
12
12,000
10,000
[12, 28, 28]
CUB
5,994
5,794
[3, 299, 2991
112
28
200
IncoIn
5,994
5,794
299, 299]
22
7
200
CelebA
13,507
3,376
[3. 64, 64]
6
256
MNIST-Add MWIST-Add a visual arithmetic task based on the MNIST 330 dataset. Each sample [0, 1] 1| in this task is formed by 12 digit "operands", where the i-th channel x,..; of x contains a grayscale zero-to-nine digit representing the i-th operand. We construct this dataset such that some operands are inherently harder to predict. To do so, (i) constrain each of the first four operands to be at most 2, (ii) constrain each of the four middle operands to at most 4, and (iii) constrain each of the last four operands to be at most 9.
Each sample x in this task is annotated with a binary label corresponding to whether the sum of the digits in all 12 operands is least half the maximum attainable (i.e., least 30). We construct concept annotations c E {0, 1 for x by providing the one-hot encoding representations of each operand in as annotation. Because the first four operands be most 2, and the next 4 and 8 operands can be at most 4 and 9, respectively, this results in a total of 4 x 3 + 4 x + 4 x 10- = 72 concepts organised into groups of 12 mutually exclusive concepts. To construct MNIST- set, we generate 12,000 training samples by randomly selecting all 12 operands of each sample from MNIST's 's training set. Similarly, to construct MNIST-Add's 's test set, we generate 10,000 test samples by randomly selecting all 12 operands of each sample from MNIST's test set.
training
MWIST-Add- Incomp is visual arithmetic task that modifies MNIST-Add so that its set of concept annotations incomplete with respect to the task label. This is done by randomly selecting out of the 12 groups of concepts in MNIST-Ade and providing the concepts corresponding to those 8 groups to all models at train-time (making sure the same group of 8 groups is selected across all baselines).
WWVST-Add-Incomme
CUB CUB visual bird- sird-recconntion task where each sample x E [0, 1]3x299x represents an RGB image of a bird obtained from the CUB dataset [31]. Each of these images comes with set of attribute annotations e.g., "beak_ tarke__ttee______ and "bird_ size" as well as a downstream label y representing the classification of one of 200 different bird types. In this work, we construct concept annotations for this dataset using the same k 112 bird attributes selected by Koh et 4 in their CBM work. These 112 concepts can be grouped into 28 groups of mutually exclusive concepts (e.g., "white wing", "black wing", etc). Following (4, we preprocess all of the images in this dataset by normalising, randomly flipping, and randomly cropping each image during training. We treat this dataset representative real-world task where one has access to complete set of concept annotations at train-time.
15

---

CUB- -IB-Incomp is a concept-incompee variant of the CUB task constructed by ran- domly selecting 25% of all concept groups in CUB and providing train-time annotations only for those concepts. For fairness, we ensure that the same groups are selected when training all baselines, resulting in dataset with 7 groups of mutually exclusive concepts and 22 concepts.
CUB-Incomp
CelebA The CelebA task is generated from samples in the Celebrity Attributes Dataset the task and concept annotations defined in [5]. For each image x in CelebA, we first select the 8 most balanced attributes aj,.... as] from the 40 binary attributes available; we then build a vector of binary concept annotations using only the 6 most balanced attributes [a1.-... as], meaning that attributes #7 and ag are not provided as concept annotations. To simulate a concep--incomplete task, we construct a task label y via the base- 10 representation of the number formed by [a1,'.. ag). This results in a total of 1 2 256 classes. As in [5), we downsample every image to (3, 64, 64) and randomly subsample images in CelebA's training set by selecting every 12 sample.
32 using
A.4
Model Architecture and Baselines Details
In this section, we describe the architectures and training hyperparameters used in our experiments.
A.4.1 Model Architectures and Hyperparameters
To ensure a fair comparison across methods, we strive to use the exact same architectures for the concept encoder g and label predictor f across all baselines. Moreover, when possible, we follow the architectural choices selected in [5] to ensure that we are comparing our own InCCEM architecture against highly fine-tuned CEMs. The only tasks where this not possible, as they were not a part of the original CEM evaluation, are the MNIST-based tasks. Below, we describe our architectural choices for f and g for each of our tasks. Moreover, for all tasks and models we include the choices of loss weights X and Aconcept where, for notational simplicity, we use Aconcepr to indicate the strength of the concept loss in both CEM-based models as well as in jointly trained CBMs. Finally, as in (5), for both CEMs and IntCESs we use embedding size of m 16, a RandInt probability pmn 0.25, and single linear layer for all concept embedding generators { of )) and the shared scoring model
UNIST-Based Tasks For all MNIST-based tasks, the concept encoder g begins by processing its input using convolutional neural network backbone with 5 convolutional layers, each with sixteen 3 x 3 filters and batch normalisation layer 43 before their nonlinear activation (a leaky-ReLU [44), followed by linear layer with yout outputs. When working with CFM-based models, the output of this backbone will be fout 128. from which the concept embedding generators {(o= : o7 }}= can produce embeddings {(C, c ) and from which we can construct g. Otherwise, when working with CBM- models, the output of this backbone will be the number of training concepts Joot = k. meaning that y will be given by the backbone's output. For all models, we use as a label predictor fa Multi-Layer Perceptron (MLP) with hidden layers {128, 128) and leaky ReLU activations in between them. We use an MLP for f rather than simple linear layer, as it is commonly done, because we found that based models were unable to achieve high accuracy on these tasks unless nonlinear model was used for f (the same was not observed in CEM-based model as they have higher bottleneck capacity).
For CEMS, InCCEMs, and Joint CBM-Logit models, we use Aconcept = 10 across both MNIST-based tasks. This is only different for Joint --igmodo were we noticed instabilities when setting this value to 10 (leading to large variances) and reduced it to Aconcopt 5 (reducing it lower than this value seem to decrease validation performance). All models are trained using a batch size of B 2048 for maximum of E- 500 epochs. To avoid models learning a majority class classifier, we weight each concept's 's cross entropy loss by its frequency and we use a weighted binary cross entropy loss for the task loss. Finally, for both MNIST tasks, the best performing IntCEM, determined by the model with the highest area under the intervention vs validation accuracy curve when A varies through (5, 1, (), was obtained when A 1.
CUB-Based Tasks
As with our MNIST-based tasks, for all CUB-based tasks we construct concept
encoder 9 by passing our input through a backbone model formed by ResNet-34 45] whose output linear layer has been modified to have Your activations. The output of this backbone is set to Jout k
16

---

for all models and it is used to construct in the same way as described for the MNIST-based tasks. For all models, we use label predictor a linear layer.
For CHMs, IntCEMs, and Joint CBM-Logit models, we use Aconcept = 5 across both CUB-basee tasks. The only exception is in CUB CUB-Incomp, where noticed high variance in Joint CBM-Sigmoid when setting this value to 5 and reduced it to Aconcept = 1 as a response. All models are trained using a batch size of B = 256 for a maximum of E' 300 epochs and using a weight decay of 4 x 10-5 as in [4] As in the MNIST-based models, we scale each concept's binary cross entropy loss by factor inversely proportional to its frequency to help the model learn the more imbalanced concepts. Finally, for both CUB tasks, the best performing IntCEM was obtained when Aroll 1.
CelebA We use the same architecture and training configuration as in the CUB-based tasks for all CelebA models. The only differences between models trained in the CUB-taske and those trained in CelebA, is that we set the batch size to = 512 and train for a maximum of E = 200 epochs models. When
while setting the concept loss weight Aconcape for all joint CBMs and iterating over different values of Avall for IntCEM, we found that Anall 5 works best for this task.
A.4.2 Training Hyperparameters
In our experiments, we endeavour to provide all models with a fair computational budget at train-time. To that end, we set a large maximum number of training epochs and stop training early based on each model's own validation loss. Here, stop early if a model's 'ss validation loss does not decrease for 15 epochs. We monitor the validation loss by sampling 20% of the training set to make our validation set for each task. This validation set is also used to select the sest-perfoming models whose results we report in Section 4 We always train with stochastic gradient descent with batch size B (selected to well-utilise our hardware), initial learning rate Fimitall and momentum 0.9. Following the hyperparameter selection in reported in [5). we use Finitial = 0.01 for all tasks with the exception of the CelebA task where this is set to Finitial 0.005. Finally, help find better local minima in our loss function during training, we decrease the learning rate by a factor of 0.1 whenever the training loss plateaus for 10 epochs.
A.4.3 IntCEM-Specific Parameters
As discussed in Section 3 IntCEM requires one to select a prior for its initial trajectory length and intervention mask, well we describe who we set these values for all of our experiments, Later, we discuss the effect of some of these hyperparameters on our results in our ablation studies in Appendix A.6
architecture to be used for the intervention policy model th. Below
Prior Selection In our experiments, to allow our prior to imitate how we intervene on IntCEMs at test-tine, construct p(p) by partitioning into sets of mutually exclusive concepts and selecting concepts belonging to the same group with probability Pmnt = 0.25. Complementing this prior, we construct the intervention trajectory horizon's 's prior p(T) using discrete uniform prior Unif(1, T where Tmax is annealed within [2, 6 growing by a factor of 1.005 every training step. This was done to help the model stabilise its training by first learning meaningful concept representations before employing significant resources in improving its learnable policy tb.
Intervention Policy Architecture For all tasks, we parameterise IntCEM's learnable policy e using a leaky-ReLU MLP with hidden layers (128, 128, 64, 64) and Batch Normalisation layers in between them. When groups of mutually exclusive concepts are known at train-tima (as in and MNIST based tasks), we let v 's output distribution operate over groups of mutually exclusive concepts rather than individual concepts. This allows for more efficient training process that better aligns with how these concepts will be intervened on at test-time. Finally, when training IntCEMs global gradient clipping to bound the global norm of gradients to 100. This helps us avoid exploding gradients [46] as we backpropagate gradients through our intervention trajectory.
tasks
A.5 Computational Resource Usage
As mentioned in Sections 4 and 5 IntCEMs bear an additional overhead in their training times compared to traditional CEMS due to their need to sample intervention trajectories during training. In this section, we first study how training times vary across all of our baselines. Then we show that
17

---

IntCEM's 's training time overhead is amortised in practice if one considers the efficiency of IntCEM's learnt policy over existing Kate-of-the-aan intervention policies.
Table A.2: Average wall-clock training times per epoch, epochs until convergence, and the number of parameters for basclines in the CUB task. All of these results were obtained by training 5 differently- initialised models for each baseline using the same hardware as specified in Appendix A.
Model
Epoch Training Time (s) Epochs To Convergence Number of Parameters
Joint CBM-Sigmoid
37.75 + 1.99
225.67 + 27.18
2.14 x 107
Joint CBM-Logit
36.76 + 0.70
210.67 + 22.48
2.14 x 107
Sequential CBM
31.08 + 1.17
284.67 + 28.67
2.14 x 107
Independent CBM
31.07 + 1.16
284.67 # 28.67
2.14 X 107
CEM
40.77 + 0.48
174.00 # 34.88
2.57 X 107
InICEM
64.89 + 0.44
224.00 # 26.77
2.60 x 10?
In Table |A2 we show the wall-clock times of a single training epoch for each of our baselines as well as the number of epochs it took the model to converge (as determined by our early stopping procedure defined in Appendix A.44). Our results indicate that, compared to traditional CEMs, InCCEMs take approximately 60% longer than CEMs per training epoch while also taking approximately ~20% longer to converge (in terms of epochs). Nevertheless, we underscore three main observations from these results. First, the results in Table A.2 are very highly implementation-ant hatrriio--nnd-addwareeee -eeendde,, meaning that they should be taken with grain of salt as they could vary widely across different implementations, batch sizes, and machines. Second, although on average we observe that InICEMs take more epochs to converge than traditional CEMs, the variances we observe across different random initialisations are large. This means that these differences are not likely be statistically significant; in fact, we observe some runs where InCEMS converged faster than CEMs. Finally, as seen in Table A.3] these extra training costs may be amortised in practice when one considers that InCEE''s learnt policy is much faster than an equivalent competitive intervention policy like CooP. This implies that if these models are expected to be intervened frequently in practice, the overhead spanning from IntCEM's training procedure may result in better amortised computational resource usage over CEMs intervened on with the CooP policy.
Table A.3: Average wall-clock times taken to query the dynamic policies for trained InCCMMs across different tasks. Notice that all values are shown in microseconds (ms) and that these results can be highly hardware dependent, as they can be heavily affected if one runs a policy in a machine with GPU access. Nevertheless, all of these results were obtained by running all intervention policies using the same hardware as specified in Appendix
T.. A. and averaging over the entire test set.
Dataset
Random (ms) Learnt Policy (ms) CooP (ms) UCP (ms)
Skyline (ms)
MNIST-Add
0.38 0.02
0.37 + 0.02
0.03
0.04
0.44 0.03
AWIST-Add-Incomp
0.33 + 0.02
0.32 + 0.01
0.31 0.04
0.35
CUB
2.91 # 0.13
2.90 + 0.200
4.17
2.99
+
CUB-
3.19 0.39
0.35
0.85
3.08 0.51
4.73 0.27
CelebA
0.70 0.01
0.76 + 0.04
0
0.68 = 0.06
0.76 + 0.06
A.6 Exploring IntCEM's Hyperparameter Sensitivity
In this section, explore how our results reported in Section 4 change as vary several of IntCEM's hyperparameters, specifically, Andli Aconcept- 7, and Pin and Tmax-
We begin by exploring its sensitivity to its two loss weights, Aroil and Aconcapt. and show that, although such parameters have an impact mostly on the unintervened accuracy of InICEMs, our model is able to outperform CEMs once interventions are introduced throughout multiple different values for such weights. Then, we investigate how sensitive IntCEM is to its intervened prediction loss scaling factor 7 and show that its performance varies very little y is modified. Finally, explore how the hyperparameters we selected for IntCEM's ' prior (pm and Tmux) affect its performance and describe similar results to those observed when varying IntCEM's loss weight hyperparameters.
For all the results we describe below, we train IntCEMs on the CUB task while fixing ? = 1.1, Aconcope 5, A L, Pint 0.25, and Tmax = 6 unless any of these hyperparameters is the target
18

---

Task Ancuracy (%)
100
Mean Concepl, AUC (%)
100
90
90
80
70
70
50
50
40
80
20
20
10
Groups Intervened
Aroll
IntCEM Orcnncegt
10)
IntCEM Aconncpt 1)
InCCEM Acanceep 0)
IntCEM (Aconcept
InICEM (Aconcopt 0.1)
CEM (Aconceet
(a) Awencep ablation study in CUB.
Mran Concepl, AUC (%)
100
100
90
Task Accuracy
80
70
60
50
25
GEM
Groups Intervened
Avall
IntCEM (Arai 10)
IntCEM (Arat 1) IntCEM (Ara 0.1)
IntCEM (Mri
IntCEM (rr 5)
CEM
(b) Avo ablation study in CUB.
Figure A.1: Ablation studies CUB for InCCEM's A and Aconces loss weights. Notice that although initial unintervened performance may vary across different hyperparameters, in all instances we observe that InICEMs outperform traditional CEMs (pink diamonds) after being intervened on.
of our ablation. Furthermore, include reference the accuracy of CEM trained in this same task. Finally, when showing intervention performance, all interventions in IntCEMs are made following their learnt policy v while interventions in CEMs made following the CooP policy (the best-performing policy for CEMs based on our experiments in Section
A.6.1 Rollout and Concept Loss Weights Ablations
In Figure A.I we show the results of varying Avoil and Aconcopi in 0,,, 10} in the CUB task. Our ablation studies show that InICEM's initial unintervened performance can vary these two parameters are changed, particularly when Acomcept is small or A is large. Similarly, we observe that the average concept AUC of IntCEM decreases as Aconcept is very small, as one would expect by Acomcopi's 's definition, and is affected as we increase the value of A described in Section 4.1 Nevertheless, and perhaps more importantly, we notice that for all hyperparameter configurations. IntCEM outperforms traditional CEM's intervened task accuracy within five interventions. This observed even when Amm = O or Asoncepe = 0 and suggests that our model's preconditioning to intervention receptiveness is robust to chunges in its hyperparumeters. Moreover, we highlight that IntCEM's sensitivity to Aconcept is not inherent to our architecture but rather inherent to any jointly trained CBMs and CEMs, as shown by Koh et al. 4 in their original CBM evaluation.
Looking our Aual ablation study, we observe that InICEM's concept accuracy and unintervened task accuracy seem to decrease as A increases. We hypothesise that happens as our model has greater incentive depend on interventions, and therefore can afford make more concept and task mispredictions in the absence of interventions. Furthermore, we observe that the variance in IntCEM's unintervened task performance and concept performance as Aroil changes are less significant than that observed when Aconcept changes, in fact achieving competitive accuracies for all values except when A = 10. This suggests that when fine-tuning IntCEM's s loss hyperparameters using a validation set, done in our experiments, it is recommended to leave A within a small value (we found
19

---

Mcan Concept AUC (%)
100
Task Accuracy (%)
90
90
80
70
60
70
50
10 Groups Intervened
20
25
0.5
1.1 1.5
CEM
IntCEM (y = 2)
IntCEM (y= 1) IntCEM (y 0.9)
IntCEM (y = 0.5)
IntCEM (y = 1.5)
CEM
IntCEM (y 1.1)
Figure A.2: Ablation study for the loss scaling factor Our results show that IntCEM's task and concept accuracies robust changes of 7 within reasonable range [0.5.
success with Artl E [0.1,5) unfortunately needs fine tuning for all jointly trained 3BM-based architectures.
5)) while spending more efforts fine-tuning Aconcopa a hyperparameter that
A.6.2 Task Loss Scaling Factor
In Section we defined E [1, oc) as a hyperparameter that determines the penalty for mispredicting the task label with interventions (0)) versus mispredicting with interventions yT() at the end of the trajectory. In all experiments reported in Section 4 we fixed y 1.1. In our ablation study for y (see Figure (.2), we show that InICEM's receptiveness to interventions, and it mean concept AUC, is robust and not significantly affected as y varies within reasonable range [0.5, 2]. In particular, and as expected, we observe that for a higher value of the model has slightly lower unintervened accuracy while attaining slightly higher intervened accuracy after a few interventions. Nevertheless, these differences are small and suggest that our method is robust to minor variations in this hyperparameter.
A.6.3 Prior Hyperparameters Ablation
IntCEM uses two prior distributions to sample trajectories at train-time: (i) p(ps) )) = Bernoulli(paa) on the initially sampled masks (1))) E {0, 13a and (ii) p(T) = Unif(1, Tmax) ) on the horizon, or length, of the intervention trajectory be sampled. Our ablation studies on both Tmax and Pust show that IntCEMs outperform CEMs when the number of interventions increases, regardless of the parameters observe that when Tmax is small, e.g. Tmax high, e.g. Pin = 0.75, InCCEM has a slower increase in performance as it is intervened. We believe this is consequence of model not having long-enough trajectories at train-time, therefore failing to condition the model to be receptive to long chains of interventions. This is because as Pint is larger, the number of concepts initially intervened will be higher expectation, leading to potentially less impactful concepts to select from at train-time. Similarly, as Tmax is lower, IntCEM's trajectories will be shorter and therefore it will not be able to explore the intervention space enough learn better long-term trajectories. Therefore, based on these results our recommendation for these values, leveraging performance and train efficiency, in our experiments) and Pin to a value within [0, 0.5) (we use Pm = 0.25 in our experiments).
used for Pmn and Tmax- Nevertheless,
or Pins is
T Tmax to small value in [5, 10] (we use Tix
A.7 Additional Results and Details on Intervention Policy Evaluation
In this section, we provide further details on the set-up of our intervention policy experiments and include additional results on the impact of intervening IntCEMs and our baselines.
20

---

Mean Concept AUC (%)
100
Task Accuracy (%)
90
80
70
70
15
0.250.5
Groups Intervened
Pint
IntCEM (Pint = 1)
IntCEM (Pint = 0.5)
IntCEM (Pine 0)
IntCEM (Pint = 0.75)
IntCEM (Pint = 0.25)
CEM
(a) Pmu ablation study in CUB.
Mean Concept AUC (%)
100
100
Task Accuracy (%)
90
90
80
70
80
60
70
50
0
10
15
20
25
5
10
15
CEM
Groups Intervened
Tmax
IntCEM (Tmax = 15)
IntCEM (Tmax = 5) IntCEM (Tmax 1)
CEM
IntCEM (Tmax 10)
(b) Taax ablation study in CUB.
Figure A.3: Ablation studies in CUB for IntCEM's priors' hyperparameters. Notice that, in previous ablations, although initial unintervened performance may vary across different hyperparameters, regardless of hyperparameter selection IntCEM outperforms traditional CEMs (shown in pink dia- monds above) within 12 concept groups being intervened.
A.7.1 Intervention Policy Details
Intervening on Logit CBMs Because Joint CBM-Logits use log-probabilities in their bottleneck rather than sigmoidal zero-one probabilities, intervening their bottlenecks is slightly different than intervening on Joint CBM-Sigmoidal models. As in [4], we avoid setting intermediate concepts to out-of-distribution values by setting è to its empirical 95 percentile value (computed over the training distribution) when we intervene on concept e; and want to indicate that its ground truth value 1. Similarly, when intervene concept C; and want to indicate that its ground truth value à = 0, we set it to its empirical 5-th percentile value (computed over its training distribution). This allows us extend the intervention formulation in Equation ( to CBMs with logit bottlenecks by letting at be the 95-th percentile value of the empirical training distribution of è and letting é, be the 5-th percentile value of the same distribution.
CooP CooP selects the best next concept to intervene on by considering weighted sum of (i) an uncertainty measurement of each concept's predicted probability (weighted by hyperparameter a), (ii) the expected change in the probability of the currently predicted class (weighted by hyperparameter 9), and (iii) the cost acquiring each concept (weighted by hyperparameter Yeoop). For simplicity, in this work, we treat all concepts as having the same acquisition cost; i.e., we set Ycorp = 0 and only fine-tune a and 3. We leave the exploration of variants our method which consider different acquisition costs for future work.
21

---

IntCEM
CBM-Logit
CBM-Sigmold
Sequential
Independent
Task Task Accuracy (%)
MNIST
04
79
79
79
79
79
12
74
12
74
12
74
74
12
MNIST Add Incomp
Task: Accuracy (%)
98
98
98
8 8 3
833
93
93
88
88
88
103
CUB Task Aecuracy (%)
a g 8
70
8 8 8
58
1216202442
48
12 26
16202422
48
1216203428
48
12126620244
8 1216202280
CUB fncomp Task Accuracy (%)
97
5 3 3 13
57
57
ColebA Task Task Accuracy (%)
E 3 &
64
64
54
54
54
54
54
24
24
24
24
24
14
14
Groups Intervened
Groups Intervened Groups Intervened Groups Intervened Groups Intervened Groups Intervened
X
Skyline (Oracle)
Random
UCP
CooP
CVA
CVI
Figure A.4: Task accuracy when intervening with different policies (colours) on different methods (columns) and datasets (rows). We observe that, across all tasks and datasets, IntCEMs outperform all other baselines when intervened on. This difference is particularly sharp on the more complicated datasets such CUB Incomp and CelebA.
When using CooP for any of our baselines, we select & and 3 by performing a grid search over (0.1, 1, 10, 100} and B (0.1, 1, 10, 100) and selecting the pair of hyperparameters that give us the best area under the validation intervention curve. Given that this process can be costly (in our runs it may take up 445 minutes run), we estimate the area under the validation intervention curve by checking CooP performance after intervening with 1%, 5%, 25%, and 50% of the available concept groups for a given task.
Behavioural Cloning Policy To learn a behavioural cloning policy which imitates Skyline through
demonstrations the latter policy, we train a leaky-ReLU MLP with hidden layers {256, 128} that maps previous bottleneck č and mask of previously intervened concepts μ to & outputs indicating the log-probar pigprrbbabilities selecting each concept as the next intervention. This model is trained by generating 5, 000 demonstrations ((C, μi), Tsky) of "Skyline".
Each demonstration is formed by (1) selecting a training sample x from the task's training set uniformly at random, (2) generating an initial intervention mask by selecting, uniformly at random, 1 (nif(O, k) concepts to intervene on, (3) producing an initial bottleneck é j(x, H,, c) using concept encoder y of the model we will intervene and x corresponding concept labels and (4) generating target concept intervention 7aky by calling the Skyline policy with inputs (c, (t). This BC model is then trained to minimise the cross entropy loss between its predicted probability distribution
22

---

Task Accuracy (%)
CUB-Income
100
80
60
40
0
2
4
6
Groups Intervened
IntCEM CEM
Joint IntCBM-Sigmoid
Joint IntCBM-Logit Joint CBM-Logit
Joint CBM-Sigmoid
Figure A.5: Test accuracy for traditional CBMs trained with IntCEM's loss functions (InCCBM)) on the CUB-Incomp dataset. Our results suggest that although there may be merit incorporating InCEMM's loss into traditional CBMs, their use of scalar concept representation severely limits their stability and unintervened performance.
and Taky for 100 epochs using stochastic gradient descent with learning rate 0.01, batch size 256, weight decay 4 10-3 and momentum 0.9.
We highlight that when selecting concepts from both the BC policy and the learnt policy s at test-time, deterministically select the concept with the maximum reg-pobbability that has not yet been selected by μ.
A.7.2 Additional Results
As discussed in Section 4 in Figure A.4 we observe that IntCEMs achieve significantly higher task accuracies across all tasks under interventions, regardless of the intervention policy used. We notice, however, that the gap between the optimal intervention policy and InCCEM's best-perfone best-performing policy tends be larger in the simpler datasets. Further work can investigate the origins of this gap as well as mechanisms to bridge it.
A.8 Extending IntCEM's Loss Traditional CBMs
Because IntCEM's loss function only assumes that one can intervene on given model at train-time, this loss function is general enough to be applicable to traditional Joint CBMs. In Figure A.5 we show the results of intervening IntCEM's Looll and Cped terms (InCCBMs). These results show that IntCBMs, in particular when using a logit bottleneck,
joint CBMs whose training objectives have been modified include
achieve a higher test accuracy than their traditional CBM counterparts when presented with large number of test-time interventions. Nevertheless, we note the following limitations: (i) observe that for IntCBMs with sigmoidal bottlenecks, there is improvement regardless of the use of interventions, and in fact, adding Cped and C to traditional CBMs can in fact lead worse performance; (ii) even in IntCBMs with logit bottlenecks whose intervened performance outperforms their CBM counterparts, we observe high variance, especially when the number of intervened concepts is small; and (iii) in comparison IntCEMs, the improvements of training loss in IntCBMs with logit activations are still significantly underperforming when receiving littl-to-none interventions. We believe that these observations arise for two core reasons: first, using concept embeddings allows for a better flow of information between the concept encoder and the label predictor, leading to embedding-
models being more robust to concept incompleteness than their scalar-based based counterparts (especially with respect to sigmoidal bottleneck). Second, using embeddings rather than scalar concept representations enable gradients to flow from the label predictor into the concept encoder even after interventions are made, something that is not possible with traditional CBMs as setting a bottleneck activation to a fixed value (e.g., 0 or the 5-th percentile of the activation) is gradient-blocking operation. This also leads to significantly more stable gradient propagation throughout an intervention trajectory, explaining the high variance IntCBMs. These results suggest that using nigh-dimensional embedding representations for concepts may be crucial
23

---

CUB
CUB-Incomp
CelebA
80
Task Accuracy (%)
70
70
60
60
50
50
40
40
20
30
30
20
20
10
10
10
4
12
16 20 24 28
0
2
8
4
6
8
Groups Adversarially Intervened
Groups Adversarially Intervened
Groups Adversarially Intervened
IntCEM (Leant Policy 4)
InICEM (Random Policy)
CEM (Random Policy)
Task accuracy for InCEMSs and CEMs as we adversarially intervene on an increasingly large number of concept groups (by intentionally selecting the wrong concept label). For IntCEMs we show interventions following its learnt intervention policy v/ as well as a Random policy. For CEMs, all interventions are done following Random intervention policy.
Figure A.6:
component for these models to be receptive to interventions, and further, simply including higher train-time penalty when mispredicting a label after an intervention trajectory is not enough for model become competitive with and without interventions.
A.9 Effect of Adversarial Interventions
In their evaluation, Espinosa Zarlenga et al. observed that CEMs were somewhat more robust to adversarial interventions (i.e., interventions that are intentionally incorrect) compared to existing CBM architectures where robustness was defined as being able to withstand certain number of wrong interventions before significantly dropping in performance. This suggests that CEMs may have a form of error correction in their embeddings that allow the label predictor to correct intervention mistakes. In contrast, as posited in our discussion in Section 5 by encouraging IntCEMs to be receptive to test-time interventions, we are also incentivising them to be receptive to all forms of interventions, even if those interventions are adversarial in nature. This is because explicit train-time intervention loss, which encourages IntCEMs to positively respond to test-time interventions, has the underlying assumption that interventions, when provided at train-time or test- time, are correct. Hence, we hypothesise that the same error correction observed in CEMss will not be seen in IntCEMs, leading models' performance to significantly drop when interventions wrong. We find evidence supporting this hypothesis when performing adversarial interventions across all our baselines the CUB, CUB Incomp, and CelebA tasks. Specifically, observe that an InCCEM's performance drastically drops when receiving only a handful of adversarial interventions, with this result being even more damaging if we follow its learnt intervention policy vh. These results suggest that a very interesting and impactful direction for future work can be to explore mechanisms introduce error correction IntEE-llike models.
model's
in Figure A.6
A.10 Leakage in InICEM's Learnt Representations
Concept leakage 36 10 is known issue in CBMs where CBM's concept encoder learns to encode unnecessary information in its learnt concept representations in order to bypass information from the input its label predictor. It has been empirically shown that leakage may lead to less interpretable concept representations (36] and to detrimental interventions on CBMs [10]. The latter a consequence of a CBM's label predictor learning rely on information that accidentally leaked into the concept representations, information that is necessarily destroyed at intervention- time when an expert overwrites a bottleneck activation with its ground truth value. Nevertheless, the same is not true in embedding based architectures, such as CEMs and InCCMMs, where interventions simply involve "swapping" an embedding with one that can still carry leaked information into the label predictor (see Equation []. Hence, in this section we hypothesize that higher leakage may, in fact, be a contributor to IntCEMs' better performance when being intervened on as the model may learn to take better advantage of this leakage to learn to be more receptive to interventions at test-time.
24

---

Table A.4: Oracle Impurity Score (OIS) for all fointly-trained baselines across all tasks, Higher OIS values indicate higher leakage in a model's learnt concept representations. Joint CBM-Sigmoid
Dataset
IntCEM
CEM
Joint CBM-Logit
MNIST-Add
30.97 0.29
28.25 0.44
13.14 = 0.27
20.96 = 0.02
MNIST-AdddIIncmp
36.73 # 0.23
34.09 + 0.47
13.28 # 0.27
26.12 # 0.08
CUB
45.86 2 0.29
42.54 2.30
21.01 0.58
41.66 # 0.49
OIS
CUB-Incomp
38.79 1.41
42.13 # 3.48
27.81 # 2.02
30.32 # 3.09
CelebA
50.87 4.07
40.63 4.83
29.65 = 16.51
24.11 = 09.30
We evaluate our hypothesis by measuring the Oracle Impurity Score (OIS) [00 of concept represen- tations learnt by CBMs, CEMs, and InCCEMs across all tasks. This score, between and measures how much extra information, on average, each learnt concept representation captures from other possibly unrelated concepts (with higher scores representing higher impurities and, therefore, more leakage between concepts). Our results, shown in A.b show that not only is there significantly more leakage in CEMs compared to CBMs, as one would expect given their higher capacity, but InICEM's embeddings are capturing more impurities than CEM's embeddings across all tasks but one, providing evidence towards our hypothesis. This preliminary study suggests that contrary to common assumptions, leakage may not always be undesired and could be a healthy byproduct of more expressive concept representations in models that accommodate such expressivity. Nevertheless, this phenomenon may also potentially lead to IntCEM's ss concept representations being less interpretable than those CEMs when such leakage is unaccounted for. Therefore, we encourage future work to explore leakage's 's positive and negative consequences in embedding-basee bbasiidgg concept models to design better inductive biases for effective interventions.
A.11 Frequency of Selected Concepts by IntCEM's policy
we show the frequencies of concept groups selected by IntCEM's v policy when intervening on the validation set CUB. To study how much this policy changes across different initialisations and random seeds for the same hyperparameters and models, show these histograms across multiple instances identically trained IntCEMs with different random seeds. We notice that although, in most cases, concepts such "upperparts colour" and "size" seem to be highly preferred in the early steps of intervention, there seems to relatively large variance across multiple seeds. This may suggest that there several equally informative good concepts may request at given time to obtain similar improvements, something that we leave for future work to explore.
In Figure A.7
A.12 Tabulation of Results
This section presents Tables A.5 A.6, and A.7 set of tabular summaries of our results introduced in Section 4 These summaries show the precise means and standard deviations (computed over five different random seeds) of test-time accuracies after interventions for all baselines and tasks when deploying various intervention policies. They represent tabular versions of the results presented in Figures
4 and respectively.
25

---

Solected Counts
Leart: Policy Selection at Stop
Salected Counts
Learnt Policy Selection. After Steps
Selected Counts
Learat Policy Selection After 12 Steps
2010
5000
2000
2300
Concept Group
Concopt Group
Concept Group
Solected Counts
Lears: Policy Selection at Step
Saleccted Counts
Learnt Policy Selection. After Staps
Selected Counts
Learat Policy Selection After Stops
6000
saco
Good
2000
D0 Rom
UN
Concept Group
Concept Group
Concept Group
Polac Selection at Step
Counts
Policy Selection Alter Steps
Learat Polcy Selection After Stops
6000
Solected
800
2000
Solected
2am
Frgers
Concept Group
Concept Group
Concept Group
Solected Counts
Policy Selection st Step
Salected Counts
Learnt Policy Selection. Steps
Salected Counts
Learat Policy Selection After Steps
Good
6000
2ap0
Concept Group
Con.egt Group
Coacept Group
Learst Policy Selection st Step
Salecttd Coants
Learnt Policy Selection. After Steps
Salvcted Counte
Learat Policy Selection After Steps
4000
Solected
500
2000
2000
Concegt Group
Concegt Group
Concept Group
Figure A.7: Frequency of selected concept groups by InCCEM's learnt policy 1e after given number of interventions in CUB's 's validation set. Each column represents the distribution of intervened groups after fixed number of interventions. Similarly, each row represents different initialisation and seed for lntC before training.
26

---

Table A.5: Tabular summary Figure B
Task predictive performance (aceuracy or AUC in %) across all tasks and baselines when intervening on a fixed fraction of the set of available concepts at test time (indicated on the left-hand side of the dataset names). Except for "InCCEM (Learnt Policy 4)", all interventions use random intervention policy. We write in bold the best performance results
across baselines following a random intervention policy.
Dutaset
InCEM (Random)
CEM
Joint Sigmaid
Logit
Ind.
Seq.
[Learn Policy
MUUSTTMMd -dd
93.78 0.09
91 0.41
66.47 + 1.11
79,86 3.12
54.11 + 1.09
88.88 a 0.67
94 75 0.25
255
XAIST Add Incoxp
90.88 0.25
0.13
84.00 0.36
88.22 0.92
83.18 0.22
84.68 0.83
77.59 1.57
0..76=0000
= 1.57
56.06 a 0.77
65.40+
4..5
91.69 a 0.29
0.49
CIB-Incunp
0.34
7 0.53
6094 3.66
62.97 7.19
4
0.36
25.63
26
95.70
13.36
1:20
2.330
53.71
98.7
505
MA15T-AA6-- Incorp
CUB
96.14 0.35
0.21
89 1.53
0.96
85.73 1.12
92:28
0.74
77.96 1.73
36.37 1.18
92.19 0.77
74.24 5:68
92.98
99,17 0.14
CTE Iucoup
85.86 2 0.29
82.32 0.58
66.12 3.81
66.30
54.35 2 0.35
4 1.85
S0069 0.30
OclebA
$2.01 = 0.45
38,82 1.52
27.19 1.37
21.20 2.74
27.79 + 1.07
29.11 0.40
52.01 1.21
RSLST Add
97.51 0.08
95.01 0.37
91 1.05
S1.31 3:45
56.97 a 0.9h
91.38
95, 2 01/9
75%
MH157 Add- Incoxp
CUB
98.98 = 0.14
93.48 0.25
93.95 = 1:95
90.35 0.96
95.73 1.00
68.09 1 1.00
91.18 1.61
78,02 1.90
96.97 0.53
58. 15 0.68
90.09 :
79.81
4:996
94, 1 0.24
99 0U03
CLR-- Incomp
91.74 a 0.19
86.93 0.61
71 54.44 5.03
6.336
61.92 0.52
51.00 1.76
96 0126
CallbA
56,99 a 0.37
41.99 1.39
28.04 1.42
23.20 3.82
28.63 144
50.04 0.48
66.48 1.10
NOIST-XXlH
0.04
3434
3.69
53,45 103
$4.35
100%
NHIITT-RAA Incoxp
0.11
99.00 0.04
1.73
1:34
1.95
08.97
3.72
CUB Incomp
96.52 0.19
0.67
69:54
96.52
CelebA
70.02 0.62
48.10 1.72
20 1.30
30.75 99.89
29.45 2.02
$1.02
70.02
Table A.6: Tabular summary of Figure 4 including performance in other tasks as shown in Ap-
pendix A.7 but excluding performance when using the "Random" policy (see Table AA3 for those results). ) Here, we show the task accuracy of IntCEMs and CEMs when intervening with different test-time policies as we vary the fraction of concept groups intervene on (indicated on the left-hand side of the dataset names).
H818T AMM
SEID 4.35
0:.0
43 10 11
51.00 0:41
01.38-000
0:33
45.13-00 70
50:07 -0
93.57 01220
18.55 0.00
9488 11.25
59.09 27.75
56.91 1.5
$4.52 20.55
89:44 1.10
$5.01 2.350
99:35 2015
95.15 1.81
CUB-22 2eccapcpppp
2..31
50.00 0.22
62 21.4
80277 17.47
29.09 0.55
18.58 20.31
1906 2.33
75.35.2.033 55
50.15 20:31
23755
500
MITST- M-0- "eooin
07.57 13.55
11.10 6g
10:07 PM
01.10-11
01 51 0..50
03
19 13.00-10 11
91 70 0.46
0:30
01:71 17
10.00-130 51
01.13 0.
07.05 00100
80
0-1 Km
91 -0 18
00:07 0011
I:: 1 0.69
W:ST
Caloba
65.47 8 0.50
46.20 46.29=1..0
65.47 11.T
1.555=1.59
20.50 0.34
42.68
50.54 0.50
41.6 0:
70.64 0047
48.56 1,33
25 10.0
05-11 77
01100 15
19:31 11.6
11
18 10
11
1001091 11:00
191127-Ad5-- -oompp
00222
91.08
10:43:1.00
2015
08.51 0074
1.62
1.55
CUB Jeccnppppppp
50.23 0.30
95.37 0.33
96.24
6.70
02.32
05.35
95.81 0.45
$5.37
96.65 00225
91.83 (u99
08.52 0.67
47.43 0 163
63
9.55
11.51:1.66
G4 0.7
45.55 1.40
54 45 0.47
45 46
70:55 -0:52
43.0 + 1,65
HD-ST- AM
9031
20
20.0
1005
NIEST-
91.99 1111
0138 11
91.99
41.53-1.0
0:11
01:53-110
01.90 -0.1
91.55 1.0
M6 11.04
96.79
1-72
96 11.04
48.75 1:72
M9 91 0004
06.75 1:75
90.0 0101
05.75
1.72
09:00 0145
W: 15 177
11.9
01.47 0.87
11.9
43.47 11877
0LE9
99.12 0.19
00.13 0.19
4
Table A.7: Tabular summary of Figure
Here, we show task performance when intervening on InCEMSs following test-time policies v CooP, Random, and BC-Skyline. The fraction of concept intervened for each dataset is shown the left-hand-side of the table.
Dataset
Random
CooP
Learnt Policy
BC-Skyline
IntCEM no şi (Random)
MNIST-AGE
93.78 0.09
0.14
94.75 + 0.28
93.20 + 0.19
93.81 0.18
ANIST-Add-Inee
90.88 0.25
90.09
91.69 = 0.39
90.18 0.35
90.98 0.18
25%
CUB
$8.5320.22
0.25
94.10 = 0.49
88.43 0.29
88.46 0.46
CUcoIn
0.34
0.23
81.71 0.23
79.06 0.41
78.15 + 0.30
CelebA
12.25
50.72 0.56
47.63 + 1.49
48.47 + 0.47
33.45 + 0.58
HNIST-Add
0.17
94.74
96.71 0.20
94.56 0.47
95.69 0.11
50%
-Ad-T-AAd-IIcomm
92.07
91.31
0.59
92.98 + 0.31
91.56 0.43
92.13 a 0.16
CUB
96.14
0.11
0.144
96,28 0.19
95.89 + 0.40
CUB-Incomp
0.28
0.13
90.69
8.444
85.31 + 0.41
CelebA
52.01 0.45
65.47
62.01 + 1.21
59.95 + 0.84
40.71 + 0.56
MNIST Add
97.51
98.22 0..9
96.39
97.50
75%
MNIST- MNIST-AAddIIncop
93.48
9,82
04.18+0.24
93.38 0.39
93.54
CUB
98.98
99.88
99.82
99.67 0.10
98.87 + 0.26
Incomp
9..74
0.33
04.9600.22
93.22 +
91.53 +
CelebA
56.99
6.654
66.48
64.63 0.93
4..37
MNIST-AC
0.004
99.01
0.04
99.51 0.04
100%
M -nccomp
94.99
94.99
94.99
9.19
95.03
CUB
99.90
99.90
99.90
9..90
99.85
CUB- Incomn
9.15
9..9
96.52
9..
96.20
CelebA
7.602
7.662
7..02
0.72
5..39
27


Reviews:
[{"review_id": "aCRdgevpIF", "reviewer": "Reviewer_Zgwp", "summary": "The authors introduce IntCEMs, an extension of Concept Embedding Models designed\nspecifically to react correctly to external interventions to the learned concepts.\nCompared to regular CEMs, IntCEM feature two additional elements:  a policy that,\nessentially, guesses what interventions a human expert would do on the model for\nany given input, and a penalty that regularizes the model to achieve high accuracy\nunder interventions.  The idea is to ensure the model performs well even after\ninterventions - under the assumption the model requires interventions to ``ask\nfor help'', i.e., to obtain the ground-truth label of certain concepts at test\ntime.  Experiments show that in fact IntCEMs outperform CEMs and CBMs on four\ndata sets at test time under interventions, even from a different policy.\n\n**Post-rebuttal update**: Increasing the score by one point under the assumption\nthe authors will fairly highlightly leakage as a limitation of CBMs/CEMs/IntCEMs\nand its interaction with interventions in the paper.  I am still a bit concerned about\nimpact, but the contents of the paper are good quality overall.", "questions": "Q1. It was shown by:\n\n  Mahinpei et al. \"Promises and pitfalls of black-box concept learning models\", 2021.\n\n  Margeloiu et al. \"Do Concept Bottleneck Models Learn as Intended?\", 2021.\n\nthat CBMs suffer from \"concept leakage\", whereby learned concepts encode information from unintended sources (like unobserved concepts), with negative consequences of interpretability.  I am wondering whether 1) How IntCEMs fare in this regard; specifically, I am only aware of two self-interpretable models that specifically address this, namely:\n\n  Havasi et al. \"Addressing Leakage in Concept Bottleneck Models\", 2022.\n\n  Marconato et al. \"GlanceNets: Interpretable, Leak-proof Concept-based Models\", 2022.\n\nA brief comment on IntCEMs vs leakage would be welcome - this is an important issue.  2) Robustness to interventions can prevent leakage - the link is that the model is specifically trained to be robust against replacing a (possibly leaky) concept prediction with a (presumably non-leaky) concept annotation.\n\nQ2. Why did you compare only against CEMs and CBMs?  There are other self-interpretable models out there, like the two aforementioned works, part-prototype networks, concept whitening, self-explainable neural networks, and many others.  They all support interventions.", "limitations": "I think so, in section 5 there is a brief discussion on some limitations (e.g. time complexity).  The issue of concept leakage is not mentioned though.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ English is good, narrative is generally good but parts of it are confusing, see below\n+ Motivation is sensible\n+ Proposed approach is also sensible\n+ Empirical evaluation is generally positive, with only some minor drops in concept quality depending on choice of \\lambda_roll\n+ Related work is sufficient", "weaknesses": "- One issue is limited significance, in the sense that the authors tackle a problem that affects only a certain type of operation (namely, a *specific* type of interventions - not all possible interventions) performed on a certain type of model (CEMs, CBMs).  It is true that the performance gain of IntCEM vs CEM under interventions can be substantial (CelebA in figure 3), but it is generally more modest (CUB, CUB-Incomp; the diff in perf for MNIST-Add and MNIST-Add-Incomp are somewhat biased by the difference at Groups Intervened = 0, which is not due to extra robustness to interventions).  This is not a deal breaker, and indeed my score is positive, but it explains why I decided not to go above weak accept.\n\n- Writing is a bit confusing.  It took me a while to understand where the ``learning to receive help'' fits into the abstract and introduction.  The idea is that models can ask users to help them at test time by - essentially - requesting the ground-truth label of certain concepts, which means that this work is closer in spirit to concept-level active learning (but at test time) rather than to interventions proper.  These have a causal connotation and can be used for changing the concepts to *any* value, not only to their ground-truth values.  See for instance all the literature on algorithmic recourse.  The target task should be made more explicit, at the bare minimum in the introduction.  I also feel the text abuses greek letters (all of section 3) a little, hindering readability.  Every time I see, say, \\eta, \\psi, \\omega... in the text, I have to look back at their definition.  The $c$ letter also appears in too many variants.  More generally, the text feels quite compressed.\n\n- Evaluation is only carried out against CEMs and CBMs.  Other self-interpretable models exist and have been published recently.  Considering the focus on CBMs, this is not a major issue, but it is an issue nonetheless.\n\nMinor issues worth fixing\n-------------------------\n\n- line 102: the range {0, 1}^k is wrong, \\tilde c_i is set to 0.5.\n\n- Considering \\mu_i also appears in the definition of \\tilde c_i, Eq 1 is probably too complex for what it needs to achieve.  Is it possible to   simplify it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Zgwp,\n\nThank you for your very insightful feedback. We are glad you found our paper’s motivation, evaluation, and relevant work interesting and generally positive. Below we address your questions.\n\n### More explicit definition of  “interventions”\n\nThis is a great point. We want to clarify that, as you correctly mentioned, the term “intervention” here is not used as in the causality literature and is more related to the field of active feature acquisition (e.g. [1]). This aligns with your intuition regarding seeing this as a form of concept-level active learning at test time. Nevertheless, the concept learning community widely uses the term “concept intervention” for the operation in which an expert provides a binary concept label at test time for a CBM-like model. Hence we maintained the term to avoid confusion within this paper’s “host” field of study.\n\nTo address possibly confusing future readers from other subfields, we will state what we mean with the term “concept interventions” early on in our introduction. Furthermore, we will elaborate on how these differ from their causal counterparts and how they are related to active feature acquisition in our background section.\n\n### IntCEM’s performance gains\n\nThe scale in some tasks in Figures 3 and 4 is large as we needed to fit baselines underperforming w.r.t. IntCEM. However, looking at CUB’s performance values in a table view of these figures (see Table 1 of our supplement), we observe a significant absolute boost of 6.5% for IntCEM vs CEM when randomly intervening on 50% of concepts (we achieve an almost perfect performance after randomly intervening on 75% of the concepts). What is more important, however, is that by intervening following $\\psi$ rather than performing just random interventions, IntCEM’s absolute gains in CUB when intervening with 50% of the concepts shift to ~9.5% (with an almost perfect acc of  99.17%).  This boost comes with negligible test time computational costs and without the need to (re)learn any further policies.\n\nAs for the MNIST tasks, these tasks showcase how our training procedure can lead to richer embeddings that allow **performance boosts even without interventions**. We will add a small sentence to highlight this in Section 4.1. Moreover, to make this analysis easier for external readers, we will include Table 1 of our rebuttal supplement in a new Appendix.\n\n### Evaluation baselines\n\nTo the best of our knowledge, the main non-graph-based prior works that have evaluated concept interventions are CBMs (they introduced concept interventions formally), Concept-based Model Extraction (CME) [2], CEMs [3], and leakage-free CBMs [4]. Leakage-free CBMs avoid using soft representations that are prone to leakage by using hard representations for their bottleneck together with an optional side channel and training procedure that intentionally “flows/leaks” information not captured by the hard known concepts. Nevertheless, although these additions are noteworthy and interesting, this model effectively operates like a CBM in practice and the extra leakage bypass it offers is a constrained version of the bypass allowed by embeddings in CEMs. Similarly, CMEs were shown to underperform vanilla CBMs when intervened on [2] and are post-hoc explainability methods rather than interpretable architectures. Because of these reasons, we focused on CEMs and CBMs as they capture the key mechanisms in all of the methods mentioned above.\n\nWe indicate why we did not include other concept-based interpretable methods as baselines for our evaluation:\n\n- GlanceNets: Although GlanceNets support concept interventions, these go beyond the original scope of the architectural changes between GlanceNets and CBMs. The lack of intervention experiments in GlanceNet’s paper supports this. Furthermore, concept interventions in GlanceNets are not too different from those in Hybrid-CBMs [3], which are known to underperform w.r.t CEMs.\n\n- Concept Whitening (CW): To the best of our knowledge, Concept Whitening does not support concept interventions as it is unclear how one would set entire feature maps at test-time to trigger a concept intervention. This difficulty is even greater when one considers that CW usually performs a reduction over these maps to determine a concept’s “activation level”. Notice that their manuscript does not allude to concept interventions as they slightly precede CBMs.\n\n- Self-explaining Neural Networks (SENNs) and Prototypical part networks (ProtoPNets): Both of these methods work in a concept unsupervised manner (they do receive feedback from task labels but not from concept labels). Therefore it is not immediately clear how to perform concept interventions given that one needs to assign semantics to the discovered concepts first. Such a task is non-trivial to evaluate and goes beyond the scope of our own work.\n\n- Other more recent methods (label-free CBMs, post-hoc CBMs): There are very recent methods that do support concept interventions (e.g., label-free CBMs and post-hoc CBMs). However, these were published less than two months before the submission deadline (at ICLR 2023), which meant that we could not reasonably include them as part of our evaluation.\n\nWe will clarify these choices in our evaluation section by justifying our baselines as done above.\n\n### Misc minor fixes\n\nThanks for finding this typo! As for Eq 1, it could be simplified by using a piecewise definition, however our original intent of expressing Eq 1 as it is was to make it clear how one may right a concept intervention operation in a differentiable manner.\n\n## References\n\n[1] Li et al. \"Active feature acquisition with generative surrogate models.\" ICML 2021.\n\n[2] Kazhdan et al. \"Now you see me (CME): concept-based model extraction.\" arXiv:2010.13233 (2020).\n\n[3] Espinosa Zarlenga et al. \"Concept embedding models\" NeurIPS 2022.\n\n[4] Havasi et al. \"Addressing leakage in concept bottleneck models.\" NeurIPS 2022."}, {"review_id": "jyQVG48hJd", "reviewer": "Reviewer_oMUi", "summary": "The authors in this paper proposed a novel method of improving test-time interventions for Concept Bottleneck Models (CBMs). Although many CBM works showcase their ability to do intervention, none of them explicitly motivate the learned model to do well on intervention during training phase, hence hindering their ability and reliability to do intervention correctly. This work builds on top of concept embedding models (CEMs) and constructs an additional probability distribution (parameterized by neural networks) to learn what concepts to intervene on. To learn this distribution, the authors proposed a composite of three losses, where the first loss (Rollout loss) follows a greedy strategy of selecting concepts that yield the maximum probability of the given class, the second (task prediction loss) penalizes wrong predictions with many interventions, in which the penalty scales with the number of intervention performed. Finally, the third loss concept loss is just binary cross entropy to promote the correctness of the concept prediction. Empirical results on MNIST and CUB (and their variants) show that the proposed formulation performs competitively against state-of-the-art CEMs/CBMs that also utilizes interventions to improve performance. The authors also argued that the sampling from the learned distribution is much more scalable than current state-of-the-art methods. ", "questions": "1. I understand that “the single MC sampling per training step” is valid, but what is the main motivation for doing so? Is it to improve training speed? And do you speculate, (even better, show with ablation studies) that improving this estimation can improve convergence when optimizing your objective? \n2. In Figure 3, can you explain  why intervention could cause a drop in performance for Join CBM-Logit? Since the proposed objective explicitly optimizes for CE after intervention, would you agree that your method trained to prevent a drop in performance after intervening from happening?  \n3. How long does it take to train typical model with the proposed objective? I would imagine the rollout loss can be slow when T is large.  Does restricting T to be small heavily impacts the performance of the trained model? \n", "limitations": "The limitations of this work is addressed in the main manuscript and is well-stated. The answer to my questions above can potentially brought up as limitation, and if so, I would suggest the authors to add that to the limitation section. ", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written and is organized in a chronological and clear manner. Introduction sufficiently motivates the problem formulation. The background and related work is written neatly so that the paper is sufficiently self-contained. The description of the method and notation is also clear and easy to follow.  And experiments have clear objectives and empirical results clearly demonstrate the effectiveness of their method. \n\nGiven CBMs are a very popular topic, and that there is a lack of work in the current field that focuses on improving test-time interventions in CBMs, I consider the formulation proposed in this work  of significantly novel. It definitively addresses a limitation of CBMs/CEMs and proposed a good formulation that addresses that limitation. \n", "weaknesses": "There are no major weaknesses in this paper. But a few things that I suggest the authors to add for the sake of completeness:\n\nAs the main selling point of interpretable-by-design methods are interpretability, it feels a bit odd that authors did not add any examples of interventions on a particular sample. Even though the metrics clearly show the method is effective, a few concrete test samples and use of intervention can better illustrate what interventions do, how the posterior of the task prediction changes before/after interventions, etc. Potentially, illustrations could help readers who may perhaps be from a different field. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oMUi,\n\nThank you so much for the very encouraging review and the extremely valuable feedback that came with it. We are very glad that you found our work significantly novel, well-written, and clearly motivated. Below we answer the questions you raised in your review.\n\n### Motivation for single MC sample\n\nAs you rightly suggested, the primary motivation for a single MC sample is efficiency at training time (as this has to be done for each training step). During early experimentation with our method, we did some informal studies on how the number of samples affected our model’s performance. We noticed that the gains from more MC samples are marginal in practice yet have a significant cost to wall-clock training times. Intuitively, we believe this can be because a sample is seen multiple times during training (across different epochs) and therefore there is not much benefit in adding more mask samples during a specific epoch if the same sample will be eventually seen again in a future epoch (and we will sample different initial intervention mask then). One way to think of this effect is akin to how increasing MC samples during a VAE’s training has marginal improvements while making training times longer. As such, we opted for a single MC sample.\n\n### Detrimental interventions in Joint-CBM-Logit\n\nThis is an excellent and interesting question. Our hypothesis is that traditional CBMs can react negatively after an intervention is performed because of *concept leakage* (see our global reply for more information regarding this). Leakage occurs when a CBM-like model learns to exploit a concept’s continuous nature to encode unnecessary information as part of a concept’s representation. Such impurities can have detrimental effects when intervening especially if the intervention operation results in a destructive state of this continuous space (e.g., setting a concept probability to $1$ or $0$ as done when intervening in a Joint-CBM). This is because such an operation would destroy any extra information that the downstream label predictor might’ve otherwise exploited to predict the output class, leading to detrimental concept interventions. When using logits over sigmoidal bottlenecks, we allow much more flexibility in the concept activations and therefore the chance for concept leakage is higher (and also a possible reason why the same drop in Figure 3 is not observed for Joint-CBM-Sigmoid). Notice furthermore that when the set of concept annotations at training time is incomplete w.r.t. the downstream task (as in CUB-Incomp), the chances of leakage in traditional CBMs are much higher as the model’s bottleneck will be underprovided and therefore the model will have to make trade-offs between task accuracy and concept purity/accuracy.\n\nSuch leakage occurs in IntCEMs; however, as you mention, it is unlikely to have a detrimental effect on the model’s performance after the intervention is performed due to how IntCEM is trained and how our model operates when intervened on. This is because (1) we incentivise our model to be reactive to interventions at train time with the hope that they will learn to be more accurate with very little external feedback at test time, and, more importantly, (2) interventions in IntCEMs are **not destructive** as we only change the embedding that we pass into the downstream label predictor, allowing the model to still exploit extra information encoded in the learnt embeddings even after the intervention has been performed.\n\nWe realise this could have been better explained in our experimentation section and will include this leakage-motivated discussion and hypothesis in our updated discussion section.\n\n### Training time of IntCEM and effect of $T$\n\nWe studied the effect of IntCEM’s regularisers on its training times in Appendix A.5 (particularly Table A.2). To summarise, we observed that the training times in IntCEMs increase by about 60% compared to CEMs. However, it is worth noting that we observed a large variance in training times (with even some IntCEM runs converging faster than CEM runs!) and that the extra training costs of IntCEMs amortise in practice as running the learnt intervention policy is much more efficient than equivalent post-hoc intervention policies (see table A.3 for details). Training time computations are also implementation-specific so they could be further optimised in the future.\n\nAs for the value of $T$, you are correct in stipulating that this affects both training times and performance. We included an ablation study showing the effect of $T$ in our model’s performance in Appendix A.6.3 (particularly Figure A.3). There, we observe significant performance differences only when T is very small (e.g., $T=1$) relative to larger values of $T$ (although notably, IntCEM is still better than CEM in this instance after a larger number of interventions). In practice, we observe that increasing $T$ above a small value (around $5$) results in negligible gains. Some intuition for this result is that one has fewer “interesting” concepts for a given sample to select from after long trajectories, leading to diminishing returns as we increase the value of $T$.\n"}, {"review_id": "3LASOUzt55", "reviewer": "Reviewer_23Ef", "summary": "The authors proposed a modified version of CEM that is better at receiving human test-time intervention by explicitly incorporating intervention into the training stage. Specifically, an intervention prediction module is trained to behavior clone an optimal-greedy intervention policy (Skyline). The CEM is trained to optimize for the correctness of the initial guess as well as the prediction after the predicted intervention. The idea of the method is simple and straightforward. Extensive experiments explore design choices and show strong results in practical settings.", "questions": "* How severe is information leakage for CEMs? This is tangent to this work's contribution but since this work is based on the CEM framework, perhaps the authors could comment on whether IntCEM succeeds because of the leakage of downstream task information into the concept embeddings?\n* Typo:\n    * P2L39 artefact -> artifact", "limitations": "Yes.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well-written. Extremely easy read and all concerns I have about the method are covered in experiments.\n* Simple and effective idea anchored on the premise that models not explicitly trained to receive intervention might not handle intervention well (empirical support in  Fig 3, 4).\n* Questions proposed in P6L221 - 230 effectively guides reader through the work and the authors' through process.\n* The authors care about how this method could realistically be applied in real-world settings.\n    * Heuristics for tuning hyperparameters for the losses are provided. \n    * Considers realistic settings of applying IntCEM where human-intervention might not be greedy-optimal and show empirically that IntCEM still performs better than baselines under non-optimal interventions (even random intervention).\n    * Observe desirable properties for application in the real world (P7L265)", "weaknesses": "* The only weakness is lacking human study but this could be conducted rather straightforwardly. Given the extensive study on intervention policies, I believe this to be merely nitpicking.\n* This is obviously a work that would induce plenty of follow-up work. Perhaps adding a section for what types of design choices have not been explored yet (e.g. other RL methods besides BC) could help stimulate the research community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 23Ef,\n\nThank you so much for the very encouraging review and the valuable feedback and suggestions. We are glad you found our work very easy to read, well-motivated, and friendly for reproduction/real-world use. Below we discuss some of the points you raised in your review, including answers to your questions (which led to what we believe are some important new additions to our manuscript!). \n\n### Section detailing future work\n\nWe agree this is a useful addition. We will include this section in our updated manuscript outlining some of the future directions we have discussed in our rebuttal with you (e.g., user studies and a better understanding of leakage) and the other reviewers (e.g., evaluation of large-scale concept-based models).\n\n### Lack of user study\n\nPlease see our global rebuttal reply for a reply to this concern. \n\n### Information leakage and IntCEMs\n\nWe believe that your intuition on how leakage may play a part in IntCEM is correct and we present some evidence in favour of it in our global reply for this rebuttal. These results will be included as a new appendix in our updated manuscript.\n"}, {"review_id": "oJIuMn29px", "reviewer": "Reviewer_4KHx", "summary": "The intervenability of concept bottleneck models has been taken for granted thus far. There have been numerous methods that tried intervening or propose policies to intervene. However, previous methods did not go so far as to optimize models for interventions. This work proposes intervention-aware training and extends Concept Embedding Models for this purpose. With experiments across MNIST, CUB, and CelebA, the authors demonstrate that IntCEM achieves on-par downstream performance compared to CEMs, and investigate the intervention performance with two sets of automated studies.\n\n**After the rebuttal:** I read and acknowledged the authors rebuttal. I am leaning towards an acceptance position after the rebuttal, given that the authors clarified my questions. I find the paper's message interesting and the presentation to be clear and well-written, the reason I am not giving a higher score is the lack of real-world settings to test the method in more complex settings and where the models do not need concept annotations at the training time.", "questions": "1. In terms of the story, while the premise of intervention awareness sounds interesting at face value, I still think the case needs to be made for such a need. I believe it could help to give a concrete use case before laying out the method. E.g. L118-119 authors mention an interesting example, “may be hard for the model to predict on its own”. Can authors make it more concrete, give real world examples / formalize this? When would it hurt to not be intervention aware? When would we benefit from being intervention aware? \n\n2. I would like to ask the authors’ reasoning about the first weakness I raised in the Weaknesses section. If intervention awareness is indeed to improve the interaction between the model and the human, how should we think about the current results? Is there a way to deduce that models are more ready for humans? Can we infer that humans would prefer this notion of intervention awareness over previous methods (or not having it)?\n\n3. In my understanding, according to the loss function after L194, the selection policy $\\psi$ is optimized to select the concept that would increase the probability of the ground-truth label being predicted the most, upon the intervention. Would this not bias the selection policy towards more typical interventions for the class (e.g. where $P(Y|C)$ is high)? For instance, say that a bird specie X is 95% of the time red but 5% of the time blue, whereas other birds are more often blue. Then, wouldn’t this loss function incentivize the selection policy to pick the color concept for blue X birds?\n\n4. Similar to the above, while Section 4.3 studies the performance improvement from the policies, it is not clear what these policies pick. Do the authors have any analyses that explain what the chosen concepts are? For instance, are these the concepts that are specific to a class? Are these the most common concepts for the class, are there any other patterns there? \n\n5. (Minor) I would appreciate having the Tables in addition to the figures, I’m sorry if I missed them but I could not find them in the Appendix either. Numbers could be hard to read from the Tables.", "limitations": "I think the most significant limitation is the lack of an actual real-world study to verify the claims. In my understanding, it is not possible to infer this from the current results. I would appreciate hearing the authors' reasoning about this.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Optimizing a model for intervention awareness is an interesting and strong idea. Many works did indeed attempt interventions, whereas it is infrequent to optimize the model for this purpose and the intervenability has been taken for granted. I do have a question about whether the current form of this is useful (Q1), yet I still believe the idea is worth sharing with the community.\n\n2. I believe having an intervention policy has implications beyond interventions per se. By looking at the Rollout Loss, I could imagine the policy revealing locally important / less important concepts and performing selective inference. While this is a hypothesis that needs verification, this could be an interesting direction to think about.\n\n3. (Minor) I appreciate the clarity in the writing. Formalization is clear and easy to digest even though there are nuanced tricks to make the method work.", "weaknesses": "1. In my understanding, there is a disconnect between the promise/message of the paper and the experiments. Concretely;\n\n- 1.1 In many parts of the paper, there is a theme around getting the models ready for expert feedback, which can make sense. However, I do not fully understand how we can infer that the current method achieves something related to a user or an expert. \n\n- 1.2 In this regard, a major weakness is the lack of a user study. There are many arguments in the paper that touch upon optimizing the model for user interventions, whereas there are no experiments that verify this argument. An experiment where the users used these models, intervened on them, and IntCEM performed better than baselines would support the argument here.\n\n- 1.3 Furthermore, the selection policy encodes a certain belief: That the users will choose concepts that would improve downstream performance. An alternative user metric would be showing that users prefer the concepts chosen by the selection policy. This may or not may not be the case. One could have also hypothesized that the policy can pick concepts that the users would also like to intervene on more than other policies, but again, there is no supporting evidence for this argument either.\n\nIn general, I believe this is an important disconnect that needs addressing.\n\n2. The usability of the method suffers from the requirement to have concept annotations at training time, as do vanilla CBMs and CEMs. However, there are still concept-based models that could be worth exploring in this context that are already cited in the paper as [26, 27] which are Post-hoc CBMs and Label-Free CBMs that also scale to ImageNet/COCO. The authors make the argument that the method would complement those architectures as well, and I can believe this hypothesis, but this needs verification. \n\n3. The above weakness also leads to another one, which is the lack of experimental settings closer to the real-world such as ImageNet, COCO, or medical datasets that were tested before in the above papers. I do think experiments would benefit from expanding beyond CUB and CelebA. Personally, I believe this would also allow impactful user studies.\n\n4. It is hard for me to interpret the numbers in Sections 4.2 and 4.3. This is also connected to the 3rd point above – but the improvements in MNIST or CUB seem 1-2%. Further, as a minor comment, while there are Figures, all of the y-axes are in different scales, and I cannot really tell the numbers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 4KHx,\n\nThank you for your feedback and valuable questions. We appreciate that you found our work novel and clear.  Below we answer the questions you raised.\n\n### Message of the paper and experiment design\n\nWe believe there may be a misunderstanding of the methodology underlying our paper, particularly when seen in relation to prior work. IntCEMs are designed to better uptake user/expert feedback at test time (i.e., via “concept interventions”). However, we do not change how we think of concept interventions compared to prior work; intervention awareness is simply – yet powerfully – a training-time change that models that support concept interventions (in particular embedding-based models) can make use of to improve their receptiveness to interventions. We evaluate concept interventions in the same way as prior works in this area (e.g., [1-5]). The underlying assumption of our and their evaluation is that an expert can correctly answer a query of the form “What is the correct binary value of concept X?”; our evaluation is just a way to simulate such queries when we know the ground truth concepts. While there are important grounds for questioning these assumptions (see [5]) – in this first instantiation, we believe our experiments are substantial towards demonstrating the potential value of IntCEMs and follow the way the community has been evaluating concept interventions. \n\n\n### Selection policy \n\nWe emphasize that there is an important difference between what a policy may suggest and what the user may opt to intervene on. There is no assumption in our work dictating that a user must provide a concept label when requested by $\\psi$. In fact, in Section 4.2 we show that our model significantly outperforms competing state-of-the-art methods even in the case where a user randomly selects which concepts to intervene on. Nevertheless, we agree that one may want to encode a user’s preferences into the intervention policy. This is an exciting direction and one we are keen to explore next with user studies (we will mention these two in a new future work section as suggested by reviewer 23Ef).\n\nWe highlight, however, that in this work we operate under the reasonable and common assumption that expert queries can be asked and an expert will answer them correctly. This is no different to assumptions made in the active learning and active feature acquisition communities, both popular and mature fields of study.\n\n\n### Intervention policy and concept typicality \n\nNotice that our learnt policy is dynamic (i.e., it is a function of the current sample and the previous interventions) rather than static (i.e., concept intervention is fixed a priori). As such, IntCEM’s policy $\\psi$ does not pick the same sequence of concepts for all samples ( see Figure 1 in the rebuttal supplement for an example). Hence, it would not necessarily always pick the colour blue if the IntCEM is, say, very confident that the input sample is blue.\n\n### Extension to other kinds of concept models and datasets\n\nAs you correctly mentioned, the principles underlying IntCEMs could be applied to other models. We include details on possible extensions for traditional CBMs in our Appendix (see A.8) and discuss these results in our discussion. As for even more recent concept-based architectures such as post-hoc CBMs and label-free CBMs that are applicable to larger scale datasets lacking complete concept annotations (e.g., ImageNet and COCO), we learnt about them at ICLR 2023 only a couple of weeks before NeurIPS’ deadline. This near overlap with their publication and the NeurIPS paper submission deadline meant that we could not sensibly include them as part of our experiments and that these works fell within the two-month NeurIPS baseline exclusion period. Because of this, we focused on evaluating variants of CBMs and CEMs on datasets with a complete set of concept annotations across all samples. Nevertheless, we find these works very interesting and exciting, which is why we suggested as part of future work that one may explore applying IntCEM’s core design principles to these sorts of models. Further verification is sensible for future work, but we do not think this is essential for affirming the validity of our intervention-aware design. Finally, note that, unlike CBMs, IntCEM does not require a complete set of concept annotations because we its use of embeddings.\n\n\n### IntCEM improvements and presentation of results\n\nWe agree that the scales in our figures might make exact values hard to determine when analysing the gains obtained with IntCEMs. To address this, we have followed your suggestion and made a simplified table version of Figures 3 and 4 that we will include in a new appendix. Due to lack of space, we show only the table corresponding to Figure 3 (and only for some a fixed set of fractions of intervened concepts) in our attached supplement as Figure 1.\n\nLooking at this new table and Figure 3, we see a significant difference between IntCEMs and CEMs across all datasets once interventions are performed. For example, in CUB we observe a difference of 6.5% absolute improvement for IntCEM vs CEM when randomly intervening on 50% of the available concepts while in CelebA we observe a difference of ~13.20%. This table also shows that by intervening following IntCEM’s learnt policy $\\psi$, IntCEM can achieve nearly perfect accuracy (99.17%) after intervening on 50% of the concepts.  This boost comes with negligible test time computational costs and without the need to (re)learn any further policies.\n\n\n## References\n\n[1] Koh et al. \"Concept bottleneck models.\" ICML 2020.\n\n[2] Kazhdan et al. \"Now you see me (CME): concept-based model extraction.\" arXiv:2010.13233 (2020).\n\n[3] Espinosa Zarlenga et al. \"Concept embedding models\" NeurIPS 2022.\n\n[4] Chauhan et al. \"Interactive concept bottleneck models.\" AAAI 2023.\n\n[5] Collins et al. \"Human Uncertainty in Concept-Based AI Systems.\" arXiv e-prints (2023)."}]

Author rebuttal:
We thank the reviewers for their very insightful feedback and for taking the time to read our work carefully. Their feedback has certainly improved the quality of our manuscript. We hope to address your concerns in this rebuttal and its corresponding supplementary document. We reply to questions shared by two or more reviewers in this general rebuttal while replying to specific questions raised by reviewers in their respective rebuttals.

# Summary of Supplement

In the rebuttal’s supplement (see PDF attached below), we include the following figures and tables:

- Table 1 represents a (simplified) tabular version of Figure 3 to help better analyse each model’s performance when a fixed fraction of concepts are intervened on. This table will be added to the Appendix in our updated manuscript.

- Figure 1 depicts an application of IntCEM’s learnt policy to two CUB test samples. This figure will be added in the main body of our updated manuscript in experimental Section 4.3 where we discuss IntCEM’s learnt policy $\psi$.

- Table 2 shows the Oracle Impurity Score (a measurement of leakage described below) for our jointly-trained baselines across all datasets. This constitutes a **new experiment** introduced in this rebuttal, the purpose of which is to understand better how leakage works in IntCEM and the role it may play in concept interventions with embedding-based models. We will include this table in a new Appendix section with a brief summary of its main results included in our discussion section.

# Answers to common questions

### Concrete example of IntCEM’s policy at test-time (reviewers oMUi and 4KHx)

In Figure 1 of the supplement, we show two concrete examples of intervening on an IntCEM in CUB to help clarify how our model works and how following its policy $\psi$  can change its posterior label distribution in practice. These examples will be included in Section 4.3  where we discuss how $\psi$ performs.

### Leakage in IntCEM’s embeddings (reviewers Zgwp and 23Ef)

As mentioned by reviewers Zgwp and 23Ef (and also related to one of reviewer oMUi’s questions), leakage [1] is a known issue in traditional CBMs. In fact, it has been empirically shown by [2] that leakage may lead to detrimental interventions because a CBM’s downstream label predictor may learn to rely on information leaked from concept representations and this information is necessarily destroyed when performing interventions in CBMs. This is because to intervene in a CBM’s bottleneck, one must overwrite a neuron’s activation to one that is aligned with the ground truth value of the concept that is represented by that neuron (e.g., a bottleneck’s neuron may be set to $0$ if the concept it represents is known to be “off” for the current sample, removing all information that could have been encoded in the continuous value previously outputted by that neuron).

 This issue does not arise in CEMs and IntCEMs because **embedding-based concept interventions are not destructive**. Indeed, as described in Section 2 of our paper, when intervening on either of these models, one essentially performs a swap between embeddings, allowing the model to maintain any leaked information from the input/other concepts/task labels as part of the embedding of each concept. Therefore, we do not expect interventions to be significantly affected by leakage in concept representations as long as a concept’s distribution of positive ($\hat{\mathbf{c}}^{(+)}$) and negative ($\hat{\mathbf{c}}^{(-)}$) embeddings are easy to discriminate by the downstream label predictor.

As keenly pointed out by reviewer 23Ef, we hypothesize that higher leakage may contribute to IntCEMs’ better performance when being intervened on. To evaluate this hypothesis, we measure the Oracle Impurity Score (OIS) [2] of concept representations learnt by CBMs, CEMs, and IntCEMs across all tasks. This score, between 0 and 1, essentially measures how much extra information, on average, each learnt concept representation captures from other possibly unrelated concepts (with higher scores representing higher impurities and therefore more leakage between concepts). Our results, shown in Table 2 of our rebuttal’s supplement, show that (1) as one would expect, there is significantly more leakage in CEMs compared to CBMs and, (2) more importantly, IntCEM embeddings are also capturing more impurities than CEM’s embeddings across all datasets we tested except for $\texttt{CUB-Incomp}$, providing evidence towards our hypothesis. Given that this preliminary leakage study may open the door for potentially useful and interesting future research, we will include this experiment in a new appendix (with a brief discussion of it introduced in our paper’s discussion). Our updated discussion will highlight that contrary to common assumptions, leakage may not always be undesired and could be a healthy byproduct of more expressive concept representations in models that accommodate such expressivity. 

### User studies  (reviewers 23Ef and 4KHx)

We agree that a human user study could reveal very interesting results and complement our paper's extensive experiments and main contributions.  We have, however, left this for future work as we believe some exciting research directions spawning from a formal user study for concept interventions would merit a whole paper on their own. We are indeed very excited and optimistic about exploring these future avenues.

Following Reviewer 23Ef’s suggestion of including a future work section for our manuscript, we will update our paper to suggest user studies as part of future work. 

### References

[1] Mahinpei et al. "Promises and pitfalls of black-box concept learning models." Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI at ICML (2021).

[2] Espinosa Zarlenga  et al. "Towards robust metrics for concept representation evaluation." AAAI (2023).


Meta-review: This is a very well-written paper, with a significant novel contribution. The method is well explained and the experiments are well designed and demonstrate the effectiveness of the proposed model. One limitation is the the lack of an actual real-world study to verify the claims, which would be nice to see in follow-up work.
Decision: Accept (spotlight)