Title: Learning to Receive Help: Intervention-Aware Concept Embedding Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Explainable Artificial Intelligence

Concept Bottleneck Models

Concept-based Explainability

Interpretability

XAI

Concept Interventions
Abstract: Concept Bottleneck Models (CBMs) tackle the opacity of neural architectures by constructing and explaining their predictions using a set of high-level concepts. A special property of these models is that they permit concept interventions, wherein users can correct mispredicted concepts and thus improve the model's performance. Recent work, however, has shown that intervention efficacy can be highly dependent on the order in which concepts are intervened on and on the model's architecture and training hyperparameters. We argue that this is rooted in a CBM's lack of train-time incentives for the model to be appropriately receptive to concept interventions. To address this, we propose Intervention-aware Concept Embedding models (IntCEMs), a novel CBM-based architecture and training paradigm that improves a model's receptiveness to test-time interventions. Our model learns a concept intervention policy in an end-to-end fashion from where it can sample meaningful intervention trajectories at train-time. This conditions IntCEMs to effectively select and receive concept interventions when deployed at test-time. Our experiments show that IntCEMs significantly outperform state-of-the-art concept-interpretable models when provided with test-time concept interventions, demonstrating the effectiveness of our approach.

Learning to Receive Help:

Abstract

Concept Bottleneck Models (CBMs) tackle the opacity of neural architectures by constructing and explaining their predictions using a set of high-level concepts. A special property of these models is that they permit concept interventions , wherein users can correct mispredicted concepts and thus improve the model's performance. Recent work, however, has shown that intervention efficacy can be highly dependent on the order in which concepts are intervened on and on the model's architecture and training hyperparameters. We argue that this is rooted in a CBM's lack of train-time incentives for the model to be appropriately receptive to concept interventions. To address this, we propose Intervention-aware Concept Embedding models (IntCEMs), a novel CBM-based architecture and training paradigm that improves a model's receptiveness to test-time interventions. Our model learns a concept intervention policy in an end-to-end fashion from where it can sample meaningful intervention trajectories at train-time. This conditions IntCEMs to effectively select and receive concept interventions when deployed at test-time. Our experiments show that IntCEMs significantly outperform state-of-the-art concept-interpretable models when provided with test-time concept interventions, demonstrating the effectiveness of our approach.

1 Introduction

It is important to know how to ask for help, but also important to know how to receive help. Knowing how to react to feedback positively allows for bias correction [1] and efficient learning [2] while being instrumental for mass collaborations [3]. Nevertheless, although the uptake of feedback is ubiquitous in real-world decision-making, the same cannot be said about modern artificial intelligence (AI) systems, where deployment tends to be in isolation from experts whose feedback could be queried.

Progress in this aspect has recently come from Explainable AI, where interpretable Deep Neural Networks (DNNs) that can benefit from expert feedback at test-time have been proposed [4-7]. In particular, Concept Bottleneck Models (CBMs) [4], a family of interpretable DNNs, enable expert-model interactions by generating, as part of their inference process, explanations for their predictions using high-level 'concepts'. Such explanations allow human experts to better understand a CBM's prediction based on interpretable units of information (e.g., input is a 'cat' because it has 'paws' and 'whiskers') rather than low-level features (e.g., input pixels). This design enables humans to provide feedback to the CBM at test-time via concept interventions [4] (Figure 1), a process specific to CBM-like architectures in which an expert analyses a CBM's inferred concepts

Krishnamurthy (Dj) Dvijotham

Mateja Jamnik

Legs

...

Hot dog

Bun

Hot dog

Not Hot Dog

Legs

Sesame Seeds

...

Bun

Human Expert

Observe

Explanation

Intervene

Observe

Sample

and corrects mispredictions. Concept interventions allow a CBM to update its final prediction after incorporating new expert feedback, and have been shown to lead to significant performance boosts [4, 5, 8, 9]. Nevertheless, and in contrast to these results, recent works have found that concept interventions can potentially increase a CBM's test-time error depending on its choice of architecture [10] and its set of train-time annotations [5]. This discrepancy suggests that a CBM's receptiveness to interventions is neither guaranteed nor fully understood.

In this paper, we posit that the sensitivity of a CBM's receptiveness to interventions is an artefact of its training objective function. Specifically, we argue that CBMs lack an explicit incentive to be receptive to interventions as they are neither exposed to interventions at train-time nor optimised to perform well under interventions. To address this, we propose a novel intervention-aware objective function and architecture based on Concept Embedding Models (CEMs) [5], a generalisation of CBMs that represent concepts as high-dimensional embeddings. Our architecture, which we call Intervention-aware Concept Embedding Model (IntCEM), has two key distinctive features. First, it learns an intervention policy end-to-end which imposes a prior over which concepts an expert could be queried to intervene on next to maximise intervention receptiveness. Second, its loss function includes an explicit regulariser that penalises IntCEM for mispredicting the task label after a series of train-time interventions sampled from its learned policy. We highlight the following contributions:

We introduce IntCEM, the first concept-interpretable neural architecture that learns not only to explain its predictions using concepts but also learns an intervention policy dictating which concepts an expert should intervene on next to maximise test performance.

We show that IntCEM significantly outperforms all baselines in the presence of interventions while maintaining competitiveness when deployed without any interventions.

We demonstrate that, by rewriting concept interventions as a differentiable operation, IntCEM learns an efficient dynamic intervention policy that selects performance-boosting concepts to intervene on at test-time.

2 Background and Previous Work

Concept Bottleneck Models Methods in Concept-based Explainable AI [11-17] advocate for explaining DNNs by constructing explanations for their predictions using high-level concepts captured within their latent space. Within these methods, Concept Bottleneck Models (CBMs) [4] provide a framework for developing concept-based interpretable DNNs.

Given a training set D := { ( x ( i ) , c ( i ) , y ( i ) ) } N i =1 , where each sample x ∈ R n (e.g., an image) is annotated with a task label y ∈ { 1 , · · · , L } (e.g., 'cat' or 'dog') and k binary concepts c ∈ { 0 , 1 } k (e.g., 'has whiskers'), a CBM is a pair of functions ( g, f ) whose composition f ( g ( x )) predicts a probability over output classes given x . The first function g : R n → [0 , 1] k , called the concept encoder , learns a mapping between features x and concept activations ˆ c = g ( x ) ∈ [0 , 1] k , where ideally ˆ c i is close to 1 when concept c i is active in x , and 0 otherwise. The second function f : [0 , 1] k → [0 , 1] L , called the label predictor , learns a mapping from the predicted concepts ˆ c to a distribution ˆ y = f (ˆ c ) over L task classes. When implemented as DNNs, g 's and f 's parameters can be learnt (i) jointly , by minimising a combination of the concept predictive loss L concept and the task cross-entropy loss L task , (ii) sequentially , by first training g to minimise L concept and then training

f to minimise L task from g 's outputs, or (iii) independently , where g and f are independently trained to minimise their respective losses. Because at prediction-time f has access only to the 'bottleneck' of concepts activations ˆ c , the composition ( f ◦ g ) yields a concept-interpretable model where ˆ c i can be interpreted as the probability ˆ p i that concept c i is active in x .

Concept Embedding Models Because f operates only on the concept scores predicted by g , if the set of training concepts is not predictive of the downstream task, then a CBM will be forced to choose between being highly accurate at predicting concepts or at predicting task labels. Concept Embedding Models (CEMs) [5] are a generalisation of CBMs that address this trade-off using high-dimensional concept representations in their bottlenecks. This design allows information of concepts not provided during training to flow into the label predictor via two m -dimensional vector representations (i.e., embeddings) for each training concept c i : c + i ∈ R m , an embedding representing c i when it is active (i.e., c i = 1), and c -i ∈ R m , an embedding representing c i when it is inactive (i.e., c i = 0).

When processing sample x , for each training concept c i a CEM constructs c + i and c -i by feeding x into two learnable models ϕ + i , ϕ -i : R n → R m implemented as DNNs with a shared preprocessing module. These embeddings are then passed to a learnable scoring model s : R 2 m → [0 , 1] , shared across all concepts, that predicts the probability ˆ p i = s ([ˆ c + i , ˆ c -i ]) of concept c i being active. With these probabilities, one can define a CEM's concept encoder g ( x ) = ˆ c := [ˆ c 1 , · · · , ˆ c k ] ∈ R km by mixing the positive and negative embeddings of each concept to generate a final concept embedding ˆ c i := ˆ p i ˆ c + i +(1 -ˆ p i )ˆ c -i . Finally, a CEM can predict a label ˆ y for sample x by passing ˆ c (i.e., its 'bottleneck') to a learnable label predictor f (ˆ c ) whose output can be explained with the concept probabilities ˆ p := [ˆ p 1 , · · · , ˆ p k ] T . This generalisation of CBMs has been empirically shown to outperform CBMs especially when the set of concept annotations is incomplete [5].

Concept Interventions A key property of both CBMs and CEMs is that they allow concept interventions , whereby experts can correct mispredicted concept probabilities ˆ p at test-time. By updating the predicted probability of concept c i (i.e., ˆ p i ) so that it matches the ground truth concept label (i.e., setting ˆ p i := c i ), these models can update their predicted bottleneck ˆ c and propagate that change into their label predictor g (ˆ x ) , leading to a potential update in the model's output prediction.

Let µ ∈ { 0 , 1 } k be a mask with µ i = 1 if we will intervene on concept c i , and µ i = 0 otherwise. Here, assume that we are given the ground truth values ˜ c of all the concepts we will intervene on. For notational simplicity, and without loss of generality, we can extend ˜ c to be in [0 , 1] k by setting ˜ c i = c i if µ i = 1 , and ˜ c i = 0 . 5 otherwise (where 0 . 5 expresses full uncertainty but, as it will become clear next, the specific value is of no importance). Thus, this vector contains the expert-provided ground truth concept labels for concepts we are intervening on, while it assigns an arbitrary value (e.g., 0.5) to all other concepts. We define an intervention as a process where, for all concepts c i , the activation(s) ˆ c i corresponding to concept c i in a bottleneck ˆ c are updated as follows:

where ˆ c + i and ˆ c -i are c i 's positive and negative concept embeddings (for CBMs ˆ c + i = [1] and ˆ c -i = [0] ). This update forces the mixing coefficient between the positive and negative concept embeddings to be the ground truth concept value c i when we intervene on concept c i (i.e., µ i = 1 ), while maintaining it as ˆ p i if we are not intervening on concept c i . The latter fact holds as ( µ i ˜ c i +(1 -µ i )ˆ p i ) = ˆ p i when µ i = 0 , regardless of ˜ c i . Because the predicted positive and negative concept embeddings, as well as the predicted concept probabilities, are all functions of x , for simplicity we use ˜ g ( x , µ, ˜ c ) to represent the updated bottleneck of CBM ( g, f ) when intervening with mask µ and concept values ˜ c .

We note that although we call the operation defined in Equation 1 a 'concept intervention' to follow the term used in the concept learning literature, it should be distinguished from interventions in causal models [18-20]. Instead, concept interventions in this context are connected, yet not identical, to previous work in active feature acquisition in which one has access to an expert at test-time to request the ground truth value of a small number of input features [21-23].

Intervention Policies In practice, it may not be sensible to ask a human to intervene on every concept [24]; instead, we may want a preferential ordering of which concepts to query first (e.g., some concepts may be highly informative of the output task, yet hard for the model to predict). 'Intervention policies' have been developed to produce a sensible ordering for concept interventions [8, 25, 9].

These orderings can be produced independently of a particular instance (i.e., a static policy), or selected on a per-instance basis (a dynamic policy). The latter takes as an input a mask of previously intervened concepts µ and a set of predicted concept probabilities ˆ p , and determines which concept should be requested next from an expert, where the retrieved concept label is assumed to be 'ground truth'. Two such policies are Cooperative Prediction (CooP) [9] and Expected Change in Target Prediction (ECTP) [8], which prioritise concept c i if intervening on c i next leads to the largest change in the probability of the currently predicted class in expectation (with the expectation taken over the distribution given by p ( c i = 1) = ˆ p i ). These policies, however, may become costly as the number of concepts increases and share the limitation that, by maximising the expected change in probability in the predicted class, they do not guarantee that the probability mass shifts towards the correct class.

3 Intervention-Aware Concept Embedding Models

Although there has been significant work on understanding how CBMs react to interventions [5, 6, 9, 25], the fact that these models are trained without any expert intervention - yet are expected to be highly receptive to expert feedback at test-time - is often overlooked. To the best of our knowledge, the only exception comes from Espinosa Zarlenga et al. [5], wherein the authors introduced RandInt , a procedure that randomly intervenes on the model during train-time. However, RandInt assumes that a model should be equally penalised for mispredicting y regardless of how many interventions have been performed. This leads to CEMs lacking the incentive to perform better when given more feedback. Moreover, RandInt assumes that all concepts are equally likely to be intervened on. This inadequately predisposes a model to assume that concepts will be randomly intervened on at test-time.

To address these limitations, we propose Intervention-Aware Concept Embedding Models (IntCEMs), a new CEM-based architecture and training framework designed for inducing receptiveness to interventions . IntCEMs are composed of two core components: 1) an end-to-end learnable concept intervention policy ψ (ˆ c , µ ) , and 2) a novel objective function that penalises IntCEM if it mispredicts its output label after following an intervention trajectory sampled from ψ (with a heavier penalty if the trajectory is longer). By learning the intervention policy ψ in conjunction with the concept encoder and label predictor, IntCEM can simulate dynamic intervention steps at train-time while it learns a policy which enables it to 'ask' for help about specific concepts at test-time.

3.1 Architecture Description and Inference Procedure

An IntCEM is a tuple of parametric functions ( g, f, ψ ) where: (i) g is a concept encoder, mapping features x to a bottleneck ˆ c , (ii) f is a label predictor mapping ˆ c to a distribution over L classes, and (iii) ψ : R km × { 0 , 1 } k → [0 , 1] k is a concept intervention policy mapping ˆ c and a mask of previously intervened concepts µ to a probability distribution over which concepts to intervene on next. IntCEM's concept encoder g works by (1) passing an input x to a learnable 'backbone' ζ ( x ) := ( { (ˆ c + i , ˆ c -i ) } k i =1 , ˆ p ) , which predicts a pair of embeddings (ˆ c + i , ˆ c -i ) and a concept probability ˆ p i for all concepts, and (2) constructs a bottleneck ˆ c by concatenating all the mixed vectors { ˆ p 1 ˆ c + 1 + (1 -ˆ p 1 )ˆ c -1 , · · · , ˆ p k ˆ c + k + (1 -ˆ p k )ˆ c -k } . While multiple instantiations are possible for ζ , in this work we parameterise it using the same backbone as in a CEM. This implies that ζ is formed by concept embedding generating models { ( ϕ + i , ϕ -i ) } k i =1 and scoring model s . This yields a model which, at inference, is almost identical to a CEM, except that IntCEM also outputs a probability distribution ψ (ˆ c , µ ) placing a high density on concepts that may yield significant intervention boosts. At train-time, these models differ significantly, as we discuss next.

3.2 IntCEM's Training Procedure

The crux of IntCEM's training procedure, shown in Figure 2, lies in inducing receptiveness to test-time interventions by exposing our model to dynamic train-time interventions. Specifically, at the beginning of each training step, we sample an initial mask of intervened concepts µ (0) from a prior distribution p ( µ ) as well as the number T ∈ { 1 , · · · , k } of interventions to perform on top of µ (0) from prior T ∼ p ( T ) . Next, we generate a trajectory of T new interventions { η ( t ) ∈ { 0 , 1 } k ∼ p ( η ; ω ( t ) ) } T t =1 by sampling the t th intervention η ( t ) ∼ p ( η ; ω ( t ) ) (represented as a one-hot encoding) from a categorical distribution with parameters ω ( t ) . A key component of this work is that parameters ω ( t ) are predicted using IntCEM's policy ψ ( ˆ c ( t -1) , µ ( t -1) ) evaluated on the

CEM Backbone

Input

Intervention Trajectory with

1

0

1

0

0

Gumbel-Softmax

1

2

3

Legend

Trainable Model

Loss Computation

Sampling Operation

Intervention

previous bottleneck ˆ c ( t -1) := ˜ g ( x , µ ( t -1) , c ) and intervention mask µ ( t -1) , where we recursively define µ ( t ) as µ ( t ) := µ ( t -1) + η ( t ) for t > 0 . This allows IntCEM to calibrate its train-time trajectory of interventions to reinforce supervision on the embeddings corresponding to currently mispredicted concepts, leading to IntCEM learning more robust representations of inherently harder concepts.

We design our objective function such that it incentivises learning performance-boosting intervention trajectories and achieving high task performance as more interventions are provided:

where T := { ( µ ( t -1) , η ( t ) ) } T t =1 is the intervention trajectory while λ roll and λ concept are user-defined hyperparameters. λ roll encodes a user's tradeoff for number of interventions versus task performance gains (e.g., high λ roll encourages rapid task performance gains from few interventions, whereas low λ roll is more suitable in settings where many expert queries can be made). λ concept expresses a user's preference between accurate explanations and accurate task predictions before any interventions.

Rollout Loss ( L roll ) The purpose of L roll is to learn an intervention policy that prioritises the selection of concepts for which the associated concept embeddings fails to represent the ground truth concept labels, i.e., select concepts which benefit from human intervention. We achieve this through a train-time 'behavioural cloning' [26, 27] approach where ψ is trained to learn to predict the action taken by an optimal greedy policy at each step. For this, we take advantage of the presence of ground truth concepts and task labels during training and incentivise ψ to mimic 'Skyline', an oracle optimal policy proposed by Chauhan et al. [9]. Given a bottleneck ˆ c and ground-truth labels ( y, c ) , the Skyline policy selects c ∗ ( x , µ, c , y ) := arg max 1 ≤ i ≤ k f ( ˜ g ( x , µ ∨ 1 i , c ) ) y as its next concept, where f ( · ) y is the probability of class y predicted by f and µ ∨ 1 i represents the action of adding an intervention on concept i to µ . Intuitively, c ∗ corresponds to the concept whose ground-truth label and intervention would yield the highest probability over the ground-truth class y . We use such a demonstration to provide feedback to ψ through the following loss:

where CE ( p , ˆ p ) is the cross-entropy loss between ground truth distribution p and predicted distribution ˆ p , penalising ψ for not selecting the same concept as Skyline throughout its trajectory.

Task Prediction Loss ( L pred ) We penalise IntCEM for mispredicting y , both before and after our intervention trajectory, by imposing a higher penalty when it mispredicts y at the end of the trajectory:

Here γ ∈ [1 , ∞ ) is a scaling factor that penalises IntCEM more heavily for mispredicting y at the end of the trajectory, by a factor of γ T > 1 , than for mispredicting y at the start of the trajectory. In practice, we observe that a value in γ ∈ [1 . 1 , 1 . 5] works well; we use γ = 1 . 1 in our experiments unless specified otherwise. For an ablation study showing that IntCEM's intervention performance surpasses that of existing methods for a wide array of scaling factors γ , see Appendix A.6.

A key realisation is that expressing interventions as in Equation (1) yields an operator which is differentiable with respect to the concept intervention mask µ . This trick allows us to backpropagate gradients from L pred into ψ when the sampling operation η ( t ) ∼ p ( η | ψ (ˆ c ( t -1) , µ ( t -1) )) is differentiable with respect to its parameters. In this work, this is achieved by relaxing our trajectory sampling using a differentiable Gumbel-Softmax [28] categorical sampler (see Appendix A.2 for details).

Concept Loss ( L concept ) The last term in our loss incentivises accurate concept explanations for IntCEM's predictions using the cross-entropy loss averaged across all training concepts:

Putting everything together ( L ) We learn IntCEM's parameters θ by minimising the following:

The outer expectation can be optimised via stochastic gradient descent, while the inner expectation can be estimated using Monte Carlo samples from the user-selected priors ( p ( µ ) , p ( T )) . For the sake of simplicity, we estimate the inner expectation using a single Monte Carlo sample per training step for both the initial mask µ (0) and the trajectory horizon T . In our experiments, we opt to use a Bernoulli prior for p ( µ ) , where each concept is selected with probability p int = 0 . 25 , and a discrete uniform prior Unif ( { 1 , · · · , T max } ) for p ( T ) , where T max is annealed within [2 , 6] . Although domain-specific knowledge can be incorporated into these priors, we leave this for future work and show in Appendix A.6.3 that IntCEMs are receptive to interventions as we vary p int and T max.

4 Experiments

In this section, we evaluate IntCEMs by exploring the following research questions:

Unintervened Performance (Q1) : In the absence of interventions, how does IntCEM's task and concept predictive performance compare to CEMs's and CBM's? Does IntCEM's updated loss function have detrimental effects on its uninterverned performance?

Intervention Performance (Q2A) : Are IntCEMs more receptive to test-time interventions than state-of-the-art CBM variants?

Effects of Policies during Deployment (Q2B) : What is the impact on IntCEM's performance when employing different intervention policies at test-time?

Benefits of End-to-End Learning of ψ (Q3) : Is it beneficial to learn our intervention policy ψ at train-time?

Datasets and Tasks We consider five vision tasks: (1) MNIST-Add , a task inspired by the UMNIST dataset [29] where one is provided with 12 MNIST [30] images containing digits in { 0 , · · · , 9 } , as well as their digit labels as concept annotations, and needs to predict if their sum is at least half of the maximum attainable, (2) MNIST-Add-Incomp , a concept-incomplete [17] extension of the MNIST-Add task where only 8/12 operands are provided as concept annotations, (3) CUB [31], a bird classification task whose set of concepts cover 112 bird features, e.g., 'wing shape', selected by Koh et al. [4], (4) CUB-Incomp , a concept-incomplete extension of CUB where we provide only 25% of all concept groups, and (5) CelebA , a task on the Celebrity Attributes Dataset [32] whose task and concept labels follow those selected in [5]. For more details refer to Appendix A.3.

Models We evaluate IntCEMs against CEMs with the exact same architecture, averaging all metrics of interest over five different random initialisations. We also include CBMs that are trained jointly ( Joint CBM ), sequentially ( Seq. CBM ) and independently ( Ind. CBM ) as part of our evaluation to

be able to compare IntCEM's policy against state-of-art intervention policies developed for CBMs. Since previous work shows that interventions in joint CBMs are sensitive to its bottleneck activation function [4, 10], we include joint CBMs with a sigmoidal ( Joint CBM-Sigmoid ) and logit-based ( Joint CBM-Logit ) bottleneck. We note that we do not include other noteworthy concept-based models such as leakage-free CBMs [6], GlanceNets [33], Self-explaining Neural Networks (SENNs) [34], Concept Whitening (CW) [35], or Concept Model Extraction (CME) [16] as these either lack a well-defined mechanism to be intervened on (e.g., as in SENNs and CW) or represent more constrained versions of other baselines such as CEMs (e.g., leakage-free CBMs, GlanceNets, and CME).

Throughout our experiments, we strive for a fair comparison by using the same architectures for the concept encoder g and label predictor f of all baselines. Unless specified otherwise, for each task, we tune IntCEM's λ roll hyperparameter by varying it over { 5 , 1 , 0 . 1 } and selecting the model with the highest area under the curve defined by plotting the validation task accuracy vs number of interventions. For the CUB -based tasks and CelebA , we use the same values of λ concept selected in [5]. Otherwise, we select this value from { 10 , 5 , 1 } using CEM's validation error. Following [5], we use embeddings with m = 16 activations and a value of p int = 0 . 25 for CEM's RandInt. For further details on model and training hyperparameters, see Appendix A.4.

4.1 Task and Concept Performance in the Absence of Interventions (Q1)

First, we examine how IntCEM's loss function affects its unintervened test performance compared to existing baselines as we vary λ roll. We summarise our results in Table 1, where, for all tasks and baselines, we show the downstream task predictive test accuracy, or the AUC when the task is binary (i.e., MNIST-based tasks), and the concept performance. We measure the latter via the mean AUC of predicting ground-truth test concept c i from its corresponding predicted concept probability ˆ p i .

In the absence of concept interventions, IntCEM is as accurate as state-of-the-art baselines. Evaluating test task accuracy reveals that IntCEMs are more accurate (e.g., by ∼ 7% in CelebA) or just as accurate as other baselines across most tasks. Only in CUB do we observe a small drop in performance (less than ∼ 1% ) for the best-performing IntCEM variant. Notice that three of the five tasks we use, namely MNIST-Add-Incomp , CUB-Incomp , and CelebA , lack a complete set of training concept annotations. Therefore, our results suggest that IntCEM maintains a high task performance even with concept incompleteness - a highly desired property for real-world deployment. Finally, the performance improvements of IntCEMs over CEMs in MNIST , MNIST-Incomp , and CelebA suggest that our loss function may lead to more informative concept representations that enable more accurate label predictors to be trained.

IntCEM's concept performance is competitive with respect to traditional CEMs. The bottom half of Table 1 shows that IntCEM's mean concept AUC, particularly for low values of λ roll , is competitive against that of CEMs. The only exception is in the MNIST-based task, where Joint CBMs outperform CEMs and IntCEMs. Nevertheless, this drop is overshadowed by the poor task accuracy obtained by joint CBMs in these two tasks, highlighting that although CBMs may have better concept predictive accuracy than CEMs, their task performance is generally bounded by that of a CEM. These results demonstrate that IntCEM's explanations are as accurate as those in CEMs without sacrificing task accuracy, suggesting our proposed objective loss does not significantly affect IntCEM's performance in the absence of interventions. Finally, we observe a trade-off between incentivising the IntCEM to be highly accurate after a small number of interventions (i.e., high λ roll )

0

4

8

12

Groups Intervened

80

90

100

Task Accuracy (%)

MNIST-Add

0

2

4

6

8

Groups Intervened

80

90

MNIST-Add-Incomp

0

4

8

12 16 20 24 28

Groups Intervened

60

80

100

CUB

0

2

4

6

Groups Intervened

40

60

80

CUB-Incomp

0

2

4

6

Groups Intervened

20

40

60

CelebA

IntCEM

CEM

Joint CBM-Logit

Independent CBM

and incentivising the IntCEM to generate more accurate concept explanations (i.e., low λ roll), which can be calibrated by setting λ roll after considering how the IntCEM will be deployed.

4.2 Intervention Performance (Q2)

Given that a core appeal of CBM-like models is their receptiveness to interventions, we now explore how test-time interventions affect IntCEM's task performance compared to other models. We first study whether IntCEM's training procedure preconditions it to be more receptive to test-time interventions by evaluating the task accuracy of all baselines after receiving random concept interventions. Then, we investigate whether IntCEM's competitive advantage holds when exposing our baselines to state-of-the-art intervention policies. Below, we report our findings by showing each model's task accuracy as we intervene, following [4], on an increasing number of groups of mutually exclusive concepts (e.g., 'white wing' and 'black wing').

IntCEM is significantly more receptive to test-time interventions. Figure 3, shows that IntCEM's task accuracy after receiving concept interventions is significantly better than that of competing methods across all tasks, illuminating the benefits of IntCEM preconditioning for receptiveness. We observe this for both concept-complete tasks (i.e., CUB and MNIST-Add ) as well as conceptincomplete tasks (i.e., rest). In particular, we see significant gains in CUB and CelebA ; IntCEMs attain large performance improvements with only a handful of random interventions ( ∼ 10% with ∼ 25% of concepts intervened) and surpass all baselines.

IntCEM's performance when intervened on with a randomly selected set of concepts can be better than the theoretically-proven optimal intervention performance achievable by CEMs alone. We explore IntCEM's receptiveness to different test-time intervention policies by computing its test performance while intervening following: (1) a Random intervention policy, where the next concept is selected, uniformly at random, from the set of unknown concepts, (2) the Uncertainty of Concept Prediction (UCP) [8] policy, where the next concept is selected by choosing the concept c i whose predicted probability ˆ p i has the highest uncertainty (measured by 1 / | ˆ p i -0 . 5 | ), (3) the Cooperative Policy (CooP) [9], where we select the concept that maximises a linear combination of its predicted uncertainty (akin to UCP) and the expected change in the predicted label's probability when intervening on that concept, (4) Concept Validation Accuracy (CVA) static policy, where concepts are selected using a fixed order starting with those whose validation errors are the highest as done by Koh et al. [4], (5) the Concept Validation Improvement (CVI) [9] static policy, where concepts are selected using a fixed order starting with those concepts that, on average, lead to the biggest improvements in validation accuracy when intervened on, and finally (6) an oracle Skyline policy, which selects c ∗ on every step, indicating an upper bound for all greedy policies. For details on each policy baseline, including how their hyperparameters are selected, see Appendix A.7.

We demonstrate in Figure 4 that IntCEM consistently outperforms CEMs regardless of the test-time intervention policy used, with CooP and UCP consistently yielding superior results. Further, we uncover that intervening on IntCEM with a Random policy at test-time outperforms the theoretical best performance attainable by a CEM ( Skyline ) under many interventions, while its best-performing policy outperforms CEM's Skyline after very few interventions. These results suggest that IntCEM's train-time conditioning not only results in IntCEMs being more receptive to test-time interventions

CUB

0

4

8

12 16 20 24 28

Groups Intervened

74

79

84

89

94

99

Task Accuracy (%)

IntCEM

0

4

8

12 16 20 24 28

Groups Intervened

74

79

84

89

94

99

CEM

CelebA

0

2

4

6

Groups Intervened

29

39

49

59

69

IntCEM

0

2

4

6

Groups Intervened

29

39

49

59

69

CEM

Skyline (Oracle)

Random

UCP

CooP

CVA

CVI

0

4

8

12

Groups Intervened

92.5

95.0

97.5

Task Accuracy (%)

MNIST-Add

0

2

4

6

8

Groups Intervened

90

92

94

MNIST-Add-Incomp

0

4

8 1216202428

Groups Intervened

80

90

100

CUB

0

2

4

6

Groups Intervened

80

90

CUB-Incomp

0

2

4

6

Groups Intervened

40

60

CelebA

IntCEM (Random Policy)

IntCEM (CooP Policy)

IntCEM (BC-Skyline Policy)

IntCEM (Learnt Policy

)

IntCEM no

(Random Policy)

but may lead to significant changes in an IntCEM's theoretical-optimal intervention performance. We include analyses on our remaining datasets, revealing similar trends, in Appendix A.7.

4.3 Studying IntCEM's Intervention Policy (Q3)

Finally, we explore IntCEM's intervention policy by evaluating test-time interventions predicted by ψ . To understand whether there is a benefit of learning ψ end-to-end (i.e., in conjunction with the rest of model training), rather than learning it post-hoc after training an IntCEM, we compare it against a Behavioural Cloning (BC) [26] policy ' BC-Skyline ' trained on demonstrations of Skyline applied to a trained IntCEM. Furthermore, to explore how ψ 's trajectories at train-time help improve IntCEM's receptiveness to test-time interventions, we study an IntCEM trained by sampling trajectories uniformly at random ('IntCEM no ψ '). Our results in Figure 5 suggest that: (1) IntCEM's learnt policy ψ leads to test-time interventions that are as good or better than CooP's, yet avoid CooP's computational cost ( up to ∼ 2x faster as seen in Appendix A.5), (2) learning ψ end-to-end during training yields a better policy than one learnt through BC after the IntCEM is trained, as seen in the BC policy's lower performance; and (3) using a learnable intervention policy at train-time results in a significant boost of test-time performance. This suggests that part of IntCEM's receptiveness to interventions, even when using a random test-time policy, can be partially attributed to learning and using ψ during training. We show some qualitative examples of rollouts of ψ in the CUB dataset in Figure 6.

5 Discussion and Conclusion

Concept Leakage Within IntCEMs Previous work [36, 6, 10] has shown that CBMs are prone to encoding unnecessary information in their learnt concept representations. This phenomenon, called concept leakage , may lead to less interpretable concept representations [36] and detrimental concept interventions in CBMs [10]. Given that interventions in IntCEMs involve swapping a concept's predicted embeddings rather than overwriting an activation in the bottleneck, such interventions are distinct from those in CBMs as they enable leaked information to be exploited after an intervention

True Label

Predicted

: Least Flycatcher

Least Flycatcher

Ovenbird

Blue headed Vireo

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

p

(ˆ

y

|

ˆ

c

)

Before Interventions

Least Flycatcher

Ovenbird

Blue headed Vireo

After Intervention #1:

breast pattern

Least Flycatcher

Ovenbird

Blue headed Vireo

After Intervention #2:

leg color

Least Flycatcher

Ovenbird

Blue headed Vireo

After Intervention #3:

wing shape

Least Flycatcher

Ovenbird

Blue headed Vireo

After Intervention #4:

forehead color

Least Flycatcher

Ovenbird

Blue headed Vireo

After Intervention #5:

upperparts color

True Label

Predicted

: American Crow

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

0

.

0

0

.

2

0

.

4

0

.

6

0

.

8

1

.

0

p

(ˆ

y

|

ˆ

c

)

Before Interventions

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

After Intervention #1:

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

After Intervention #2:

wing shape

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

After Intervention #3:

breast pattern

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

After Intervention #4:

size

Pelagic Cormorant

American Crow

White necked Raven

Black Tern

After Intervention #5:

leg color

is performed. This means our loss function may incentivise IntCEM to exploit this mechanism to improve a model's receptiveness to interventions. In Appendix A.10 we explore this hypothesis and find that we can detect more leakage in IntCEM's concept representations than in those learnt by CBMs and CEMs. This suggests that, contrary to common assumptions, leakage may be a healthy byproduct of more expressive concept representations in models that accommodate such expressivity. Nevertheless, we believe further work is needed to understand the consequences of this leakage.

Limitations and Future Work To the best of our knowledge, we are the first to frame learning to receive concept interventions as a joint optimisation problem where we simultaneously learn concepts, downstream labels, and policies. IntCEMs offer a principled way to prepare concept-based models for interventions at test-time, without sacrificing performance in the absence of interventions. Here, we focused on CEMs as our base model. However, our method can be extended to traditional CBMs (see Appendix A.8), and future work may consider extending it to more recent concept-based models such as post-hoc CBMs [37], label-free CBMs [38], and probabilistic CBMs [39].

Furthermore, we note some limitations. First, IntCEM requires hyperparameter tuning for λ roll and λ concept. While our ablations in Appendix A.6 suggest that an IntCEM's performance gains extend across multiple values of λ roll, users still need to tune such parameters for maximising IntCEM's utility. These challenges are exacerbated by the computational costs of training IntCEMs due to their train-time trajectory sampling (see Appendix A.5). However, such overhead gets amortised over time given ψ 's efficiency over competing policies. Further, we recognise our training procedure renders IntCEMs more receptive to any form of intervention, including adversarial interactions (see Appendix A.9). This raises a potential societal concern when deciding how our proposed model is deployed. Future work may explore such issues by incorporating error-correcting mechanisms or by considering intervention-time human uncertainty [29].

Finally, our evaluation was limited to two real-world datasets and one synthetic dataset, all of which have medium-to-small training set sizes. Therefore, future work may explore how to apply our proposed architecture to larger real-world datasets and may explore how to best deploy IntCEM's policy in practice via large user studies.

Conclusion A core feature of concept-based models is that they permit experts to interpret predictions in terms of high-level concepts and intervene on mispredicted concepts hoping to improve task performance. Counter-intuitively, such models are rarely trained with interventions in mind. In this work, we introduce a novel concept-interpretable architecture and training paradigm - InterventionAware Concept Embedding Models (IntCEMs) - designed explicitly for intervention receptiveness. IntCEMs simulate interventions during train-time, continually preparing the model for interventions they may receive when deployed, while learning an efficient intervention policy. Given the cost of querying experts for help, our work addresses the lack of a mechanism to leverage help when received, and demonstrates the value of studying models which, by design, know how to utilise help.

Acknowledgments and Disclosure of Funding

The authors would like to thank Naveen Raman, Andrei Margeloiu, and the NeurIPS 2023 reviewers for their insightful and thorough comments on earlier versions of this manuscript. MEZ acknowledges support from the Gates Cambridge Trust via a Gates Cambridge Scholarship. KMC acknowledges support from the Marshall Commission and the Cambridge Trust. AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1, The Alan Turing Institute, and the Leverhulme Trust via CFI. GM is funded by the Research Foundation-Flanders (FWO-Vlaanderen, GA No 1239422N). MJ is supported by the EPSRC grant EP/T019603/1.

References

Luke Gusukuma, Austin Cory Bart, Dennis Kafura, and Jeremy Ernst. Misconception-driven feedback: Results from an experimental study. In Proceedings of the 2018 ACM Conference on International Computing Education Research , pages 160-168, 2018.

Noreen M Webb. Group collaboration in assessment: Multiple objectives, processes, and outcomes. Educational Evaluation and Policy Analysis , 17(2):239-261, 1995.

AnHai Doan, Raghu Ramakrishnan, and Alon Y Halevy. Mass collaboration systems on the world-wide web. Communications of the ACM , 54(4):86-96, 2010.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning , pages 5338-5348. PMLR, 2020.

Mateo Espinosa Zarlenga, Barbiero Pietro, Ciravegna Gabriele, Marra Giuseppe, Francesco Giannini, Michelangelo Diligenti, Shams Zohreh, Precioso Frederic, Stefano Melacci, Weller Adrian, et al. Concept embedding models: Beyond the accuracy-explainability trade-off. In Advances in Neural Information Processing Systems , volume 35, pages 21400-21413. Curran Associates, Inc., 2022.

Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems , 2022.

Lucie Charlotte Magister, Pietro Barbiero, Dmitry Kazhdan, Federico Siciliano, Gabriele Ciravegna, Fabrizio Silvestri, Mateja Jamnik, and Pietro Lio. Encoding concepts in graph neural networks. arXiv preprint arXiv:2207.13586 , 2022.

Sungbin Shin, Yohan Jo, Sungsoo Ahn, and Namhoon Lee. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302.14260 , 2023.

Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. arXiv preprint arXiv:2212.07430 , 2022.

Mateo Espinosa Zarlenga, Pietro Barbiero, Zohreh Shams, Dmitry Kazhdan, Umang Bhatt, Adrian Weller, and Mateja Jamnik. Towards robust metrics for concept representation evaluation. arXiv e-prints , pages arXiv-2301, 2023.

David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6541-6549, 2017.

Ruth Fong and Andrea Vedaldi. Net2vec: Quantifying and explaining how concepts are encoded by filters in deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8730-8738, 2018.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability Beyond Feature Attribution: Quantitative Testing With Concept Activation Vectors (TCAV). In International conference on machine learning , pages 2668-2677. PMLR, 2018.

Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (CaCE). arXiv preprint arXiv:1907.07165 , 2019.

Amirata Ghorbani, James Wexler, James Zou, and Been Kim. Towards automatic concept-based explanations. arXiv preprint arXiv:1902.03129 , 2019.

Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Liò, and Adrian Weller. Now you see me (cme): concept-based model extraction. arXiv preprint arXiv:2010.13233 , 2020.

Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. On completeness-aware concept-based explanations in deep neural networks. Advances in Neural Information Processing Systems , 33:20554-20565, 2020.

Judea Pearl et al. Models, reasoning and inference. Cambridge, UK: CambridgeUniversityPress , 19(2):3, 2000.

Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. Explaining classifiers with causal concept effect (cace). arXiv preprint arXiv:1907.07165 , 2019.

Matthew O'Shaughnessy, Gregory Canal, Marissa Connor, Christopher Rozell, and Mark Davenport. Generative causal explanations of black-box classifiers. Advances in neural information processing systems , 33:5453-5467, 2020.

Hajin Shim, Sung Ju Hwang, and Eunho Yang. Joint active feature acquisition and classification with variable-size set encoding. Advances in neural information processing systems , 31, 2018.

Yang Li and Junier Oliva. Active feature acquisition with generative surrogate models. In International Conference on Machine Learning , pages 6450-6459. PMLR, 2021.

Ryan Strauss and Junier B Oliva. Posterior matching for arbitrary conditioning. Advances in Neural Information Processing Systems , 35:18088-18099, 2022.

Matthew Barker, Katherine M Collins, Krishnamurthy Dvijotham, Adrian Weller, and Umang Bhatt. Selective concept models: Permitting stakeholder customisation at test-time. AAAI HCOMP , 2023.

Ivaxi Sheth, Aamer Abdul Rahman, Laya Rafiee Sevyeri, Mohammad Havaei, and Samira Ebrahimi Kahou. Learning from uncertain concepts via test time interventions. In Workshop on Trustworthy and Socially Responsible Machine Learning, NeurIPS 2022 , 2022.

Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15 , pages 103-129, 1995.

Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 627-635. JMLR Workshop and Conference Proceedings, 2011.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144 , 2016.

Katherine M. Collins, Matthew Barker, Mateo Espinosa Zarlenga, Naveen Raman, Umang Bhatt, Mateja Jamnik, Ilia Sucholutsky, Adrian Weller, and Krishnamurthy Dvijotham. Human uncertainty in concept-based ai systems. AIES , 2023.

Li Deng. The MNIST database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-ucsd birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

Emanuele Marconato, Andrea Passerini, and Stefano Teso. Glancenets: Interpretable, leak-proof concept-based models. Advances in Neural Information Processing Systems , 35:21212-21227, 2022.

David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. arXiv preprint arXiv:1806.07538 , 2018.

Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence , 2(12):772-782, 2020.

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314 , 2021.

Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR 2022 Workshop on PAIR2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data , 2023.

Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations , 2023.

Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574 , 2023.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703 , 2019.

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: a system for large-scale machine learning. In Osdi , volume 16, pages 265-283. Savannah, GA, USA, 2016.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning , pages 448-456. pmlr, 2015.

Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml , volume 30, page 3. Atlanta, Georgia, USA, 2013.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063 , 2(417):1, 2012.

A Appendix

A.1 Software and Hardware Used

Software For this work, we evaluate all baselines using the MIT-licensed implementations of both CBMs and CEMs made public in [5]. Our implementation of IntCEM is built on top of that repository using PyTorch 1.12 [40], an open-source deep learning library with a BSD license. We incorporated CooP into this library by basing ourselves on the code released under an MIT license by Chauhan et al. [9] as part of their original publication; their code was written in TensorFlow, and as such, we converted it to PyTorch. All numerical plots and graphics have been generated using Matplotlib 3.5, a Python-based plotting library with a BSD license, while all conceptual figures were generated using draw.io, a free drawing software distributed under an Apache 2.0 license. Furthermore, when building our experiment tasks, we use TensorFlow's distribution [41] of MNIST [30] and CelebA [32] and use the processing scripts made public by Koh et al. [4] to generate our concept annotated CUB-tasks. All of our code, including configs and scripts to recreate results shown in this paper, has been released as part of CEM's official public repository found at https://github.com/mateoespinosa/cem.

Resources All our non-ablation experiments were run in a shared GPU cluster with four Nvidia Titan Xp GPUs and 40 Intel(R) Xeon(R) E5-2630 v4 CPUs (at 2.20GHz) with 125GB of RAM. In contrast, all of our ablation experiments were run on a separate GPU cluster with 4x Nvidia A100-SXM-80GB GPUs, where each GPU is allocated 32 CPUs. We estimate that to complete all of our experiments, including prototyping and initial explorations, we required approximately 350 to 370 GPU hours.

A.2 Additional Details on the Gumbel-Softmax Sampler

As introduced in Section 3, expressing interventions using Equation (1) allows us to differentiate the output embedding for concept c i with respect to the intervention mask µ after an intervention has been performed . Therefore, if the output of the sampler p ( η | ψ (ˆ c ( t -1) , µ ( t -1) )) used to generate intervention η is differentiable with respect to its parameters ψ (ˆ c ( t -1) , µ ( t -1) ) , then we can use the gradients from ∇ CE ( y, f (˜ g ( x , µ ( T ) , c )) ) , one of the terms in L pred, to update the parameters of ψ through standard back-propagation. This is because the intervention mask used at the end of the training trajectory is given by µ ( T ) = µ (0) + ∑ T t =1 η ( t ) , allowing the gradient of ˜ g ( x , µ ( T ) , c ) to be back-propagated into the generating functions of each of the interventions { η ( t ) } T t =1 . Therefore, when parameterising our learnable policy ψ as a DNN with weights θ ψ , this policy can better update θ ψ during training so that its interventions decrease both the prediction loss L pred at the end of any intervention trajectory as well as its mean rollout loss L roll .

In practice, we construct a differentiable categorical sampler p ( η | ψ (ˆ c ( t -1) , µ ( t -1) )) using a GumbelSoftmax distribution [28]: a differentiable and continuous relaxation of the categorical distribution. To use this distribution, we begin by interpreting ψ (ˆ c , µ ) = ω ∈ ( -∞ , 0) k as the log-probability that each concept will be selected for the next intervention. Then, we proceed by sampling a noise vector h ∈ R k from a Gumbel distribution such that each dimension h i of this vector is independently sampled from h i ∼ Gumbel (0 , 1) . This noise allows us to introduce non-determinism in our system, akin to the use of Gaussian noise in a Variational Autoencoder's reparameterisation trick [42]. Finally, the crux of the Gumbel-Softmax trick is to use this noise to shift the log-probabilities predicted by ψ and generating a sharp continuous representation of the sample through a softmax activation function:

Here, τ ∈ [0 , ∞ ) is a temperature hyperparameter indicating the sharpness of the output continuous representation of the sample (smaller values leading to samples that are closer to a one-hot representation of a vector) and Softmax ( ω ) is the softmax function whose i -th output is given by

In this process, one can easily sample the Gumbel noise variables h i from Gumbel (0 , 1) by first sampling u i ∼ Uniform (0 , 1) and then computing h i = -log( -log( u i )) . In this work, we always use a temperature of τ = 1 and make the output of the Gumbel Softmax discrete (i.e., into a one-hot

representation) using the Straight-Through estimator proposed by Jang et al. [28] which still enables continuous gradients to be back-propagated. Note, this procedure permits one to accidentally sample a previously intervened concept from p ( η | ψ (ˆ c ( t -1) , µ ( t -1) )) (as there may be a non-zero probability assigned to a concept c i where µ ( t -1) i = 1 ); to guard against this possibility, after sampling an intervention η ( t ) from the Gumbel Softmax distribution, we clamp all values of the new intervention mask µ ( t ) = µ ( t -1) + η ( t ) to be between 0 and 1 for all t .

A.3 Dataset Details

In this Appendix, we describe the datasets and tasks used in Section 4. A summary of each task's properties can be found in Table A.1 and a detailed discussion of each task can be found below.

MNIST-Add MNIST-Add is a visual arithmetic task based on the MNIST [30] dataset. Each sample x ∈ [0 , 1] 12 × 28 × 28 in this task is formed by 12 digit 'operands', where the i -th channel x i, : , : of x contains a grayscale zero-to-nine digit representing the i -th operand. We construct this dataset such that some operands are inherently harder to predict. To do so, we (i) constrain each of the first four operands to be at most 2, (ii) constrain each of the four middle operands to at most 4, and (iii) constrain each of the last four operands to be at most 9.

Each sample x in this task is annotated with a binary label y corresponding to whether the sum of the digits in all 12 operands is at least half the maximum attainable (i.e., at least 30 ). We construct concept annotations c ∈ { 0 , 1 } 72 for x by providing the one-hot encoding representations of each operand in x as an annotation. Because the first four operands can be at most 2 , and the next 4 and 8 operands can be at most 4 and 9 , respectively, this results in a total of 4 × 3 + 4 × 5 + 4 × 10 = 72 concepts organised into groups of 12 mutually exclusive concepts. To construct MNIST-Add 's training set, we generate 12,000 training samples by randomly selecting all 12 operands of each sample from MNIST's training set. Similarly, to construct MNIST-Add 's test set, we generate 10,000 test samples by randomly selecting all 12 operands of each sample from MNIST's test set.

MNIST-Add-Incomp MNIST-Add-Incomp is a visual arithmetic task that modifies MNIST-Add so that its set of concept annotations is incomplete with respect to the task label. This is done by randomly selecting 8 out of the 12 groups of concepts in MNIST-Add and providing the concepts corresponding to those 8 groups to all models at train-time (making sure the same group of 8 groups is selected across all baselines).

CUB CUB is a visual bird-recognition task where each sample x ∈ [0 , 1] 3 × 299 × 299 represents an RGB image of a bird obtained from the CUB dataset [31]. Each of these images comes with a set of attribute annotations - e.g., 'beak_type' and 'bird_size' - as well as a downstream label y representing the classification of one of 200 different bird types. In this work, we construct concept annotations c for this dataset using the same k = 112 bird attributes selected by Koh et al. [4] in their CBM work. These 112 concepts can be grouped into 28 groups of mutually exclusive concepts (e.g., 'white wing', 'black wing', etc). Following [4], we preprocess all of the images in this dataset by normalising, randomly flipping, and randomly cropping each image during training. We treat this dataset as a representative real-world task where one has access to a complete set of concept annotations at train-time.

CUB-Incomp CUB-Incomp is a concept-incomplete variant of the CUB task constructed by randomly selecting 25% of all concept groups in CUB and providing train-time annotations only for those concepts. For fairness, we ensure that the same groups are selected when training all baselines, resulting in a dataset with 7 groups of mutually exclusive concepts and 22 concepts.

CelebA The CelebA task is generated from samples in the Celebrity Attributes Dataset [32] using the task and concept annotations defined in [5]. For each image x in CelebA, we first select the 8 most balanced attributes [ a 1 , · · · a 8 ] from the 40 binary attributes available; we then build a vector of binary concept annotations c using only the 6 most balanced attributes [ a 1 , · · · , a 6 ] , meaning that attributes a 7 and a 8 are not provided as concept annotations. To simulate a concept-incomplete task, we construct a task label y via the base-10 representation of the number formed by [ a 1 , · · · , a 8 ] . This results in a total of l = 2 8 = 256 classes. As in [5], we downsample every image to (3 , 64 , 64) and randomly subsample images in CelebA's training set by selecting every 12 th sample.

A.4 Model Architecture and Baselines Details

In this section, we describe the architectures and training hyperparameters used in our experiments.

A.4.1 Model Architectures and Hyperparameters

To ensure a fair comparison across methods, we strive to use the exact same architectures for the concept encoder g and label predictor f across all baselines. Moreover, when possible, we follow the architectural choices selected in [5] to ensure that we are comparing our own IntCEM architecture against highly fine-tuned CEMs. The only tasks where this is not possible, as they were not a part of the original CEM evaluation, are the MNIST-based tasks. Below, we describe our architectural choices for f and g for each of our tasks. Moreover, for all tasks and models we include the choices of loss weights λ roll and λ concept where, for notational simplicity, we use λ concept to indicate the strength of the concept loss in both CEM-based models as well as in jointly trained CBMs. Finally, as in [5], for both CEMs and IntCEMs we use an embedding size of m = 16 , a RandInt probability p int = 0 . 25 , and a single linear layer for all concept embedding generators { ( ϕ -i , ϕ + i ) } k i =1 and the shared scoring model s .

MNIST-Based Tasks For all MNIST-based tasks, the concept encoder g begins by processing its input using a convolutional neural network backbone with 5 convolutional layers, each with sixteen 3 × 3 filters and a batch normalisation layer [43] before their nonlinear activation (a leaky-ReLU [44]), followed by a linear layer with g out outputs. When working with CEM-based models, the output of this backbone will be a g out = 128 , from which the concept embedding generators { ( ϕ -i , ϕ + i ) } k i =1 can produce embeddings { (ˆ c -i , ˆ c + i ) } k i =1 and from which we can construct g . Otherwise, when working with CBM-based models, the output of this backbone will be the number of training concepts g out = k , meaning that g will be given by the backbone's output. For all models, we use as a label predictor f a Multi-Layer Perceptron (MLP) with hidden layers { 128 , 128 } and leaky ReLU activations in between them. We use an MLP for f rather than a simple linear layer, as it is commonly done, because we found that CBM-based models were unable to achieve high accuracy on these tasks unless a nonlinear model was used for f (the same was not observed in CEM-based model as they have higher bottleneck capacity).

For CEMs, IntCEMs, and Joint CBM-Logit models, we use λ concept = 10 across both MNIST-based tasks. This is only different for Joint CBM-Sigmoid were we noticed instabilities when setting this value to 10 (leading to large variances) and reduced it to λ concept = 5 (reducing it lower than this value seem to decrease validation performance). All models are trained using a batch size of B = 2048 for a maximum of E = 500 epochs. To avoid models learning a majority class classifier, we weight each concept's cross entropy loss by its frequency and we use a weighted binary cross entropy loss for the task loss. Finally, for both MNIST tasks, the best performing IntCEM, determined by the model with the highest area under the intervention vs validation accuracy curve when λ roll varies through { 5 , 1 , 0 . 1 } , was obtained when λ roll = 1 .

CUB-Based Tasks As with our MNIST-based tasks, for all CUB-based tasks we construct a concept encoder g by passing our input through a backbone model formed by a ResNet-34 [45] whose output linear layer has been modified to have g out activations. The output of this backbone is set to g out = k

for all models and it is used to construct g in the same way as described for the MNIST-based tasks. For all models, we use as a label predictor f a linear layer.

For CEMs, IntCEMs, and Joint CBM-Logit models, we use λ concept = 5 across both CUB-based tasks. The only exception is in CUB-Incomp , where we noticed a high variance in Joint CBM-Sigmoid when setting this value to 5 and reduced it to λ concept = 1 as a response. All models are trained using a batch size of B = 256 for a maximum of E = 300 epochs and using a weight decay of 4 × 10 -5 as in [4]. As in the MNIST-based models, we scale each concept's binary cross entropy loss by a factor inversely proportional to its frequency to help the model learn the more imbalanced concepts. Finally, for both CUB tasks, the best performing IntCEM was obtained when λ roll = 1 .

CelebA We use the same architecture and training configuration as in the CUB-based tasks for all CelebA models. The only differences between models trained in the CUB-tasks and those trained in CelebA , is that we set the batch size to B = 512 and train for a maximum of E = 200 epochs while setting the concept loss weight λ concept = 1 for all joint CBMs and CEM-based models. When iterating over different values of λ roll for IntCEM, we found that λ roll = 5 works best for this task.

A.4.2 Training Hyperparameters

In our experiments, we endeavour to provide all models with a fair computational budget at train-time. To that end, we set a large maximum number of training epochs and stop training early based on each model's own validation loss. Here, we stop early if a model's validation loss does not decrease for 15 epochs. We monitor the validation loss by sampling 20% of the training set to make our validation set for each task. This validation set is also used to select the best-performing models whose results we report in Section 4. We always train with stochastic gradient descent with batch size B (selected to well-utilise our hardware), initial learning rate r initial, and momentum 0 . 9 . Following the hyperparameter selection in reported in [5], we use r initial = 0 . 01 for all tasks with the exception of the CelebA task where this is set to r initial = 0 . 005 . Finally, to help find better local minima in our loss function during training, we decrease the learning rate by a factor of 0 . 1 whenever the training loss plateaus for 10 epochs.

A.4.3 IntCEM-Specific Parameters

As discussed in Section 3, IntCEM requires one to select a prior for its initial trajectory length and intervention mask, as well as an architecture to be used for the intervention policy model ψ . Below we describe who we set these values for all of our experiments. Later, we discuss the effect of some of these hyperparameters on our results in our ablation studies in Appendix A.6.

Prior Selection In our experiments, to allow our prior to imitate how we intervene on IntCEMs at test-tine, we construct p ( µ ) by partitioning c into sets of mutually exclusive concepts and selecting concepts belonging to the same group with probability p int = 0 . 25 . Complementing this prior, we construct the intervention trajectory horizon's prior p ( T ) using a discrete uniform prior Unif (1 , T max ) where T max is annealed within [2 , 6] growing by a factor of 1 . 005 every training step. This was done to help the model stabilise its training by first learning meaningful concept representations before employing significant resources in improving its learnable policy ψ .

Intervention Policy Architecture For all tasks, we parameterise IntCEM's learnable policy ψ using a leaky-ReLU MLP with hidden layers { 128 , 128 , 64 , 64 } and Batch Normalisation layers in between them. When groups of mutually exclusive concepts are known at train-time (as in CUB-based tasks and MNIST-based tasks), we let ψ 's output distribution operate over groups of mutually exclusive concepts rather than individual concepts. This allows for a more efficient training process that better aligns with how these concepts will be intervened on at test-time. Finally, when training IntCEMs we use global gradient clipping to bound the global norm of gradients to 100 . This helps us avoid exploding gradients [46] as we backpropagate gradients through our intervention trajectory.

A.5 Computational Resource Usage

As mentioned in Sections 4 and 5, IntCEMs bear an additional overhead in their training times compared to traditional CEMs due to their need to sample intervention trajectories during training. In this section, we first study how training times vary across all of our baselines. Then we show that

IntCEM's training time overhead is amortised in practice if one considers the efficiency of IntCEM's learnt policy ψ over existing state-of-the-art intervention policies.

×

In Table A.2 we show the wall-clock times of a single training epoch for each of our baselines as well as the number of epochs it took the model to converge (as determined by our early stopping procedure defined in Appendix A.4). Our results indicate that, compared to traditional CEMs, IntCEMs take approximately 60% longer than CEMs per training epoch while also taking approximately ∼ 20% longer to converge (in terms of epochs). Nevertheless, we underscore three main observations from these results. First, the results in Table A.2 are very highly implementation-and-hardware-dependent, meaning that they should be taken with a grain of salt as they could vary widely across different implementations, batch sizes, and machines. Second, although on average we observe that IntCEMs take more epochs to converge than traditional CEMs, the variances we observe across different random initialisations are large. This means that these differences are not likely to be statistically significant; in fact, we observe some runs where IntCEMs converged faster than CEMs. Finally, as seen in Table A.3, these extra training costs may be amortised in practice when one considers that IntCEM's learnt policy is much faster than an equivalent competitive intervention policy like CooP. This implies that if these models are expected to be intervened frequently in practice, the overhead spanning from IntCEM's training procedure may result in better amortised computational resource usage over CEMs intervened on with the CooP policy.

A.6 Exploring IntCEM's Hyperparameter Sensitivity

In this section, we explore how our results reported in Section 4 change as we vary several of IntCEM's hyperparameters, specifically, λ roll , λ concept , γ , and p int and T max.

We begin by exploring its sensitivity to its two loss weights, λ roll and λ concept, and show that, although such parameters have an impact mostly on the unintervened accuracy of IntCEMs, our model is able to outperform CEMs once interventions are introduced throughout multiple different values for such weights. Then, we investigate how sensitive IntCEM is to its intervened prediction loss scaling factor γ and show that its performance varies very little as γ is modified. Finally, we explore how the hyperparameters we selected for IntCEM's prior ( p int and T max) affect its performance and describe similar results to those observed when varying IntCEM's loss weight hyperparameters.

For all the results we describe below, we train IntCEMs on the CUB task while fixing γ = 1 . 1 , λ concept = 5 , λ roll = 1 , p int = 0 . 25 , and T max = 6 unless any of these hyperparameters is the target

0

5

10

15

20

25

Groups Intervened

20

30

40

50

60

70

80

90

100

Task Accuracy (%)

0 0.1 1

5 10

CEM

roll

0

10

20

30

40

50

60

70

80

90

100

Mean Concept AUC (%)

IntCEM (

concept

= 10)

IntCEM (

concept

= 5)

IntCEM (

concept

= 1)

IntCEM (

concept

= 0.1)

IntCEM (

concept

= 0)

CEM (

concept

=5

)

(a)

λ

concept

ablation study in

CUB

.

0

5

10

15

20

25

Groups Intervened

30

40

50

60

70

80

90

100

Task Accuracy (%)

0 0.1 1

5 10

CEM

roll

60

70

80

90

100

Mean Concept AUC (%)

IntCEM (

roll

= 10)

IntCEM (

roll

= 5)

IntCEM (

roll

= 1)

IntCEM (

roll

= 0.1)

IntCEM (

roll

= 0)

CEM

(b)

λ

roll ablation study in

CUB

.

of our ablation. Furthermore, we include as a reference the accuracy of a CEM trained in this same task. Finally, when showing intervention performance, all interventions in IntCEMs are made following their learnt policy ψ while interventions in CEMs are made following the CooP policy (the best-performing policy for CEMs based on our experiments in Section 4).

A.6.1 Rollout and Concept Loss Weights Ablations

In Figure A.1 we show the results of varying λ roll and λ concept in { 0 , 0 . 1 , 1 , 5 , 10 } in the CUB task. Our ablation studies show that IntCEM's initial unintervened performance can vary as these two parameters are changed, particularly when λ concept is small or λ roll is large. Similarly, we observe that the average concept AUC of IntCEM decreases as λ concept is very small, as one would expect by λ concept's definition, and is affected as we increase the value of λ roll, as described in Section 4.1. Nevertheless, and perhaps more importantly, we notice that for all hyperparameter configurations, IntCEM outperforms a traditional CEM's intervened task accuracy within five interventions. This is observed even when λ roll = 0 or λ concept = 0 and suggests that our model's preconditioning to intervention receptiveness is robust to changes in its hyperparameters . Moreover, we highlight that IntCEM's sensitivity to λ concept is not inherent to our architecture but rather inherent to any jointly trained CBMs and CEMs, as shown by Koh et al. [4] in their original CBM evaluation.

Looking at our λ roll ablation study, we observe that IntCEM's concept accuracy and unintervened task accuracy seem to decrease as λ roll increases. We hypothesise that happens as our model has a greater incentive to depend on interventions, and therefore can afford to make more concept and task mispredictions in the absence of interventions. Furthermore, we observe that the variance in IntCEM's unintervened task performance and concept performance as λ roll changes are less significant than that observed when λ concept changes, in fact achieving competitive accuracies for all values except when λ roll = 10 . This suggests that when fine-tuning IntCEM's loss hyperparameters using a validation set, as done in our experiments, it is recommended to leave λ roll within a small value (we found

0

5

10

15

20

25

Groups Intervened

70

80

90

100

Task Accuracy (%)

0.5 0.9

1

1.1 1.5

2

CEM

50

60

70

80

90

100

Mean Concept AUC (%)

IntCEM (

= 2)

IntCEM (

= 1.5)

IntCEM (

= 1.1)

IntCEM (

= 1)

IntCEM (

= 0.9)

IntCEM (

= 0.5)

CEM

success with λ roll ∈ [0 . 1 , 5] ) while spending more efforts fine-tuning λ concept, a hyperparameter that unfortunately needs fine tuning for all jointly trained CBM-based architectures.

A.6.2 Task Loss Scaling Factor

In Section 3, we defined γ ∈ [1 , ∞ ) as a hyperparameter that determines the penalty for mispredicting the task label y with interventions µ (0) versus mispredicting y with interventions µ ( T ) at the end of the trajectory. In all experiments reported in Section 4, we fixed γ = 1 . 1 . In our ablation study for γ (see Figure A.2), we show that IntCEM's receptiveness to interventions, and it mean concept AUC, is robust and not significantly affected as γ varies within a reasonable range [0 . 5 , 2] . In particular, and as expected, we observe that for a higher value of γ , the model has slightly lower unintervened accuracy while attaining slightly higher intervened accuracy after a few interventions. Nevertheless, these differences are small and suggest that our method is robust to minor variations in this hyperparameter.

A.6.3 Prior Hyperparameters Ablation

IntCEM uses two prior distributions to sample trajectories at train-time: (i) p ( µ i ) = Bernoulli ( p int ) on the initially sampled masks µ (0) ∈ { 0 , 1 } k and (ii) p ( T ) = Unif (1 , T max ) on the horizon, or length, of the intervention trajectory to be sampled. Our ablation studies on both T max and p int show that IntCEMs outperform CEMs when the number of interventions increases, regardless of the parameters used for p int and T max. Nevertheless, we observe that when T max is small, e.g. T max = 1 , or p int is high, e.g. p int = 0 . 75 , IntCEM has a slower increase in performance as it is intervened. We believe this is a consequence of our model not having long-enough trajectories at train-time, therefore failing to condition the model to be receptive to long chains of interventions. This is because as p int is larger, the number of concepts initially intervened will be higher on expectation, leading to potentially less impactful concepts to select from at train-time. Similarly, as T max is lower, IntCEM's trajectories will be shorter and therefore it will not be able to explore the intervention space enough to learn better long-term trajectories. Therefore, based on these results our recommendation for these values, leveraging performance and train efficiency, is to set T max to a small value in [5 , 10] (we use T max = 6 in our experiments) and p int to a value within [0 , 0 . 5] (we use p int = 0 . 25 in our experiments).

A.7 Additional Results and Details on Intervention Policy Evaluation

In this section, we provide further details on the set-up of our intervention policy experiments and include additional results on the impact of intervening on IntCEMs and our baselines.

0

5

10

15

20

25

Groups Intervened

70

80

90

100

Task Accuracy (%)

0 0.250.50.75 1

CEM

pint

50

60

70

80

90

100

Mean Concept AUC (%)

IntCEM (

pint

= 1)

IntCEM (

pint

= 0.75)

IntCEM (

pint

= 0.5)

IntCEM (

pint

= 0.25)

IntCEM (

pint

= 0)

CEM

(a)

p

int

ablation study in

CUB

.

0

5

10

15

20

25

Groups Intervened

70

80

90

100

Task Accuracy (%)

1

5

10

15

CEM

Tmax

50

60

70

80

90

100

Mean Concept AUC (%)

IntCEM (

Tmax

= 15)

IntCEM (

Tmax

= 10)

IntCEM (

Tmax

= 5)

IntCEM (

Tmax

= 1)

CEM

(b)

T

max ablation study in

CUB

.

A.7.1 Intervention Policy Details

Intervening on Logit CBMs Because Joint CBM-Logits use log-probabilities in their bottleneck rather than sigmoidal zero-one probabilities, intervening on their bottlenecks is slightly different than intervening on Joint CBM-Sigmoidal models. As in [4], we avoid setting intermediate concepts to out-of-distribution values by setting ˆ c i to its empirical 95-th percentile value (computed over the training distribution) when we intervene on concept c i and want to indicate that its ground truth value is ˜ c i = 1 . Similarly, when we intervene on concept c i and want to indicate that its ground truth value is ˜ c i = 0 , we set it to its empirical 5-th percentile value (computed over its training distribution). This allows us to extend the intervention formulation in Equation (1) to CBMs with logit bottlenecks by letting ˆ c + i be the 95-th percentile value of the empirical training distribution of ˆ c i and letting ˆ c -i be the 5-th percentile value of the same distribution.

CooP CooP selects the best next concept to intervene on by considering a weighted sum of (i) an uncertainty measurement of each concept's predicted probability (weighted by hyperparameter α ), (ii) the expected change in the probability of the currently predicted class (weighted by hyperparameter β ), and (iii) the cost of acquiring each concept (weighted by hyperparameter γ coop). For simplicity, in this work, we treat all concepts as having the same acquisition cost; i.e., we set γ coop = 0 and only fine-tune α and β . We leave the exploration of variants of our method which consider different acquisition costs for future work.

0

4

8

12

74

79

84

89

94

99

MNIST-Add

Task Accuracy (%)

IntCEM

0

4

8

12

74

79

84

89

94

99

CEM

0

4

8

12

74

79

84

89

94

99

Joint CBM-Logit

0

4

8

12

74

79

84

89

94

99

0

4

8

12

74

79

84

89

94

99

0

4

8

12

74

79

84

89

94

99

Independent CBM

0

4

8

73

78

83

88

93

98

MNIST-Add-Incomp

Task Accuracy (%)

0

4

8

73

78

83

88

93

98

0

4

8

73

78

83

88

93

98

0

4

8

73

78

83

88

93

98

0

4

8

73

78

83

88

93

98

0

4

8

73

78

83

88

93

98

0

4

8 12 16 20 24 28

48

58

68

78

88

98

CUB

Task Accuracy (%)

0

4

8 12 16 20 24 28

48

58

68

78

88

98

0

4

8 12 16 20 24 28

48

58

68

78

88

98

0

4

8 12 16 20 24 28

48

58

68

78

88

98

0

4

8 12 16 20 24 28

48

58

68

78

88

98

0

4

8 12 16 20 24 28

48

58

68

78

88

98

0

2

4

6

37

47

57

67

77

87

97

CUB-Incomp

Task Accuracy (%)

0

2

4

6

37

47

57

67

77

87

97

0

2

4

6

37

47

57

67

77

87

97

0

2

4

6

37

47

57

67

77

87

97

0

2

4

6

37

47

57

67

77

87

97

0

2

4

6

37

47

57

67

77

87

97

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

CelebA

Task Accuracy (%)

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

0

2

4

6

Groups Intervened

14

24

34

44

54

64

74

Skyline (Oracle)

Random

UCP

CooP

CVA

CVI

When using CooP for any of our baselines, we select α and β by performing a grid search over α ∈ { 0 . 1 , 1 , 10 , 100 } and β ∈ { 0 . 1 , 1 , 10 , 100 } and selecting the pair of hyperparameters that give us the best area under the validation intervention curve. Given that this process can be costly (in our runs it may take up to ∼ 45 minutes to run), we estimate the area under the validation intervention curve by checking CooP performance after intervening with 1% , 5% , 25% , and 50% of the available concept groups for a given task.

Behavioural Cloning Policy To learn a behavioural cloning policy which imitates Skyline through demonstrations of the latter policy, we train a leaky-ReLU MLP with hidden layers { 256 , 128 } that maps a previous bottleneck ˆ c and a mask of previously intervened concepts µ to k outputs indicating the log-probabilities of selecting each concept as the next intervention. This model is trained by generating 5 , 000 demonstrations ((ˆ c , µ ) , η sky ) of 'Skyline'.

Each demonstration is formed by (1) selecting a training sample x from the task's training set uniformly at random, (2) generating an initial intervention mask µ by selecting, uniformly at random, l ∼ Unif (0 , k ) concepts to intervene on, (3) producing an initial bottleneck ˆ c = ˜ g ( x , µ, c ) using concept encoder g of the model we will intervene on and x 's corresponding concept labels c , and (4) generating a target concept intervention η sky by calling the Skyline policy with inputs (ˆ c , µ ) . This BC model is then trained to minimise the cross entropy loss between its predicted probability distribution

0

2

4

6

Groups Intervened

40

60

80

100

Task Accuracy (%)

CUB-Incomp

IntCEM

CEM

Joint CBM-Logit

and η sky for 100 epochs using stochastic gradient descent with learning rate 0 . 01 , batch size 256 , weight decay 4 × 10 -5 , and momentum 0 . 9 .

We highlight that when selecting concepts from both the BC policy and the learnt policy ψ at test-time, we deterministically select the concept with the maximum log-probability that has not yet been selected by µ .

A.7.2 Additional Results

As discussed in Section 4, in Figure A.4 we observe that IntCEMs achieve significantly higher task accuracies across all tasks under interventions, regardless of the intervention policy used. We notice, however, that the gap between the optimal intervention policy and IntCEM's best-performing policy tends to be larger in the simpler datasets. Further work can investigate the origins of this gap as well as mechanisms to bridge it.

A.8 Extending IntCEM's Loss to Traditional CBMs

Because IntCEM's loss function only assumes that one can intervene on a given model at train-time, this loss function is general enough to be applicable to traditional Joint CBMs. In Figure A.5 we show the results of intervening on joint CBMs whose training objectives have been modified to include IntCEM's L roll and L pred terms ( IntCBMs ). These results show that IntCBMs, in particular when using a logit bottleneck, can achieve a higher test accuracy than their traditional CBM counterparts when presented with a large number of test-time interventions. Nevertheless, we note the following limitations: (i) we observe that for IntCBMs with sigmoidal bottlenecks, there is no improvement regardless of the use of interventions, and in fact, adding L pred and L roll to traditional CBMs can in fact lead to worse performance; (ii) even in IntCBMs with logit bottlenecks whose intervened performance outperforms their CBM counterparts, we observe a high variance, especially when the number of intervened concepts is small; and (iii) in comparison to IntCEMs, the improvements of our training loss in IntCBMs with logit activations are still significantly underperforming when receiving little-to-none interventions. We believe that these observations arise for two core reasons: first, using concept embeddings allows for a better flow of information between the concept encoder and the label predictor, leading to embedding-based models being more robust to concept incompleteness than their scalar-based counterparts (especially with respect to sigmoidal bottleneck). Second, using embeddings rather than scalar concept representations enable gradients to flow from the label predictor into the concept encoder even after interventions are made , something that is not possible with traditional CBMs as setting a bottleneck activation to a fixed value (e.g., 0 or the 5-th percentile of the activation) is a gradient-blocking operation. This also leads to significantly more stable gradient propagation throughout an intervention trajectory, explaining the high variance in IntCBMs. These results suggest that using high-dimensional embedding representations for concepts may be a crucial

0

4

8

12 16 20 24 28

Intervened

0

10

20

30

40

50

60

70

80

Task Accuracy (%)

CUB

0

2

4

6

8

Intervened

0

10

20

30

40

50

60

70

CUB-Incomp

0

2

4

6

8

Intervened

0

10

20

30

CelebA

IntCEM (Leant Policy

)

IntCEM (Random Policy)

CEM (Random Policy)

component for these models to be receptive to interventions, and further, simply including a higher train-time penalty when mispredicting a label after an intervention trajectory is not enough for a model to become competitive with and without interventions.

A.9 Effect of Adversarial Interventions

In their evaluation, Espinosa Zarlenga et al. [5] observed that CEMs were somewhat more robust to adversarial interventions (i.e., interventions that are intentionally incorrect) compared to existing CBM architectures - where robustness was defined as being able to withstand a certain number of wrong interventions before significantly dropping in performance. This suggests that CEMs may have a form of error correction in their embeddings that allow the label predictor to correct intervention mistakes. In contrast, as posited in our discussion in Section 5, by encouraging IntCEMs to be receptive to test-time interventions, we are also incentivising them to be receptive to all forms of interventions, even if those interventions are adversarial in nature. This is because our model's explicit train-time intervention loss, which encourages IntCEMs to positively respond to test-time interventions, has the underlying assumption that interventions, when provided at train-time or testtime, are correct. Hence, we hypothesise that the same error correction observed in CEMs will not be seen in IntCEMs, leading our models' performance to significantly drop when interventions are wrong. We find evidence supporting this hypothesis when performing adversarial interventions across all our baselines on the CUB , CUB-Incomp , and CelebA tasks. Specifically, as seen in Figure A.6, we observe that an IntCEM's performance drastically drops when receiving only a handful of adversarial interventions, with this result being even more damaging if we follow its learnt intervention policy ψ . These results suggest that a very interesting and impactful direction for future work can be to explore mechanisms to introduce error correction in IntCEM-like models.

A.10 Leakage in IntCEM's Learnt Representations

Concept leakage [36, 10] is a known issue in CBMs where a CBM's concept encoder learns to encode unnecessary information in its learnt concept representations in order to bypass information from the input to its label predictor. It has been empirically shown that leakage may lead to less interpretable concept representations [36] and to detrimental interventions on CBMs [10]. The latter is a consequence of a CBM's label predictor learning to rely on information that accidentally leaked into the concept representations, information that is necessarily destroyed at intervention-time when an expert overwrites a bottleneck activation with its ground truth value. Nevertheless, the same is not true in embedding-based architectures, such as CEMs and IntCEMs, where interventions simply involve 'swapping' an embedding with one that can still carry leaked information into the label predictor (see Equation 1). Hence, in this section we hypothesize that higher leakage may, in fact, be a contributor to IntCEMs' better performance when being intervened on as the model may learn to take better advantage of this leakage to learn to be more receptive to interventions at test-time.

We evaluate our hypothesis by measuring the Oracle Impurity Score (OIS) [10] of concept representations learnt by CBMs, CEMs, and IntCEMs across all tasks. This score, between 0 and 1 , measures how much extra information, on average, each learnt concept representation captures from other possibly unrelated concepts (with higher scores representing higher impurities and, therefore, more leakage between concepts). Our results, shown in Table A.4, show that not only is there significantly more leakage in CEMs compared to CBMs, as one would expect given their higher capacity, but IntCEM's embeddings are capturing more impurities than CEM's embeddings across all tasks but one, providing evidence towards our hypothesis. This preliminary study suggests that contrary to common assumptions, leakage may not always be undesired and could be a healthy byproduct of more expressive concept representations in models that accommodate such expressivity. Nevertheless, this phenomenon may also potentially lead to IntCEM's concept representations being less interpretable than those in CEMs when such leakage is unaccounted for. Therefore, we encourage future work to explore leakage's positive and negative consequences in embedding-based concept models to design better inductive biases for effective interventions.

A.11 Frequency of Selected Concepts by IntCEM's policy

In Figure A.7 we show the frequencies of concept groups selected by IntCEM's ψ policy when intervening on the validation set of CUB . To study how much this policy changes across different initialisations and random seeds for the same hyperparameters and models, we show these histograms across multiple instances of identically trained IntCEMs with different random seeds. We notice that although, in most cases, concepts such as 'upperparts colour' and 'size' seem to be highly preferred in the early steps of intervention, there seems to be a relatively large variance across multiple seeds. This may suggest that there are several equally informative or good concepts one may request at a given time to obtain similar improvements, something that we leave for future work to explore.

A.12 Tabulation of Results

This section presents Tables A.5, A.6, and A.7, a set of tabular summaries of our results introduced in Section 4. These summaries show the precise means and standard deviations (computed over five different random seeds) of test-time accuracies after interventions for all baselines and tasks when deploying various intervention policies. They represent tabular versions of the results presented in Figures 3, 4, and 5, respectively.

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

1000

2000

Selected Counts

Learnt Policy Selection at Step 1

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection After 6 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2500

5000

Selected Counts

Learnt Policy Selection After 12 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection at Step 1

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2500

5000

Selected Counts

Learnt Policy Selection After 6 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2500

5000

Selected Counts

Learnt Policy Selection After 12 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

500

Selected Counts

Learnt Policy Selection at Step 1

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

Selected Counts

Learnt Policy Selection After 6 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection After 12 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

1000

2000

Selected Counts

Learnt Policy Selection at Step 1

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection After 6 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2500

5000

Selected Counts

Learnt Policy Selection After 12 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

500

1000

Selected Counts

Learnt Policy Selection at Step 1

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection After 6 Steps

wing color

upperparts color

underparts color

breast pattern

back color

tail shape

upper tail color

head pattern

breast color

throat color

eye color

bill length

forehead color

under tail color

nape color

belly color

wing shape

size

shape

back pattern

tail pattern

belly pattern

primary color

leg color

bill color

crown color

wing pattern

Concept Group

0

2000

4000

Selected Counts

Learnt Policy Selection After 12 Steps

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe intervenability of concept bottleneck models has been taken for granted thus far. There have been numerous methods that tried intervening or propose policies to intervene. However, previous methods did not go so far as to optimize models for interventions. This work proposes intervention-aware training and extends Concept Embedding Models for this purpose. With experiments across MNIST, CUB, and CelebA, the authors demonstrate that IntCEM achieves on-par downstream performance compared to CEMs, and investigate the intervention performance with two sets of automated studies.\n\n**After the rebuttal:** I read and acknowledged the authors rebuttal. I am leaning towards an acceptance position after the rebuttal, given that the authors clarified my questions. I find the paper's message interesting and the presentation to be clear and well-written, the reason I am not giving a higher score is the lack of real-world settings to test the method in more complex settings and where the models do not need concept annotations at the training time.\n\nSTRENGTHS:\n1. Optimizing a model for intervention awareness is an interesting and strong idea. Many works did indeed attempt interventions, whereas it is infrequent to optimize the model for this purpose and the intervenability has been taken for granted. I do have a question about whether the current form of this is useful (Q1), yet I still believe the idea is worth sharing with the community.\n\n2. I believe having an intervention policy has implications beyond interventions per se. By looking at the Rollout Loss, I could imagine the policy revealing locally important / less important concepts and performing selective inference. While this is a hypothesis that needs verification, this could be an interesting direction to think about.\n\n3. (Minor) I appreciate the clarity in the writing. Formalization is clear and easy to digest even though there are nuanced tricks to make the method work.\n\nWEAKNESSES:\n1. In my understanding, there is a disconnect between the promise/message of the paper and the experiments. Concretely;\n\n- 1.1 In many parts of the paper, there is a theme around getting the models ready for expert feedback, which can make sense. However, I do not fully understand how we can infer that the current method achieves something related to a user or an expert. \n\n- 1.2 In this regard, a major weakness is the lack of a user study. There are many arguments in the paper that touch upon optimizing the model for user interventions, whereas there are no experiments that verify this argument. An experiment where the users used these models, intervened on them, and IntCEM performed better than baselines would support the argument here.\n\n- 1.3 Furthermore, the selection policy encodes a certain belief: That the users will choose concepts that would improve downstream performance. An alternative user metric would be showing that users prefer the concepts chosen by the selection policy. This may or not may not be the case. One could have also hypothesized that the policy can pick concepts that the users would also like to intervene on more than other policies, but again, there is no supporting evidence for this argument either.\n\nIn general, I believe this is an important disconnect that needs addressing.\n\n2. The usability of the method suffers from the requirement to have concept annotations at training time, as do vanilla CBMs and CEMs. However, there are still concept-based models that could be worth exploring in this context that are already cited in the paper as [26, 27] which are Post-hoc CBMs and Label-Free CBMs that also scale to ImageNet/COCO. The authors make the argument that the method would complement those architectures as well, and I can believe this hypothesis, but this needs verification. \n\n3. The above weakness also leads to another one, which is the lack of experimental settings closer to the real-world such as ImageNet, COCO, or medical datasets that were tested before in the above papers. I do think experiments would benefit from expanding beyond CUB and CelebA. Personally, I believe this would also allow impactful user studies.\n\n4. It is hard for me to interpret the numbers in Sections 4.2 and 4.3. This is also connected to the 3rd point above – but the improvements in MNIST or CUB seem 1-2%. Further, as a minor comment, while there are Figures, all of the y-axes are in different scales, and I cannot really tell the numbers.\n\nQUESTIONS:\n1. In terms of the story, while the premise of intervention awareness sounds interesting at face value, I still think the case needs to be made for such a need. I believe it could help to give a concrete use case before laying out the method. E.g. L118-119 authors mention an interesting example, “may be hard for the model to predict on its own”. Can authors make it more concrete, give real world examples / formalize this? When would it hurt to not be intervention aware? When would we benefit from being intervention aware? \n\n2. I would like to ask the authors’ reasoning about the first weakness I raised in the Weaknesses section. If intervention awareness is indeed to improve the interaction between the model and the human, how should we think about the current results? Is there a way to deduce that models are more ready for humans? Can we infer that humans would prefer this notion of intervention awareness over previous methods (or not having it)?\n\n3. In my understanding, according to the loss function after L194, the selection policy $\\psi$ is optimized to select the concept that would increase the probability of the ground-truth label being predicted the most, upon the intervention. Would this not bias the selection policy towards more typical interventions for the class (e.g. where $P(Y|C)$ is high)? For instance, say that a bird specie X is 95% of the time red but 5% of the time blue, whereas other birds are more often blue. Then, wouldn’t this loss function incentivize the selection policy to pick the color concept for blue X birds?\n\n4. Similar to the above, while Section 4.3 studies the performance improvement from the policies, it is not clear what these policies pick. Do the authors have any analyses that explain what the chosen concepts are? For instance, are these the concepts that are specific to a class? Are these the most common concepts for the class, are there any other patterns there? \n\n5. (Minor) I would appreciate having the Tables in addition to the figures, I’m sorry if I missed them but I could not find them in the Appendix either. Numbers could be hard to read from the Tables.\n\nLIMITATIONS:\nI think the most significant limitation is the lack of an actual real-world study to verify the claims. In my understanding, it is not possible to infer this from the current results. I would appreciate hearing the authors' reasoning about this.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a modified version of CEM that is better at receiving human test-time intervention by explicitly incorporating intervention into the training stage. Specifically, an intervention prediction module is trained to behavior clone an optimal-greedy intervention policy (Skyline). The CEM is trained to optimize for the correctness of the initial guess as well as the prediction after the predicted intervention. The idea of the method is simple and straightforward. Extensive experiments explore design choices and show strong results in practical settings.\n\nSTRENGTHS:\n* The paper is well-written. Extremely easy read and all concerns I have about the method are covered in experiments.\n* Simple and effective idea anchored on the premise that models not explicitly trained to receive intervention might not handle intervention well (empirical support in  Fig 3, 4).\n* Questions proposed in P6L221 - 230 effectively guides reader through the work and the authors' through process.\n* The authors care about how this method could realistically be applied in real-world settings.\n    * Heuristics for tuning hyperparameters for the losses are provided. \n    * Considers realistic settings of applying IntCEM where human-intervention might not be greedy-optimal and show empirically that IntCEM still performs better than baselines under non-optimal interventions (even random intervention).\n    * Observe desirable properties for application in the real world (P7L265)\n\nWEAKNESSES:\n* The only weakness is lacking human study but this could be conducted rather straightforwardly. Given the extensive study on intervention policies, I believe this to be merely nitpicking.\n* This is obviously a work that would induce plenty of follow-up work. Perhaps adding a section for what types of design choices have not been explored yet (e.g. other RL methods besides BC) could help stimulate the research community.\n\nQUESTIONS:\n* How severe is information leakage for CEMs? This is tangent to this work's contribution but since this work is based on the CEM framework, perhaps the authors could comment on whether IntCEM succeeds because of the leakage of downstream task information into the concept embeddings?\n* Typo:\n    * P2L39 artefact -> artifact\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors in this paper proposed a novel method of improving test-time interventions for Concept Bottleneck Models (CBMs). Although many CBM works showcase their ability to do intervention, none of them explicitly motivate the learned model to do well on intervention during training phase, hence hindering their ability and reliability to do intervention correctly. This work builds on top of concept embedding models (CEMs) and constructs an additional probability distribution (parameterized by neural networks) to learn what concepts to intervene on. To learn this distribution, the authors proposed a composite of three losses, where the first loss (Rollout loss) follows a greedy strategy of selecting concepts that yield the maximum probability of the given class, the second (task prediction loss) penalizes wrong predictions with many interventions, in which the penalty scales with the number of intervention performed. Finally, the third loss concept loss is just binary cross entropy to promote the correctness of the concept prediction. Empirical results on MNIST and CUB (and their variants) show that the proposed formulation performs competitively against state-of-the-art CEMs/CBMs that also utilizes interventions to improve performance. The authors also argued that the sampling from the learned distribution is much more scalable than current state-of-the-art methods.\n\nSTRENGTHS:\nThe paper is well-written and is organized in a chronological and clear manner. Introduction sufficiently motivates the problem formulation. The background and related work is written neatly so that the paper is sufficiently self-contained. The description of the method and notation is also clear and easy to follow.  And experiments have clear objectives and empirical results clearly demonstrate the effectiveness of their method. \n\nGiven CBMs are a very popular topic, and that there is a lack of work in the current field that focuses on improving test-time interventions in CBMs, I consider the formulation proposed in this work  of significantly novel. It definitively addresses a limitation of CBMs/CEMs and proposed a good formulation that addresses that limitation.\n\nWEAKNESSES:\nThere are no major weaknesses in this paper. But a few things that I suggest the authors to add for the sake of completeness:\n\nAs the main selling point of interpretable-by-design methods are interpretability, it feels a bit odd that authors did not add any examples of interventions on a particular sample. Even though the metrics clearly show the method is effective, a few concrete test samples and use of intervention can better illustrate what interventions do, how the posterior of the task prediction changes before/after interventions, etc. Potentially, illustrations could help readers who may perhaps be from a different field.\n\nQUESTIONS:\n1. I understand that “the single MC sampling per training step” is valid, but what is the main motivation for doing so? Is it to improve training speed? And do you speculate, (even better, show with ablation studies) that improving this estimation can improve convergence when optimizing your objective? \n2. In Figure 3, can you explain  why intervention could cause a drop in performance for Join CBM-Logit? Since the proposed objective explicitly optimizes for CE after intervention, would you agree that your method trained to prevent a drop in performance after intervening from happening?  \n3. How long does it take to train typical model with the proposed objective? I would imagine the rollout loss can be slow when T is large.  Does restricting T to be small heavily impacts the performance of the trained model?\n\nLIMITATIONS:\nThe limitations of this work is addressed in the main manuscript and is well-stated. The answer to my questions above can potentially brought up as limitation, and if so, I would suggest the authors to add that to the limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce IntCEMs, an extension of Concept Embedding Models designed\nspecifically to react correctly to external interventions to the learned concepts.\nCompared to regular CEMs, IntCEM feature two additional elements:  a policy that,\nessentially, guesses what interventions a human expert would do on the model for\nany given input, and a penalty that regularizes the model to achieve high accuracy\nunder interventions.  The idea is to ensure the model performs well even after\ninterventions - under the assumption the model requires interventions to ``ask\nfor help'', i.e., to obtain the ground-truth label of certain concepts at test\ntime.  Experiments show that in fact IntCEMs outperform CEMs and CBMs on four\ndata sets at test time under interventions, even from a different policy.\n\n**Post-rebuttal update**: Increasing the score by one point under the assumption\nthe authors will fairly highlightly leakage as a limitation of CBMs/CEMs/IntCEMs\nand its interaction with interventions in the paper.  I am still a bit concerned about\nimpact, but the contents of the paper are good quality overall.\n\nSTRENGTHS:\n+ English is good, narrative is generally good but parts of it are confusing, see below\n+ Motivation is sensible\n+ Proposed approach is also sensible\n+ Empirical evaluation is generally positive, with only some minor drops in concept quality depending on choice of \\lambda_roll\n+ Related work is sufficient\n\nWEAKNESSES:\n- One issue is limited significance, in the sense that the authors tackle a problem that affects only a certain type of operation (namely, a *specific* type of interventions - not all possible interventions) performed on a certain type of model (CEMs, CBMs).  It is true that the performance gain of IntCEM vs CEM under interventions can be substantial (CelebA in figure 3), but it is generally more modest (CUB, CUB-Incomp; the diff in perf for MNIST-Add and MNIST-Add-Incomp are somewhat biased by the difference at Groups Intervened = 0, which is not due to extra robustness to interventions).  This is not a deal breaker, and indeed my score is positive, but it explains why I decided not to go above weak accept.\n\n- Writing is a bit confusing.  It took me a while to understand where the ``learning to receive help'' fits into the abstract and introduction.  The idea is that models can ask users to help them at test time by - essentially - requesting the ground-truth label of certain concepts, which means that this work is closer in spirit to concept-level active learning (but at test time) rather than to interventions proper.  These have a causal connotation and can be used for changing the concepts to *any* value, not only to their ground-truth values.  See for instance all the literature on algorithmic recourse.  The target task should be made more explicit, at the bare minimum in the introduction.  I also feel the text abuses greek letters (all of section 3) a little, hindering readability.  Every time I see, say, \\eta, \\psi, \\omega... in the text, I have to look back at their definition.  The $c$ letter also appears in too many variants.  More generally, the text feels quite compressed.\n\n- Evaluation is only carried out against CEMs and CBMs.  Other self-interpretable models exist and have been published recently.  Considering the focus on CBMs, this is not a major issue, but it is an issue nonetheless.\n\nMinor issues worth fixing\n-------------------------\n\n- line 102: the range {0, 1}^k is wrong, \\tilde c_i is set to 0.5.\n\n- Considering \\mu_i also appears in the definition of \\tilde c_i, Eq 1 is probably too complex for what it needs to achieve.  Is it possible to   simplify it?\n\nQUESTIONS:\nQ1. It was shown by:\n\n  Mahinpei et al. \"Promises and pitfalls of black-box concept learning models\", 2021.\n\n  Margeloiu et al. \"Do Concept Bottleneck Models Learn as Intended?\", 2021.\n\nthat CBMs suffer from \"concept leakage\", whereby learned concepts encode information from unintended sources (like unobserved concepts), with negative consequences of interpretability.  I am wondering whether 1) How IntCEMs fare in this regard; specifically, I am only aware of two self-interpretable models that specifically address this, namely:\n\n  Havasi et al. \"Addressing Leakage in Concept Bottleneck Models\", 2022.\n\n  Marconato et al. \"GlanceNets: Interpretable, Leak-proof Concept-based Models\", 2022.\n\nA brief comment on IntCEMs vs leakage would be welcome - this is an important issue.  2) Robustness to interventions can prevent leakage - the link is that the model is specifically trained to be robust against replacing a (possibly leaky) concept prediction with a (presumably non-leaky) concept annotation.\n\nQ2. Why did you compare only against CEMs and CBMs?  There are other self-interpretable models out there, like the two aforementioned works, part-prototype networks, concept whitening, self-explainable neural networks, and many others.  They all support interventions.\n\nLIMITATIONS:\nI think so, in section 5 there is a brief discussion on some limitations (e.g. time complexity).  The issue of concept leakage is not mentioned though.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}