Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?
Authors: Sonia Laguna
Ričards Marcinkevičs
Moritz Vandenhirtz
Julia E Vogt
Venue: NeurIPS 2024 poster (2024)
Keywords: interpretability
explainability
concepts
concept bottleneck models
model interventions
healthcare
Primary area: interpretability_and_explainability
TL;DR: We introduce concept-based interventions for black-box models, formalise the model's intervenability as a measure of intervention effectiveness, and propose a fine-tuning procedure to improve intervenability.

Abstract: Recently, interpretable machine learning has re-explored concept bottleneck models (CBM). An advantage of this model class is the user's ability to intervene on predicted concept values, affecting the downstream output. In this work, we introduce a method to perform such concept-based interventions on *pretrained* neural networks, which are not interpretable by design, only given a small validation set with concept labels. Furthermore, we formalise the notion of *intervenability* as a measure of the effectiveness of concept-based interventions and leverage this definition to fine-tune black boxes. Empirically, we explore the intervenability of black-box classifiers on synthetic tabular and natural image benchmarks. We focus on backbone architectures of varying complexity, from simple, fully connected neural nets to Stable Diffusion. We demonstrate that the proposed fine-tuning improves intervention effectiveness and often yields better-calibrated predictions. To showcase the practical utility of our techniques, we apply them to deep chest X-ray classifiers and show that fine-tuned black boxes are more intervenable than CBMs. Lastly, we establish that our methods are still effective under vision-language-model-based concept annotations, alleviating the need for a human-annotated validation set.

Beyond Concept Bottleneck Models: How to Make Black Boxes Intervenable?
Sonia Laguna; Ričards Marcinkevičs; Moritz Vandenhirtz, Julia E. Vogt Department of Computer Science, ETH Zurich, Switzerland
Abstract
Recently, interpretable machine learning has re-explored concept bottleneck models (CBM). An advantage of this model class is the user's ability to inter- vene predicted concept values, affecting the downstream output. In this work, we introduce a method to perform such concept- badeept-based interventions on pretrained neural networks, which are not interpretable by design, only given a small valida- tion with concept labels. Furthermore, we formalise the notion intervenability measure of the effectiveness of concept-based interventions and leverage this definition fine-tune black boxes. Empirically, we explore the intervenability of black-box classifiers synthetic tabular and natural image benchmarks. We focus on backbone architectures of varying complexity, from simple, fully connected neural nets to Stable Diffusion. We demonstrate that the proposed fine-tuning im- proves intervention effectiveness and often yields better-calibrated predictions. To showcase the practical utility of our techniques, classifiers and show that fine-tuned black boxes are more intervenable than CBMs. Lastly, we establish that our methods still effective under vision-language- model-based concept annotations, alleviating the need for a noman-annottan validation set.
apply them deep chest X-ray
Introduction
Molnar 2022 have seen renewed interest in concept-based predictive models and approaches to post hoc explanation, such as concept bottlenecks (Lampert et al.) 2009 Kumar et al. 2009 Koh et al.) 2020), contextual semantic interpretable bottlenecks (Marcos et al. 2020), concept whitening layers CCen et al., 2020), and concept activation vectors (B. Kim et al. 2018). Moving beyond interpretations defined in
Interpretable and explainable machine learning (Doshi- Velez Kim 2017
high-dimennional,, unwieldy input space, these techniques relate the model's inputs and outputs via additional high-level suman-understandabll attributes, also referred to as concepts. Typically, neural network models supervised to predict these attributes in dedicated bottleneck layer, post hoc explanations are derived to measure the model's sensitivity to concept variables.
This work focuses specifically the concept bottleneck models, revisited by Koh et al. (220). In brief, CBM is neural network consisting of successive concept and target prediction modules, where the final output depends on the input solely through the predicted concepts. Such models are trained on labelled data, in addition, annotated by attributes. At inference time, human user may interact with the CBM by editing the predicted concept values, which, result, affects the downstream target prediction. This act of model editing is known as an intervention. The user's ability intervene is compelling advantage of CBMs over other interpretable model classes, in that the former allows for human-model interaction.
"Equal contribution. Correspondence sonia. www.aaacleaaaaieeoooonnntt ethz.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In contrast to previous works (Yuksekgonul et al. 2023; Oikarinen et al. 2023). we focus on instance- specific interventions, i.e. performed individually for each data point. To this end, we explore two questions: (i) Given small validation set with concept labels, how can we perform instance-specity interventions directly on pretrained black-box model? (ii) How can we fine-tune the black-bo -box model to improve the effectiveness of interventions performed on it?
prediction
M
L.00
5l
0.00
0.00
1.00
probing
č 0.08
c
1.00
updated
Such spericcc interventions can be relevant
1= 0.91
19
J= 0.00
prediction
in high-stakes decisions. Our specific motivation is
K ] 0.03
cl & }= 1.00
healthcare. For instance, consider computer-aided diagnosis, where a doctor may make decisions as- sisted by a predictive model. this setting, the doctor handles patients on case-by-case basis and may ben- efit from instance-speccfic interactions with the black box. While, in principle, specialist may just over- ride predictions, in many cases, concept and target variables linked via nonlinear relationships poten- tially unknown the user. Figure contains simpli- fied, intuitive example of an instance-specific concept- based intervention for natural images. Additional and more comprehensive examples can be found in
é 1 0.09
c[ ]= 1.00
intervention
A simplified, intuitive example: an image of a grizzly bear is wrongly identified as otter. Our method allows performing concept- based intervention and flip the pre- dicted class. In order of appearance from left to right and top to bottom, the depicted con- cepts and classes are "fierce", "timid", "mus- cle", "walks", "otter", and "grizzly bear".
Figure 1:
Appendix A
This work contributes to the research on concept bottleneck models and concept- based explanations. (1) We devise a simple procedure that, given a set of concepts and small labelled validation set, allows performing concept-basee based instance-specific interventions (Figure T a pretrained black-box neural network by editing its activations at an intermediate layer. Notably, concept labels not required in the large training set, and the network's architecture does not need to be adjusted. (2) We formalise intervenability as a measure of the effectiveness of interventions loss, introduce novel fine-tuning procedure. This fine-tuning strategy is designed to improve the effectiveness of concept-based interventions. It preserves the original model's architecture and representations to be used in downstream tasks. (3) We evaluate the proposed procedures alongside several baselines on the synthetic tabular, natural image, and medical imaging data. We demonstrate that practice, for studied classification problems, we can improve the predictive performance of pretrained black-box models via concept-based interventions. We investigate fully connected and more complex backbone architectures. We show that the effectiveness of interventions improves considerably when explicitly fine-tuning for intervenability. Lastly, we observe that our methods are successful in datasets where concept labels are acquired using vision-language models (VLM), alleviating the need for human annotation.
Contributions
performed on the model. Utilising intervenability
2 Related Work
The use of high-level attributes in predictive models has been well-explored in computer vision (Lampert et al., 2009. Kumar al. (2009). Recent efforts have focused on explicitly incorporating concepts in neural networks (Koh et al. 2020 Marcos et al.] (020)), producing high-level post hoc explanations by quantifying the network's sensitivity to the attributes (B. Kim et al. 2018), y. probing (Alain & Bengio 2016 Belinkov 2022) and de-correlating and aligning the network latent space with concept variables (Chen et al. 2020). Other works (Xie et al., 2020) have studied the use of auxiliary external attributes in out-of-distribution settings. To alleviate the assumption of being given interpretable concepts, some have explored concept discovery prior to post hoc explamation (Ghorbani et al. 2019: Yeh et al. 2020). Another relevant line of work investigated concept- based counterfactual explanations (CCE) (Abid et al. 2022 S. Kim et al. 2023).
Concept bottleneck models (Koh et al. 2020) have sparked renewed interest in concept-based classification methods. Many related works have described the inherent limitations of this model
class and attempted to address them (Margeloiu et al. 2021 Mahinpei et al. 202 Marconato et al.

---

2022, Havasi et al. 2022, Sawada & Nakamura, 2022 Marcinkevics et al. 2024). Another line of research has investigated modelling uncertainty and probabilistic extensions of the CBMs (Collins et al. 2023 H. Kim et al., 2023). Most related to the current paper are the techniques for converting pretrained black-box neural networks into CBMs post hoe (Yuksekgonul et al., 2023 Oikarinen et al. 2023) by keeping the network's s backbone and projecting its activations into the concept space. Additionally, these works explore automated concept discovery using VLMs.
As mentioned, CBMs allow for concept-based instance-specifie interventions. Several follow-up works have studied interventions in further detail. Chauhaan et al. (2023) and (Sheth al. (2022) introduce adaptive intervention policies to further improve the predictive performance of the CBMs the test time. In similar vein, Steinmann et al. (2023) propose learning detect mistakes in the predicted concepts and, thus. learning intervention strategies. [Shin et al. (2023) empirically investigate different intervention procedures across various settings.
3 Methods
In this section, we define a measure for the effectiveness of concept-based interventions and present technique for intervening black-box neural networks. Furthermore, propose fine-tuning procedure to improve the effectiveness of such interventions. Additional remarks beyond the current scope included in Appendix C
In the remainder of this paper, we will adhere the following notation. Let a E X, y E y, and e E C. be the covariates, targets, and concepts. A CBM fe. parameterised by 0, is given by fo (a) = By (he (æ)). where he X + maps inputs to predicted concepts, i.c. è = h& (æ). and Pi C + y' predicts the target based on è, i.e. ý 9v (2). CBMs are trained on data points (xz, c, v) and are supervised by the concept and target prediction losses. At test time, if the user chooses to replace& with another c' C, i.e. intervene, the final prediction is given by j/ 96 (c').
Next to CBMs, we will consider black-box neural network fo X' + parameterised by Q and slice (ge. hτ) (Leino et al., 2018). defining a layer, s.t. So (ar) De (ho (a:)). We will assume that the black box has been trained end-to-end on the labelled data {(rrs y:) Lastly, for all techniques, we will assume being given small validation set { C42 y:)
3.1 Intervening on Black-box Models
Given black-box model fe and a data point (x,, y). human user might desire influence the pre- diction 9 = fe (x) made by the model via high-level and understandable concept values c', e.g. think of doctor trying interact with chest X-ray classifier (fo) by annotating their findings (c') in radiograph (ar), where findings correspond to the clinical concepts, such as the presence of edema propose simple recipe for concept-based instance-
fracture. To facilitate such interactions, specific interventions (detailed in Figure(2 that can be applied to any black-box neural network model. Intuitively, using the given validation data and concept values, our procedure edits the network's representations z = he (z), where z E Z, to align more closely with c' and, thus, affects the downstream prediction. Below, we explain this procedure step-by-step. Pseudocode implementation can be found as part of Algorithm B. in Appendix B
Step 1: Probing
Step 2: Editing Representations
Step 3: Updating Output
ground-truth concepts prodicted concepts m 9g(4)
intervention Intervened concepts (x("")
iateni space x
decision bounddary dv
updated oulput
output 9
activation vector
decis on boundary of
activation vedior 2'
intervened
Figure 2: Three steps of the intervention procedure. (i) A probe ge is trained to predict the concepts from the activation vector z. (ii) The representations are edited according to Equation T (iii) The final prediction is updated to i' based on the edited representations z'.
3

---

Step 1: Probing To align the network's s activation vectors with concepts, the preliminary step is to train a probing function (Alain & Bengio 2016; B. Kim al., 2018 Belinkov 2022), or probe for short, mapping the intermediate representations to concepts. Namely, using the given annotated validation data {( Ct2 y: }, we train a multivariate probe qg to predict the concepts e; from the representations 2, h (z:): ming E, - (4E (2+) C4), where Le is the concept prediction loss. Note that, herein, an essential design choice explored in our experiments is the (non (non)linearity of the probe. Consequently, the probing function can be used to interpret the activations in the intermediate layer and edit them.
Step 2: Editing Representations Recall that we are given a data point (ar y) and concept values c' for which an intervention needs to be performed. Note that this c' E could correspond to the ground-truth concept values or reflect the beliefs of the human subject intervening
the model. Intuitively, we seek an activation vector z', which is similar to 2 he (a)) and consistent with c' according to the previously learnt probing function q:: arg min=' d (2,22), s.t. 9g (z') c', where is an appropriate distance function applied to the activation vectors from the intermediate layer. Throughout main experiments (Section 4. utilise the Euclidean distance, which is frequently applied to neural network representations, e.g. see works by Moooadi Fard et al. (2020) and Jia et al. (202T). In Appendix F88 we additionally explore the cosine distance. Instead of the constrained problem above, we resort to minimising relaxed objective:
arg
argmin XC (gg (z'), c') + d dz,z))
x'
where, similarly to the counterfactual explanations (Wachter et al. 2017 Mothilal et al. 2020), hyperparameter À > 0 controls the tradeoff between the intervention' validity, í.e. the "consistency" of x' with the given concept values c' according to the probe, and proximity to the original activation vector In practice, optimise z' for batched interventions using Adam (Kingma & Ba 2015). Appendix F22 explores the effect of on the post-intervention distribution of representations.
The edited z' can be consequently fed into gy compute the updated output v' = I6 (z'), which could be then returned and displayed to the human subject. For example, if c' are the ground-truth concept values, we would ideally expect decrease in the prediction error for the given data point ().
Step 3: Updating Output
3.2 What is Intervenability?
Concept bottlenecks (Koh et al., 2020) and their extensions are often evaluated empirically by plotting test-set performance or error attained after intervening on concept subsets of varying sizes. Ideally, the model's test-set performance should improve when given more ground-truth attribute values. Below, we formalise this notion of intervention effectiveness, referred to as intervenability, for the concept bottleneck and black- models.
For trained CBM fe (x) = I6 (he (z)) = I6 (é), where è are the predicted concept values, we define the intervenability as follows:
(p ( (c'), v) [
E
E c'
L
fo (x) u) L
(D.
(e)
where is the joint distribution over the covariates æ. concepts c, and targets y, L is the target prediction loss, e.g. the mean squared error (MSE) or cross-entropy (CE), and denotes distribution over edited concept values c'. Observe that Equation generalises the standard evaluation strategy random concept subset and setting
of intervening the original work by Koh et al. (220)). Here, the effectiveness interventions is quantified by the gap between the regular prediction loss and the loss attained after the intervention: the larger the gap between these values, the stronger the effect interventions have. The intervenability measure is loosely related to permutation-bast based variable importance and model reliance (Fisher et al. 2019). We provide a discussion of this relationship in Appendix [
the ground-truth values, as proposed in
Note that the definition in Equation 2 can also accommodate more sophisticated intervention strate- gies, for example, similar to those studied by [Shin et al. (2023) and [Sheth et al.) (2022). An intervention strategy can be specified via the distribution π, which can be conditioned è, c, 9.

---

or even y: т (c' x, è, G. y). The set of conditioning variables may vary across application scenarios. For brevity, we will use π as a shorthand notation for this distribution. Lastly, notice that, in practice, when performing human- or application-rouundee evaluation (Doshi-Velez & Kim 2017), sampling from иT may be replaced with the interventions by human. Algorithms examples of the strategies utilised in our experiments.
and provide concrete
Leveraging the intervention procedure described in Section 3.1 analogous to Equation 2 the inter- venability for a black-box neural network fe at the intermediate layer given by (9w, h
((r.c..)-DDe' c'-xx [C" (fo (a:) L (o+ (z)),)),
where z' arg min A (9g (2) c') (z)
Recall that qg is the probe trained to predict e based on the activations hs (æ:) (step Section 3.1 Furthermore, the first line of Equation 3] edited representations z' are function of c', as denned by the second line, which corresponds to step 2 of the intervention procedure (Equation
3.3 Fine-tuming for Intervenability
Since the intervenability measure defined in Equation 3 is differentiable, a neural network can be fine-tuned by explicitly maximising it using, for example, mini- mini-batch gradient descent. We expect fine-tuning for intervenability to reinforce the model's I reliance on the high-level attributes and have regularising effect. In this section, we provide detailed description of the fine-tuning procedure (Algorithm B.1 Appendix
B and, afterwards, we demonstrate its practical utility empirically.
Naïvely optimising intervenability from Equation 3 may decrease the predictive performance. There- fore, to fine-tune an already trained black-box model fa. we combine the intervenability term with the target prediction loss, which amounts to the following optimisation problem:
min E (1,,o.)) e' f 9 (g) (z') vv) s.t. z' aag XL (ge (z), d 22, (4)
Notably, Equation can be generalised by introducing a weight for the intervenability term:
min E (mmow)DD e' (1 2y d.
(gx / (ne (z)) .)) + BC 9 (2').
(5)
s.t. z' E arg XL (gg c') + (z,
where 8 E (0, 1] is the aforementioned weight. Note that for 3 = 1. the optimisation simplifies to Equation 4 For simplicity, we treat the probe's parameters & as fixed. However, since the outer optimisation problem is defined w.r.t. d, ideally, the probe would need to be optimised as the third, inner-most level. By contrast, in the simplified setting under 3 (Equation 4), the parameters of h do not need to be optimised, and, hence, the probing function can be left fixed, as activations are not affected by the fine-tuning. We consider this case (i) computationally simplify the problem, avoiding trilevel optimisation, and (ii) keep the network's 's representations unchanged after fine-tuning for purposes of transfer learning for other downstream tasks. In practice, fine-tuning is performed by intervening on batches of data points. Since interventions can be executed online using GPU (within seconds), our approach is computationally feasible.
4 Experimental Setup
Datasets We evaluate the proposed methods on synthetic and real-world benchmarks summarised in TableeD.1 (Appendix D). Across all experiments, fine-tuning has been performed exclusively on the validation data, and evaluation-on on the test set. Further details can be found in Appendix D
For controlled experiments, we have adapted the nonlinear synthetic tabular dataset from Marcinke viés et al. (2024). We consider two data-generating mechanisms shown in Figure D.I (Appendix [. bottleneck, and incomplete. The first scenario directly matches the inference graph of the vanilla CBM. The incomplete is scenario with incomplete concepts, where e does not fully explain the association between æ and y, with unexplained variance modelled via residual connection.
5

---

Another benchmark we consider is the Animals with Attributes (AwA2) natural image dataset (Lampert et al., 2009; Xian al. 2019). It includes animal images accompanied by 85 binary attributes and species labels. To further corroborate our findings. we perform experiments on the Caltech-UCSD Birds-200-2011 (CUB) dataset (Wah et al. 201 1) (Appendix (.3). adapted for the CBM setting as described by Koh et al. (2020). We report the CUB results in Appendix F.4
To investigate settings without human-annotater concept values, we evaluate our method on CIFAR-10 (Krizhevsky et al., 2009) and the large-scale ImageNet (Russakovsky et al., 2015) natural image datasets. Following the previous literature (Oikarinen et al. (2023). we use concepts generated by GPT-3. Concept labels are produced based on CLIP (Radford et al. (021) similarities between each image and verbal descriptions. We utilise 143 attributes for CIFAR- 10 and 100 for ImageNet.
ImageNet results are reported in Appendix F.5
Finally, we explore practical setting of chest radiograph classification. Namely, test the techniques on public MIMIC-CXR (Johnson et al. (019)) and CheXpert (Irvin et al., 2019) datasets from the Beth Israel Deaconess Medical Center, Boston, MA, and Stanford Hospital. Both datasets have 14 binary attributes extracted from radiologist reports. In our analysis, the Finding/No Finding attribute is the target variable, and the remaining labels are the concepts, similar to Chauhan al. (2023), For simplicity, we retain a single X-ray per patient, excluding data with uncertain labels. The results on CheXpert are similar to those on MIMIC-CRR and can be found in Appendix F.6
Below, we briefly outline the neural network models and fine-tuning techniques compared. All methods were implemented using PyTorch (v 1.12.1) (Paszke el al. 2019). Appendix [E] provides additional details. The code is available in a repository at https:///ithub.... http://////onaaaggggaaa/////oo//
Baselines & Methods
Firstly, we train a standard neural network (BLACK BOX) without concept knowledge, í.e. on the dataset of tuples {<,,,,:));- We utilise our technique for intervening post hoc by training a probe to predict concepts and editing the network's activations (Equation [ Section 3.1). All experiments reported in Section [ use linear probe, while the nonlinearity is explored in Ap- pendix F. As an interpretable baseline, we consider the vanilla concept bottleneck model (CBM) the joint bottle-
by Koh et al.] (2020). Across all experiments, we restrict ourselves neck version, which minimises the weighted sum of the target and concept prediction losses: ming. (.))).)) [C" (fe (z) .)) aL (ha
where a > O hyperparameter control- ling the tradeoff between the two loss terms. Finally, as the primary method of interest, we apply our fine-tuning for intervenability technique (FINE-TUNED, I; Equation 4 Section 3.3) on the annotated validation set { C. b:)
In addition, as a common-sense baseline, we fine-tune the black box by training probe to predict the concepts from intermediate representations (FINE-TUNED, MT). This amounts to multitask (MT) learning with hard weight sharing (Ruder) (017)). Specifically, the model fine-tuned by minimising
the following MT loss: ming. g i (tte,c.y)~-1 L (fe (æ), y) + aL (ge (h (x)) c)]. Interventions on this model are performed using the three-step approach introduced in Section 3.1
As another baseline, we fine-tume mune the black box by appending concepts to the network's activa- tions (FINE-TUNED, A). At test time, unknown concept values set to 0.3. To prevent overfitting and handle missingness, randomly chosen concept variables are masked during training. The ob- jective is given by min, ((..e.))DD("* (āg (hh (æ), c)), y)], where ğ takes input concatenated activation and concept vectors. Note that, for this baseline, the parameters remain fixed during fine-tuning.
Last but not least, strong baseline resembling the approaches by Yuksekgonul et al. (2023) and Oikarinen et al. (2023), we train a CBM post hoc (POST HOC CBM) using sequential optimisa- tion. Our implementation follows the original methods by Yuksekgonul et al. (2023) and Oikarinen et al. (2023). while adjusting some design choices to make the techniques more readily compa- rable. The optimisation comprises two steps: (i) Ç = arg ming (tt)e..)>>>ll |C (4g (he (x)) c)], ()))))))) v)). Additionally, we explore the impact of residual mod- elling (Yuksekgonul et al. 2023) in Appendix F9 The architectures of individual modules were kept as similar as possible for a fair comparison across all techniques.
(ii) ming E(g.c.y)~ [ (ge (ge((h (x)))

---

Evaluation To compare the methods, we conduct interventions and analyse model performance under varying concept subset sizes. We report the areas under the receiver operating characteris- tic (AUROC) and precision-recall curves (AUPR) (Davis & Goadrich 2006) since these performance measures provide well-rounded summary over varying cutoff points and might be challenging to choose single cutoff in high-stakes decision areas. We utilise the Brier score (Brier 1950) to gauge the accuracy of probabilistic predictions and, in addition, evaluate calibration.
5 Results
Results on Synthetic Data Figures ff and aa) show intervention results obtained across ten independent simulations under the two generative mechanisms (Figure D.I] Appendix [.I] on the synthetic tabular data. We observe that, in principle, the proposed intervention procedure can improve the predictive performance of a black-box neural network. However, in the bottleneck scenario, interventions are considerably more effective in CBMs than in untuned black-box classifiers since the underlying generative process directly matches the CBM's architecture. Models explicitly fine-tuned for intervenability (FINE-TUNED, I) significantly improve over the original classifier, achieving intervention curves comparable to those of the CBM.
Importantly, under an incomplete concept set, black-box classifiers are superior to the ante hoc CBM because not all concepts relevant to the target prediction are given. Moreover, fine-tuning for intervenability improves intervention effectiveness while maintaining the performance gap. This experiment suggests the superiority of our method settings where the concept set does not capture all label-relevant information. Other fine-tuning strategies (FINE-TUNED, MT and FINE-TUNED, A) are either less effective or harmful, leading to a lower increase in AUROC and AUPR than attained by the untuned black box. Lastly, CBMs trained post hoc perform well in the simple bottleneck scenario, being only slightly less intervenable than FINE-TUNED, I. However, for the incomplete setting, interventions hurt the performance of the post hoc CBM. This behaviour may be related to the leakage (Havasi et al. 2022) and is not mitigated by residual modelling explored in Appendix F.
To study the influence of the validation set size (N,ai) on probing and fine-tuning, we perform ablations under the bottleneck scenario (Figure 3) For fair comparison w.r.t. sample efficiency, here, we train CBM on the dataset of the same size. While the effectiveness of interventions on FINE-TUNED, I slightly hampered by smaller validation sets, the decrease moderate. We observe impactful interventions with validation set sizes as small as 0.5% of the original one (Figure Ba)). Across all settings, our method remains superior to baselines. Importantly, our fine-tuning approach has
0.75
0.
0.3
9.65
0.7
0.60
0.7
0.55
0.6
20
10
60
Eo
100
20
40
60
20
100
60
m.concepls Intervened on
(a) Nost 50 (0.5%)
(c) Naal = 100 (%)
(e) Noa 500 (5%)
0.5
0.9
0.4
0.8
0.11
0.7
0.6
20
40
150
20
40
60
80
100
60
80
120
concepts Intercoeedd
concests mntreeee
Ccnorcts Intervened
(b) Nai = 1000 (10%)
(d) Nyal = 5000 (50%) (f) Na 10000 (100%)
Black box
CBM (full)
CBM (val)
Post hoc CBM
Fine-tuned, MT
Fine-Juned,A A
Fine-tune__1
Figure 3: Intervention results w.r.t. target AUROC on the synthetic bottleneck data. We explore the
performance under varying validation set sizes (Naa). Percentages correspond to the fractions of the original validation set. For CBMs, we report the results obtained by training on the validation (CBM val) and full training sets (CBM full). Interventions were performed on test data across ten simulations. Lines correspond to medians, and confidence bands are given by interquartile ranges.
7

---

2779
1000
0.95
1750
0.90
2795
2230
0.9000
0.83
1425
0.80
2.50
09902
0.615
0.75
92
100
100
0.9
0.75
0.83
0.8
0.0
5
0.0
0.91
0.7
0.5
20
120
concasta Plenerad
concepts Intrvateed on
concepts Intervense
%6 concapts Inaevered on
(a) Synthetic incomplete
(b) AwA
(c) CIFAR-10
(d) MIMIC-CXR
Black box
CBM
Post hoc CBM
Fine-tuned, MT
Fine-tuned, A
Fine-tuned.
Figure 4: Intervention results the (a) synthetic incomplete, (b) AwA2, (c) CIFAR-10, and (d) MIMIC-CXR datasets w.r.t. target AUROC (top) and AUPR (bottom) across ten seeds.
better sample efficiency than post hoc CBM, which exhibits a considerable dropoff in intervention effectiveness. Likewise, the performance of the CBMs decreases, suggesting their limited utility under smaller dataset sizes. Analogous results w.r.t. AUPR are reported in Appendix F.
In Table we report the test-set performance of the models without interventions (under the bottleneck mechanism). For the concept prediction, CBM outperforms black-box models, even after fine-tuning with the MT loss. However, without interventions, all models attain comparable AUROCs and AUPRS at the target prediction. Interestingly, FINE-TUNED, I results in better-calibrated probabilistic predictions with lower Brier scores than those made by the original black box and after applying other fine-tuning strategies. As evidenced by Figure F9 a) (Appendix F.7). fine-tuning has regularising effect, reducing the false overconfidence observed in neural networks (Guo et al., 2017).
In the supplementary material, we report several additional findings. Figure F2 (Appendix F.1 contains further ablations for the intervention procedure on the influence of the hyperparameters. intervention strategies, and probe. In addition, Appendix F.2 exxlorres the effect of interventions on distribution representations. Lastly, Appendix F.10 we show that the performance of the CBM on this dataset is not sensitive to the choice of the optimisation procedure Koh et al. 2020).
Results AwA2 Additionally, we explore the AwA2 dataset in Figure 4b). This is simple classification benchmark with class-wide concepts helpful for predicting the target. Hence, CBMs trained ante and post hoc highly performant and intervenable. Nevertheless, untuned black-box models also benefit from concept-base based interventions. agreement with our findings on the synthetic dataset and in contrast to the other fine-tuning methods, ours enhances the performance of black- box models. Notably, black boxes fine-tuned for intervenability even surpass CBMs. Overall, the simplicity of this dataset leads the generally high AUROCs and AUPRs across all methods.
To further investigate the impact of hyperparameters on the interventions, we have performed ablation studies on untuned black boxes. These results are reported in Figures F.4] (c)(c,, Appendix F.3 In brief, we observe that interventions are effective across all values of the X-parameter from Equation 3 (Figure (a). Expectedly, higher values yield a steeper increase in AUROC and AUPR. Figure (.4(b) compares two intervention strategies; randomly selecting concepts (random) and prioritising the most uncertain ones (uncertainty) (Shin et al. 2023) to intervene (Algorithms E.] and E.2 Appendix E). The strategy has an impact on the performance increase, with the uncertainty- based approach yielding steeper improvement. Finally, Figure F.4 c) compares linear and nonlinear probes. Here, intervening via nonlinear function leads to a higher performance increase.
To show the efficacy of our methods across different backbone architectures, in Appendix F3 we also explore AwA2 with the Inception (Szegedy al.. 2015) backbone (note that Figure 4 b)) reports the results on the ResNet-18 (He et 201 Finally. time without interventions for target and concept prediction. We observe comparable performance across methods, which all successful due to the large dataset size and the task simplicity.
lilllyy... Table contains evaluation metrics at test

---

Table 1: Test-set concept and target prediction performance without interventions. For black boxes, concepts were predicted via a linear probe. Results are reported as averages and standard deviations across ten seeds. For concepts, performance metrics were averaged. Best results are reported in bold, second best are in italics.
Concepts
Target
Dataset
Model
AUROC
AUPR
Brier
AUROC
AUPR
Brier
BLACK BOX CBM
0.716.10.018
0.710:10.017
0.008
0.686.10.043
0.675:10.046
0,460.10.003
Synthodic
0.807:10.008
0.835:10.008
0.196:10.006
0.713:20.040
0.700-10.038
0.410:100012
POST CBM
0.714.10.017
0.707100.018
0.207
0,707
0.608
0.0015
FINE-TUNED, A
0.682=00004
0.668:10:046
0.07
FINE-TUNED, MT
0.784::0003
0.780=00.014
0.186=0.006
0.687=0.046
0.668+00.043
0.711=0.003
FINE-TUNED.
0.716:100018
0.710:20.917
0.208=000006
0.655
0.685=50.051
0.205
BLACK BOX CBM
0.991-10.002
0.9791
0.027.10.006
0,996 0.000
0.926.10.020
0.199.0..038
0,903-10.001
0.979-10.002
0.025:--0000
0.908
0.892:1:0005
0.234
AWA2
POST HOC CBM
0.992.100.022
0.976:10.005
0.025:10.055
0.9961.0.000
0.929:1..018
0.170:000033
FINE-TUNED, A
0.9961.0000
0.938 100011
0.170
FINE-TUNED, MT
0.994::0.002
0.985
0.022:10.005
0.907=0.001
0.908
0.
FINE-TUNED.
0.991=:0.002
0.979:00000
0.027=000006
0.996=0000/
0.925=00.020
0.195=00.000
BLACK BOX CBM
0.713:1:0.022
0.0021000001
0.110:1:0000
0.879100001
0.504::0.004
0.920:10.006
MIMICCCR CIFAR-10
POST HOC CBM
0.675::0.009
0.785:200003
0.125=00.004
0.00
0.541:10.004
0.604
FINE TUNED, A
0.876110.002
0.508
0.896.0.0005
FINE-TUNED, MT
0.807-10.001 0.729
0.109-10.00
0.870=00.004
0.512:10.009
0.890-10.014
FINE-TUNED,
0.713-20.002
0. 0.802+0.001
0.110=0.000
0,873=0.003
0.501
0,902=00.021
BLACK BOX CBM
0.743:2:0000
0.170:2:00044
0.046=20.00/
0.709=0.006
0.706=t00009
0.401:00003
0.764:100006
0.204110.003
0.053-10.001
0.705
0.0091
0.427.10.003
POST HOC CRM
0.707
0.1544100006
0.046:10.001
0,801 2.0006
0.727-:0.000
0.001
FINE-TUNED, A
0.773+200009
0.665-10.013
0.459:10004
TINE-TUNED MT
0.748:10.008
0.187:1:0003
0.045:10.01
0.785 0.0006
0.696-10.009
0.450.10.008
FINE-TUNED,
0,744::0.005
0.172::0.005
0.046:00001
0.008
0.733-10.009
0.3142:0.115
Results with VLM-baseed Concepts To demonstrate that our approaches are effective without human-annotated concepts YYuksekgonul al. 2023] Oikarinen et al. (023), we present the results on CIFAR-IO in FigureFfc.. Here, concept labels were generated using a VLM. In addition, we explore the ImageNet in Appendix [F.) The results have been obtained using the backbone architecture of Stable Difusion (Rombach et al. 2022).. We do not include the CBM, as we cannot retrain such a large backbone due to computational constraints. By contrast, our method allows fine-tuning the pretrained network, thus being helpful where a CBM is impractical. As in the previous experiments, black boxes are, in principle, intervenable, and our fine-tuning men-tunng approach outperforms other baselines.
Application to Chest X-ray Classification To showcase the practicality of our approach, we present empirical findings on two chest X-ray datasets, MIMIC-CXR and CheXpert, primarily focusing on the former. Figure [ shows intervention curves across ten independent initialisations. Interestingly, untuned black-box neural networks are not intervenable. By contrast, after fine-tuning for intervenability. the model's predictive performance and effectiveness of interventions improve visibly and even surpass those of the CBM. Given the challenging nature of these datasets, black-box model predictions may not be as strongly reliant on the considered attributes. Moreover, CBMs do not necessarily outperform black-box networks, unlike in simpler benchmarking datasets. Finally, post hoc CBMs (even with residual modelling) exhibit a behaviour similar to the synthetic dataset with incomplete concepts: interventions have only a slight positive, no, adverse effect performance. Analogous findings for CheXpert can be found in Appendix [F6]
6 Discussion & Conclusion
This work introduces a technique for performing instance-specific concept-based interventions on any pretrained neural network post hoc. We formalise a novel measure of intervenability as the effectiveness of concept- based interventions and propose a method leveraging to fine-tun tine-tune black-box models. In contrast to CBMs (Koh et al.) 2020)) our method circumvents the need for concept labels during training, which can be a challenge in practical applications. Unlike recent works on converting black boxes into CBMs post hoc (Yuksekgonul et al. 2023 Oikarinen et al.) 2023), which generally do not explore instance-specific interventions, we propose an effective intervention method that faithful to the original architecture and representations. Lastly, we introduce and study several other

---

sense fine-tuning baselines that perform worse than the proposed method, highlighting the need for the explicit maximisation of intervenability.
The utility of our method highlighted empirically on synthetic tabular and natural image data. We show that, given small annotated validation set, black-box models trained without explicit concept knowledge are intervenable. Moreover, our fine-tuning method improves the effectiveness of the interventions, with overall better results than alternative techniques. In addition, our approach is effective in scenarios where the concept labels are generated using VLMs. Thus, we can alleviate the need for costly human annotation while maintaining improved intervention effectiveness. Lastly, we apply the techniques in a more realistic setting of chest X-ray classification, where black boxes are not directly intervenable. The proposed fine-tuning procedure alleviates this limitation, while the other post hoc techniques are ineffective or even harmful.
Limitations & Future Work Our work opens many avenues for future research and improvements. Firstly, the variant of the fine-tuning procedure considered in this paper does not affect the neural network's representations. However, it would be interesting to investigate the more general formula- tion wherein all model and probe parameters are fine-tuned end-to-end. According to our empirical findings, the choice intervention strategy, hyperparameters, and probing function can influence the effectiveness of interventions. A deeper experimental investigation of these aspects is warranted. Furthermore, we only considered a single fixed intervention strategy throughout fine-tuning, whereas further improvement could come from learning optimal strategy alongside fine-tuned weights. Beyond the current setting, we would like to apply our intervenability measure to evaluate and compare other large pretrained discriminative and also generative models.
Acknowledgments and Disclosure of Funding
MV and SL are supported by the Swiss State Secretariat for Education, Research, and Innova- tion (SERI) under contract number MB22.00047.
10

---

References
Abid, A., Yuksekkonul, M., & Zou, J. (2022). Mcaningfully debugging model mistakes using conceptual counterfactual explanations. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, & S. Sabato (Eds.), Proceedings of the 39th international conference on machine learning (Vol. 162, pp. 66-88). PMLR. Retrieved from https://prooceedings mlr www.pprsssseeee v162/ abid22a. html
Alain, G., Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes.
Retrieved from atttps://don org/10. 48550/arXiis 1610 01644
Belinkov, Y. (2022). Probing Classifiers: Promises, Shortccomings, and Advances. Computational
Linguisttics, 48(1), 207-219. Retrieved from htttp:://doi.org///e/eo 1tt.:/..111cooooo/1 ctt//oolla__0022
Breiman, L. (2001). Random forests. Machine Learning, 45(1), 5-32. Retrieved from https:// //
doi. 200000100/::111000000000000
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly weather (1175//152200088119))000011101
review, 78(1), 1-3. Retrieved from https://doi.
VOFEIT>2.0 2.0.c022
Chauhan, K., Tiwari, R., Freyberg. J., Shenoy, P., & Dvijotham, K. (2023). Interactive concept bottleneck models. Proceedings the the AAAI Conference on Artificial Intelligence. 37(5), 5948- 5955. Retrieved from htttps://ojs..... aaai: ooooooognioo index. http://///aattle/viee///////////
Chen, Z., Bei, Y., Rudin, C. (2020). Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12), 772-782. Retrieved from attpp::/dde oegiiiorg/1000 1trg/10.1033
s42256 11222660200-000 020-00265-2
Collins, K. M., Barker, M., Zarlenga, M. E., Raman, N., Bhatt, U., Jamnik, M.,... Dvijotham, K. (2023). Human uncertainty in concept-based ai systems. Retrieved from htttps:/ org/ 10. 48550/ arXiv 2303 12872 (arXiv:2303.
Cramér, H. (1999). Mathematical methods of statistics (Vol. 26). Princeton University Press.
Davis, J., & Goadrich, M. (2006). The relationship between precision and ROC curves. In Proceedings of the 23rd international conference on machine learning (p. 233--240). New York. NY. USA: Association for Computing Machinery. Retrieved from https: //doi. org/10. 1145/
1143844. 1143874
Velli- F,, & Kim, B. (2017). Towards rigorous science of interpretable machine learning.
Retrieved from https://doi org/10 10. att:0//rXiv.110. 08608 (arXiv: (arXiv:/702.er 702.08608)
Fisher, A., Rudin, C., Dominici, F. (2019). All models are wrong, but many are useful: Learning a variable's 's importance by studying entire class of prediction models simultancously. Journal of Machine Learning Research, 177), 1-81. Retrieved from http: //mmr./mm.r////// org/papers/v20/
18- 23-760.html html
Ghorbani, A., Wexler, J., Zou, J. Y. & Kim, B. (2019). Towards automatic concept- based explanations. Fox, R. Garnett (Eds.),
In H. Wallach, H. Larochelle, Beygelzimer, F. d'Alché-But Advances in neural information processing systems (Vol. 32, 9277 9277-9286). Curran Associates, Inc. Retrieved from httpp:::/pproceeingg.... neurips. cc/ hwwp//pppr.... fwt.ss/pappee//199///e5a11 7702afrc3116663333500aa111116862
.pff
Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q. (2017). On calibration of modern neural networks. In D. Precup Y. W. Teh (Eds.), Proceedings of the 34th international conference on machine
learning (Vol. 70, pp. 1321-1330). PMLR.
Havasi, M., Parbhoo, S., & Boshi-Velez. F. (2022). Addressing leakage in concept bottleneck models. In A. H. Oh, A. Agarwal, D. Belgrave, & K. Cho (Eds.), Advances neural information processing systems. Retrieved from
wwweeeerreiiie.eeefoooieeoe net net//oorumidd-tglnnii. fn9
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In 2016 ieee conference computer vision and pattern recognition (cvpr) (pp. 770-778). Retrieved from https://doi.org::
1109/CVPR. 2016. 90
Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., ... Ng, A. (2019). CheXpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, pp. 590-597). Retrieved from https://
ojs. aaai. .rgg/ 'index. fadex.pppp/e phpp/AAII/a MAA.///rriceeeeiee///BB...

---

Jia, M., Chen, B.-C., Wu, Z., Cardie, C., Belongie, S., & Lim, S.-N. (2021). Rethinking nearest neighbors for visual classification. Retrieved 088499 (arXiv:21 10..11...09499
http:::://aoi.or/1100/ 01.444550/aaii1 2112
Johnson, A. E. W., Pollard, T. J., Berkowitz, J., Greenbaum, N. R., Lungren, M. P., Deng. C.-y..... Horng. S. (2019). MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. Scientific Data, 6(1), 317. Retrieved from httpss://oo org/10..100. /0/ 1038/
841597-019-0322-0
Kim, B., Wattenberg, M., Gilmer, Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2018). Interpretabil- ity beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning (Vol. 80, pp.
PMLR. Retrieved from https://proceeddin// mlr press/ /v80/
kim18d html
Kim, E., Jung, D., Park, S., Kim, S., & Yoon, S. (2023). Probabilistic concept bottleneck models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, J. Scarlett (Eds.), Proceedings of the 40th international conference on machine learning (Vol. 202. pp. 16521-16540), PMLR. Retrieved from https://
mlr, wwwppppessspae Stteess/222///mm222222 html
Kim, S., Oh, J., Lee. S., Yu, S., Do. J., & Taghavi, T. (2023). Grounding counterfactual explanation of image classifiers to textual concept space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR) (pp. (0950..
Kingma, P., Ba, J. (2015). Adam: A method for stochastic optimization. In Y. Bengio & Y. LeCun (Eds.), 3rd international conference on learning representations, ICLR 2015. Retrieved from httt:://arxxt
org/ wtw./////////////eo/o/////eee/// 1412. 6980
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020).
Concept bottleneck models. In H. D. III & A. Singh (Eds.). Proceedings of the 37th international conference on machine learning (Vol. 119. pp. htttps://proceeddng mlr poress/v119//on html
Virtual: PMLR. Retrieved from
Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
Kumar, N. Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In 2009 ieee 12th international conference on computer vision (pp. 365-372). Kyoto, Japan: IEEE. Retrieved from https://dol org/rr../// 10. 1109/ ICCV 2009 5459250
Lampert, C. H., Nickisch, H., & Harmeling, S. (2009). Learning detect unseen object classes by between-class attribute transfer. In 2009 JEEE conference on computer vision and pattern recognition. Miami, FL, USA: IEEE, Retrieved from htttps://doi. org/ 10. 110./ CV-.///VPP.... 2009 .5206594
Leino, K., Sen, S., Datta, A., Fredrikson, M., & Li, L. (2018). Influence-directed explanations for deep convolutional networks. 2018 IEEE international test conference (ITC). IEEE. Retrieved from wtom/http:://// https:
org/ 10. 1109/ tee9///ttt... 2018. 8624792
Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., & Pan, W. (2021). Promises and pitfalls of black box concept learning models. Retrieved from pattts::/doi.. org//0....g..... 48500/ aato///xivv.../ 2106, 13314
(4rXhv:2106.13314)
Marcinkevičs, R., Reis Wolfertstetter, P., Klimiene, U., Chin-Cheong, K., Paschke, A., Zerres, J.,
Vogt, J. F. (2024). Interpretable and intervenable ulraasooorapphy machine learning models for pediatric appendicitis. Medical Image Analysis, 91, 103042. Retrieved from https://
www www.sciencedirectt. wtwcsciioiiceeeeeee sttsssccccccccceeeeeeeee httpa/////...teee//////////aa///
Marconato, E., Passerini, A., & Teso, S. (2022). GlanceNets: Interpretable, leak-proof concept-based models. (arXiv:2205.156) 15612)
Marcos, D., Fong, R., Lobry, S., Flamary, R., Courty, N., & Tuia, D. (2020). Contextual semantic interpretabilitty. In H. Ishikawa, C. Liu, T. Pajdla, & J. Shi (Eds.). Computer vision ACCV 2020 15th asian conference on computer vision, revised selected papers, part IV (Vol. 12625. pp. 351-368). Springer. Retrieved from https: //doi org/ 10. 3000 000-3-0000090-----8
Margeloiu, A., Ashman, M., Bhatt, U., Chen, Y., Jamnik, M., & Weller, A. (2021). Do concept bottleneck models learn as intended? Retrieved from https://do:: org/10 48550/ arXiv .2050. 04289
12

---

Molnar, C. (2022). Interpretable machine learning (2nd ed.). Retrieved from Stttss:/chhissee
.github. io/ wtw.nttrpretableeeeeee.e meme-mmmbbos ml-book
Moradi Fard, M., Thonet, T., & Gaussier. E. (2020). Deep k-menss: Jointly clustering with k-
means and learning representations. Pattern Recognition Letters, 138. 185-192. Retrieved from wttp://cccccaaaccc.ooooooeeeeeee www.c/ooocciiiiiic http://ww....o.oooccaoooooooaaaa
Mothilal, R. K., Sharma, A., & Tan, C. (2020). Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 607-617). New York, NY, USA: Association for Computing Machinery. Retrieved from httts:://ddo orgo 10. 1145/3351085
3372850
Oikarinen, T., Das, S., Nguyen, L. M., & Weng, T.-.. (2023). Label-free concept bottleneck models. In The 11th international conference on learning representations. Retrieved from pettps:/opeerrwveee
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury. J., Chanan, G.... Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buch E. Fox, & R. Garnett (Eds.), Advances in neural infor- mation processing systems (Vol. 32). Red Hook, NY. United States: Curran Associates. caaperffles/peeeee patpppppp/r////lll// Inc. Retrieved from httpss:/ newrips....pp... cc/ http/pppe_fieee/ppeeeeeeeeeeeeee pdf
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O.... Édouard Duches-
nay (2011). Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830. Retrieved from httn:ttt:/mmlll
wttg/ppprs//eeeooo httpl//ppppppeaaeeaaeeee 1a. 1tmmmmml
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.,... others (2021). Learning transferable visual models from natural language supervision. In International conference on machine learming (pp. 8748-8763).
Rombach, R., Blattmann, A., Lorenz, D.,, Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).
Ruder, S. (2017). Au overview of multi-task learning in deep neural networks Retrieved from https://dot org/10. 10. 48550/ /01500//rXiv... 1706. 05098 (arXiv:176..0500)
Russakovsky, O., Deng, J., Su, H., Krause, J.. Satheesh, S., Ma, S..... others (2015). ImageNet large scale visual recognition challenge. International joveraaaionnaouralll of computer vision, 115, 211-252.
Sawada, Y., & Nakamura, K. (2022). Concept bottleneck model with additional unsupervised 11w.11109/AACCS concepts. IEEE Access, 10, 41758 41765. Retrieved from hom/https://ddot wtt.://///............111111111. .2022. 3167702
Sheth, I., Rahman, A. A., Sevyeri, L. R., Havaci, M., & Kahou, S. E. (2022). Learning from uncertain concepts via test time interventions. In Workshop trustworthy and socially responsible machine learning, neurips 2022. Retrieved from http::t/oopeeeeviiiiiii
net wtw.:formmddddd/we W
Shin, S., Jo. Y., Ahn, S., & Lee, N. (2023). A closer look at the intervention procedure of concept bottleneck models. In A. Krause, E. Brunskill, K. Cho. B. Engelhardt, S. Sabato, & J. Scarlett (Eds.), Proceedings of the 40th international conference on machine learning (Vol. 202, pp. 31504- 31520). PMLR. Retrieved from httpss:///rooccenngss mlr. www.ppresssvveee wwtps//////////sn23... shwn/3a.nnn..a html
Srivastava, N., Hinton, G., Krizhevsky, A.., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56), 1929-1958. Retrieved from http: org http//ppprss/vvveee aaapprrs/ii/sriivastavaa.. srivttavaaaa. hhmllmmllllmmm.
Steinmann, D., Stammer, W., Friedrich, F, & Kersting, K. (2023). Learning to intervene on concept bottlenecks. Retrieved from https://doi. org/ /0/. 00l.48550/araa 2308. 13453 2.3.::2300..345533
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D.... Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box: Automated decisions and the GDPR. Harvard Journal of Law Technology. 31. Retrieved from https: : //doi. org/10 2139/ http://snn.0000 3063289
13

---

Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). Caltech-UCSD Birds-iiO----- 200-2011. Retrieved from poomattps://authors libraryr......... caltech. ett./reccordd/ccm33-5511111 (Techni- cal report. CNS-TR-2011-001. California Institute of Technology)
Xian, Y., Lampert, C. H., Schiele, B., & Akata, Z. (2019). Zero-shot com evalu- ation of the good, the bad and the ugly. IEEE Transactions Pattern Analysis and Machine Intelli-
gence, 41(9), 2251-2265. Retrieved from httpp://dot orgg 10. www.1...11t11eeee 1109/tpami. 2018. 2857766
Xie, S. M., Kumar, A., Jones, R., Khami, F., Ma, T., & Liang, P. (2020). In-n-out: Pre-training and self- self-training using auxiliary information for out-of-distribution robustness. In International conference on learning representations.
Yeh, C.-K., Kim, B., Arik, S., Li, C.-I.., Pfister, T., Ravikumar, (2020). On completeness-awan concept- based explanations in deep neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, & Lin (Eds.), Advances in neural information processing systems (Vol. 33, pp. 20554-20565). Curran Associates, Inc. Retrieved from http::///proceeeeigggsss..s// www.ips.................. cc/ paper
pdf
Yuksekgonul, M., Wang. M., Zou, (2023). Post-hoc concept bottleneck models. The 1/th international conference on learning representations. Retrieved from https://openrevian htt./forum?idnnA5A2200.
14

---

A Motivation & Intuitive Examples
This appendix provides additional schematics and intuitive examples, clarifying instance-specific concept-based interventions. Figure A.1 schematically summarises the principle behind our interven- tion procedure with fewer technical details than shown in Figure
prodicted concepts "1o skiper "te" "goss""
intervened concepti
"strjoae"
"peaa"
probing
Intervention procedure
impul
oulput 9
updisted
sclivalion vector
Intervened activation
outputt y
Figure A.1: Schematic summary of concept-based based instance- speccffcc interventions black-box neural network. This work introduces an intervention procedure that, given concept values c', for an input æ. edits the network's activation vector at intermediate layer, replacing it with =' coherent with the given concepts. The intervention results in an updated prediction j/
Figure A.2 extends the simplified example from the main text (Figure TT. Herein, the model wrongly predicts that the image of a grizzly bear from the AwA2 dataset (Appendix (D.2p depicts an otter. The user inspects the concepts via probe and intervenes several hand-picke common-senss variables. Our procedure updates the representations, and the predicted class is flipped to the correct one.
In addition the model 'correction', interventions allow 'steering" the model prediction. In Figure A.3] the model correctly predicts grizzly bear. The most likely prediction can be flipped to the polar bear by editing concept variables and using the proposed procedure.
prediction
9 [ottor'] ==100 Warizelyybbur
4[vhii=r:=0059
c[thhitw') 0.00
21b1g'1=0.50
[['igg]]=100 1.00
[[=ma11r]=003
('anci1')=000
[[:ear']=0.37
(l'ecnn]]-000 0.00
40ri1soez=*==0.90
[1'1ippera'']- 0.00
Potter 100.0
40=a11r]=0.91
<''ail1]] 0.00
4fchoctentarr
cl'wa d''aa ke')
[Ustronn'==024
['zzronn')= 1.00
30mecciee1-000
[nuscie']=
[0ase:i====090
c ['axetie']=
[iieoe']] 1.00
probing
intervention
updated prediction
Figure A.2: A model 'correction' example using concept-basee based instance-specific interventions on the AwA2 dataset. The black-box neural network wrongly predicts that the image depicts an otter. Using our technique, we intervene on the network's representation and flip the final prediction to the correct class.
15

---

prediction
[ 1=100
[black]] 0.00
[[wh1te1]=000
dhaits')-11 1.00
#[torown |=1.00
[brown']=-000
A[agzyy ]=0=
(eeaay" -000
A1'extmer
<'evttss'=1100
[['ctan]==088
[ 1.00
plpplooreeer J-0.2
[[azettc']=00
<'aacctto1=-11 1.00
8fsosssai1]=0000
[coastal"]]-
[['oroot'] =0.97
1ooees==1 0.00
[[osean']
Tocean' 1.00
[1uster']
< Teeterr 1
probing
Intervention
updated prediction
Figure A.3: A model 'steering' on the AwA2 dataset. By editing the predicted concepts and applying our method, we can manipulate the model into wrongly predicting that the image depicts a polar bear instead of grizzly bear.
16

---

B Fine-tuning for Intervenability
Algorithm B.I contains the detailed pseudocode for fine-tuning for intervenability described in Section 3.3 Recall that the black-box model fo is fine-tuned using a combination of the target prediction loss and intervenability defined in Equation 3 The implementation below applies the special case of 3 = 1, which leads to the simplified loss. Importantly. in this case, the parameters Φ are treated as fixed, and the probing function de does not need be fine-tuned alongside the model. Lastly, note that, in Algorithm [BI interventions are performed for whole batches of data points using the procedure described in Section 3.1
Algorithm B.1: Fine-tuning for Intervenability
Input:
Trained black-box model fe (Uhp, ha), probing function de. concept prediction loss function _ target prediction loss function Σ validation set {(z;, C. y;))==r; intervention strategy т, distance function d, hyperparameter value À > 0, maximum number of steps E, for the intervention procedure, parameter for the convergence
criterion ε >> 0 for the intervention procedure, learning rate 1)r > 0 for the intervention procedure, number of fine-tuning epochs E, mini-batch size M, learning rate 7) > 0 Output: : Fine-tuned model
Train the probing function qe on the validation set, iee. < arg ming Σx=1 L (gg' (he (xx).c.)
Step 1: Probing
for e= to E 1 do
Randomly split {1, N} into mini-batches of size M given by B for b do
Zb +- h (x%)
jo - I6 (zb)
& 9g (zb)
Sample c₁ ~ x Initialise 2 %b, 2 x; E,e, and ez
>> Step 2: Editing Representations
10
while
2/6 z/,ota|
2 Eq and es < E do
2/,ol1 2/b
12
z/. + 2/b
71 V N/ [d(ze, z1{) + AC (ge (z() cb))
> Equation
13
et s- et +1
14
end
15
V - 9up (z/)
> Step 3: Updating Output Eqqutiin
(1h. 1b) sb 1/ V
end
18 end
return fo
17

---

C Further Remarks & Discussion
This appendix contains extended remarks and discussion beyond the scope of the main text.
Design Choices for the Intervention Procedure The intervention procedure entails a few design choices, including the (non)linearity of the probing function, the distance function in the objec- tive from Equation [ and the tradeoff between consistency and proximity determined by À from Equation
[ We have explored some of these choices empirically in our ablation experiments (see Figure F.4 and Appendix F). Naturally, interventions performed on black-box models using our method are meaningful in so far as the activations of the neural network are correlated with the given high-level attributes and the probing function de can be trained to predict these attribute values accurately. Otherwise, edited representations and updated predictions are likely to be spurious and may harm the model's performance.
Should All Models Be Intervenable? Intervenability (Equation 3) in combination with the probing function, can be used to evaluate the interpretability of a black-box predictive model and help understand whether (i) learnt representations capture information about given human- uumaa--understndablle attributes and whether (ii) the network utilises these attributes and can be interacted with. However, a black-box model does not always need to be intervenable. For instance, when the given concept set is not predictive of the target variable, the black box trained using supervised learning should not and probably would not rely the concepts. On the other hand, if the model's representations are nearly perfectly correlated with the attributes, providing the ground truth should not significantly impact the target prediction loss. Lastly, the model's intervenability may depend on the chosen intervention strategy, which may not always lead to the expected decrease in the loss.
Intervenability & Variable Importance As mentioned in Section 3.2 intervenability (Fquation 2 measures the effectiveness of interventions performed on a model by quantifying a gap between the expected target prediction loss with and without performing concept-based interventions. Equa- tion 2 is reminiscent of the model reliance (MR) (Fisher et al. 2019) used for quantifying variable importance.
Informally, for a predictive model f, MR measures the importance of some feature of interest and is defined as
MR expected loss of f under noise expected loss of f without noise'
(C.1)
Above, the noise augments the inputs of and must render the feature of interest uninformative of the target variable. One practical instance of the model reliance is permutation-bant baase variable importance (Breiman 2001
Molnar 2022).
The intervenability measure in Equation 2 can be summarised informally as the difference between the expected loss of g& without interventions and the loss under interventions. Suppose intervention strategy π is specified so that it augments single concept in è with noise (Equation C.I). In that case, intervenability can be used quantify the reliance of g6 on the concept variable of interest in è. The main difference is that Equation C.I is given by the ratio of the expected losses, whereas intervenability looks at the difference of expectations.
Comparison with Conceptual Counterfactual Explanations We can draw relationship between the concept-based interventions (Equation 3) and conceptual counterfactual explanations (CCE) studied by Abid et al. (2022) and (S. Kim et al. (2023). In brief, interventions aim to "inject" concepts c' provided by the user into the network's 's representation to affect and improve the downstream prediction. By contrast, CCEs seek to identify sparse set of concept variables that could be leveraged to flip the label predicted by the classifier So- Thus, the problem tackled in the current work is different from and complementary to CCE.
More formally, following the notation from Section et al. 2022 is given by
conceptual counterfactual explanation Abid
arg min (ov hhp (x)) + =C) v/) ||/||| 3 |||||2
(C.2)
s.t. w min < w S w max
18

---

where Č is the concept bank, y' is the given target value (in classification, the opposite to the predicted ÿ). a, a are penalty weights, and [wrtin, w [ defines the desired range for weights w. Note that further detailed constraints are imposed via the definition by Abid et al. (2022).
w tiin will in the original work
Observe that the optimisation problem in Equation C.2 defined w.r.t the flipped label y and does not incorporate user-specified concepts as opposed interventions in Equation | Thus, CCEs aim to identify the concept variables that need to "added" to flip the label output by the classifier. In contrast, interventions seek to perturb representations consistently with the given concept values.
19

---

D Datasets
Below, we present further details about the datasets and preprocessing involved in the experiments (Section[]). The synthetic data can be generated using our code. AwA2, CUB, CIFAR-10, 10, ImageNet, CheXpert, and MIMIC-CXR datasets are publicly available. Table D.I provides a brief summary.
Table D.I:
Dataset summary. After any filtering or preprocessing, N is the total number of data points; is the input dimensionality; and K is the number of concept variables.
Dataset
Data
N
K
Synthetic
Tabular
50,000
1,500
30
AwA2
Tmage
37,322
224x
85
Image
11,788
224x
112
CIFAR-10
Image
60,000
128x 128
143
ImageNet
Image
1,331,167
128 128
100
CheXpert
Image
49,408
224
13
MIMIC-CXR
Image
54,276
224
13
D.1 Synthetic Tabular Data
As mentioned in Section to perform experiments in controlled manner, we generate synthetic nonlinear tabular data using the procedure adapted from Marcinkevićs et al. (2024). We explore two settings corresponding to different data-generating mechanisms (Figure (FigureD.II) D.II: (a) bottleneck and (b) incomplete. The first scenario directly matches the inference graph of the vanilla CBM (Koh et al. 2020). In the incomplete scenario, are given incomplete concepts, i.e. does not fully explain the variance in y. Here, unexplained variance is modelled as a latent variable r via the path a + -
Unless mentioned otherwise, we mainly focus on the simplest scenario shown in Figure D.I [a). Below, we outline each generative process in detail. Throughout this appendix, let N, p. and K denote the number of independent data points {xx, C₁ H covariates, and concepts, respectively, Across all experiments, set N 50,000, 1,500, and K 30. This dataset was divided according to the 20%%20%%-22 siai--vaiiaaiio-tee split.
y
(a) Bottleneck
(b) Incomplete
Figure D.I: Data-generating mechanisms for the synthetic dataset summarised as graphical models. Each node corresponds random variable. Observed variables are shown in grey.
Bottleneck In this setting, the covariates æ; generate binary-valued concepts e₁ 1) and the binary-valued target depends on the covariates exclusively via the concepts. The generative process is as follows:
1. Randomly sample R s.t. Mj Uniform (-5, 5) for 1 < S p.
2. Generate random symmetric, positive-definite matrix Σ E Roxp
3. Randomly sample a design matrix X E Nx s.t. X,,: N₂ (n, Σ)
4. Let h: R + R and g R I be randomly initialised multilayer perceptrons with ReLU nonlinearities.
5. Let 6,K 1 [IM(X,.)) 2mz)) where my median (1Ih (xI)())))))) for s siSM N
and < k < K.
6. Let (g(e))2m,)) where my = median (L (c)))))) for 1S
X.: refers to the i-th of the design matrix. f.e. the covariate vector æ;
20

---

Incomplete Last but not least, to simulate the incomplete concept set scenario, where a part of concepts are latent, we slightly adjust the procedure from the bottleneck setting above:
Follow steps 1-3 from the bottleneck procedure.
2. Let h R" + K+J and g RK+J + R be randomly initialised multilayer perceptrons with ReLU nonlincarities, where J is the number of unobserved concept variables.
3. Let u4A (mx(x))) where my median (X,)())) for S isis
and 1 < k S K J
4. Let = U.l1 and ri (i,((K for 1 SiSN.
5. Let y/ = 1 ( where my = median (19 (u.)) for sis
Note that, in steps 3-5 above, u, corresponds to the concatenation of and r,. Across all experiments, we set 90.
D.2 Animals with Attributes
Animals with Attributes 2 dataset (Lampert et al. 2009 Xian et al. 2019) serves as natural image benchmark in our experiments. It comprises 37,322 images of 50 animal classes (species), each associated with 85 binary attributes utilised as concepts. An apparent limitation this dataset is that the concept labels shared across whole classes, similar to the Caltech- UCCD Birds experiment from the original work by Koh et al. (2020). Thus, AwA2 offers a simplified setting for transfer learning across different classes and is designed address attribute-based classification and zero-shot learning challenges. In our evaluation, we used all the images in the dataset without any specialised preprocessing or preselection. All images rescaled to 224 x 224 pixels. This dataset was divided according to the 60%-20%--00 train-validation-tes split.
D.3 Caltech-UCSD Birds
Caltech-UCSD Birds-200-200 dataset (Wah et al. 2011 1) is another natural image benchmark explored in the original work on CBMs Koh al. 2020). consists of 11,788 bird photographs from 200 species (classes) and originally includes 312 concepts, such as wing colour, beak shape, etc. We have followed the preprocessing routine proposed by Koh al. (2020) and keep the original train-validation-test split to avoid spurious mixing of photographers in the data. Particularly, the final dataset includes only the 112 most prevalent binary attributes. We have included image augmentations during training, such random horizontal flips, adjustments of the brightness and saturation, and normalisation. Similar to AwA2, CUB concepts are shared across all instances of individual classes. No additional specialised preprocessing performed the images, which rescaled to resolution 224 x 224 pixels.
D.4 CIFAR-10
CIFAR- 10 Krizhevsky et al. 2009 is benchmarking natural image dataset. It includes 60,000 x2x32 colour images 10 classes, with 6,000 images per class. There are 50,000 training and 10,000 test images. To generate the validation set, we randomly hold out 10,000 images from the training data remain faithful to the original test set. Following the setup by Oiikarinn al. (2023). we generate 143 concept labels as described in Section I between each instance and the concept text embedding with thus of not the concept. We apply random horizontal flips, adjustments to brightness and saturation, resize the images to resolution of 128x 128 pixels, and apply normalisation.
(luiing VLMs by comparing the similarities
httts:///vmlt ista. at/AvA2/
https://wwss sttpp::/www.visionn hawpssnnccaleecceeee/aaaaaeeeeee 200pteeeeuu_bo_00____e11 20112011/ wttp://wwww.ooooooooo
https://www.ca toronto. httptt//eddddkkiiiiiiet html
21

---

D.5 ImageNet
ImageNe dataset (Russakovsky et 2015) is a large-scale natural image benchmark. It includes 1,000 object classes and contains 1,28 1,7 167 training, 50,000 validation, and 100,000 unlabelled test images. our experiments, we allocate half of the validation as the test set. We apply random horizontal flips, adjustments to brightness and saturation, resize images to resolution of 128x 128 pixels, and apply normalisation.
We adapt the 4,751 original concept variables introduced using GPT-3 by Oikarinen et al. (023) To ensure that concepts are predictive of the target variable and can be inspected manually, we retain 100 attributes. Specifically, we keep those with the highest correlations with the final target while prioritising their diversity. To this end, we cluster concepts into 25 groups using the k-means algorithm and sample 4 attributes from each cluster based on Cramér's V (Cramér 1999) between the concept variable and y. Final concept labels were generated using CLIP as for CIFAR- ((Sectin 4
D.6 Chest X-ray Datasets
As mentioned, we conducted an empirical evaluation on two real-world chest X-ray datasets: CheX- pert (Irvin al. 2019) and MIMIC-CXR (Johnson al. 2019). The former includes over 220,000 chest radiographs from 65,240 patients at the Stanford Hospital These images are accompanied by 14 binary attributes extracted from radiologist reports using the CheXpert labeller (Irvin al.) 2019), model trained to predict these attributes. MIMIC-CXR is another publicly available dataset containing chest radiographs DICOM format, paired with free-text radiology reports It comprises more than 370,000 images associated with 227 27,835 radiographic studies conducted at the Beth Israel Deaconess Medical Center, Boston, MA, involving 65,379 patients. Similar CheXpert, the same labeller was employed to extract the same set of 14 binary labels from the text reports. Notably, some concepts may be labelled as uncertain. Similar to Chauhan et al. (2023), we designate the Finding/No Finding attribute as the target variable for classification and utilise the remaining labels as concepts. In particular, the concepts are atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion, pleural other, pneumonia, pneumothorax, and support devices. In our implementation, we remove all the samples that contain uncertain labels and discard multiple visits of the same patient, keeping only the last acquired record- ing per subject for both datasets. All images were cropped to a square aspect ratio and rescaled to 224 x 224 pixels. Additionally, augmentations were applied during training, namely, random affine transformations, including rotation up to 5 degrees, translation up to 5% of the image's 's width and height, and shearing with maximum angle of 5 degrees. We also include random horizontal flip augmentation to introduce variation in the orientation of recordings within the dataset. Both chest radiograph datasets are divided according to the 10%%-1%%-100 10% train-vale seii--vaiidaiio-teee split.
22

---

E
Implementation Details
This section provides implementation details, such network architectures and intervention and fine-tuning procedure hyperparameter configurations. All models and procedures were implemented using PyTorch (v 1.12.1) (Paszke et al. 2019) and scikit-learn (v 1.0.2) (Pedregosa al. 2011). We run the reported experiments on a cluster of GeForce RTX 2080 GPUs with single CPU worker. The span of time elapsed to run each method is dependent on the dataset and architecture. On the synthetic tabular data, on average, it takes approx. 3 hours to train a concept bottleneck or black-box model. For the Animals with Attributes 2 and chest X-ray datasets, it takes up to 10 hours to train black boxes and CBMs. However, when using pretrained backbone, e.g. Stable Diffusion, only fine-tuning is required, the run-time of which ranges from 10 minutes to hour for all considered datasets.
Network & Probe Architectures For the synthetic tabular data, we utilise fully connected neural network (FCNN) as the black-box model. Its architecture is summarised in Table E. inn PyTorch-like pseudocode. For this classifier, probing functions are trained, and interventions are performed on the activations of the third layer, i.e. the output after line 2 in Table For AwA, CUB, MIMIC-CXR, and CheXpert, we use the ResNet- (He al., 2016) with random initialisation followed by four fully connected layers and the sigmoid or softmax activation. Probing and interventions are performed on the activations of the second layer after the ResNet- 18 backbone. Furthermore, we provide results for AwA with the Inception (Szegedy et al., 2015) backbone. For CIFAR-10 and ImageNet, showcase the scalability of our methods on the pretrained Stable Diffusion Rombach et al. (2022) backbone followed by four fully connected layers and the sigmoid softmax activation. For the CBMs, to facilitate fair comparison, we use the same architectures with the exception that the layers mentioned above were converted into bottlenecks with appropriate dimensionality and activation functions. Similar settings are used for post hoc CBMs with the addition of a linear layer mapping backbone representations to the concepts.
For fine-tuning, we utilise a single fully connected layer with an appropriate activation function as a linear probe and a multilayer perceptron with a single hidden layer as a nonlinear function. For evaluation on the test set (Table[]. we fit a logistic regression classifier from scikit-learn as a linear probe. The logistic regression is only used for evaluation purposes and not interventions.
Table E.1: Fully connected neural network architecture used as a black-box classifier in the experi- ments on the synthetic tabular data. nn stands for torch. nn; stands for torch, nn. functional input_ dim corresponds to the number of input features.
FCNN Classifier
nn. Linear (input._di, 266)
F.relu()
Dropout (0.06)
nn. Batchhorri (256)
range (2)
on, Linear (256, 256) Frrllu()
(ropout
nn. BatchMor
Li. (256, 1)
torch.signoid()
Interventions Unless mentioned otherwise, interventions on black-box models were performed using linear probes, the random-subse intervention strategy, and under À = 0.8 (Equation []. Recall that Figures F.4 and F22 provide ablation results the influence of the latter hyperparameter. Despite some variability, the analysis shows that higher values of À expectedly lead to more effective interventions. The choice of X for our experiments was meant represent the "average case", and no tuning was performed for this hyperparameter.
Similarly, we have mainly used a linear probing function and the simple random-subset intervention strategy to provide proof-of-concept results without extensive optimisation of the intervention strategy or the need for nonlinear probing. Thus, our primary focus was on demonstrating the intervenability of black-box models and showcasing the effectiveness of the fine-tuning method rather than an exhaustive hyperparameter search.
23

---

Intervention Strategies In ablation studies, we compare two intervention strategies (Figure
F.4
inspired by Shhin et al. (2023): (i) random-subeee and (ii) uncertainty-basse based.. Herein, we provide
more formal definition of these procedures described as pseudocode in Algorithms E. F.2 Recall
that given data point (a,, c, y) and predicted values é and Ù. an intervention strategy defines distribution over intervened concept values c'. Random-subset strategy (Algorithm E. replaces predicted values with the ground truth for several concept variables (k) chosen uniformly at random. By contrast, the uncertainty-bae with the ground-truth values without replacement with initial probabilities proportional to the concept prediction uncertainties, denoted by In our experiments, the components of are the outputs of the sigmoid function, and the uncertainties are computed as o; = 1/ (i: - 0.5| + E) (Shin et al. 2023 for 1 < K, where > 0 is small.
strategy (Algorithm E.2) samples concept variables to be replaced
Algorithm E.1: Random-subset Intervention Strategy
Input: data point (æ: c, y), predicted concept values é, the number of concept variables be intervened on S k S K
Output:Intervened concept values d
c' é
2 Sample I uniformly at random from [1, K} |s| = k}
cg + CT
return d'
Algorithm E.2: Uncertainty-based Intervention Strategy
Input: A data point (ar, c, y), predicted concept values è, the number of concept variables be intervened on < k S K
Output: Output:Interveenn Intervened concept values d
1 jj + 1/ (1ēz - 0.5| + E) for 1 i K, where & > 0 small ok)
2 o - (0)
c' è
4 Sample k: indices I = راررا s.t. each in is sampled without replacement from {l,..., K}
with initial probabilities given by (a + e) (Ke Exs o.). where 0 is small
cz Cy
6 return d'
Fine-tuning for Intervenability The fine-tuning procedure outlined in Section 3.3 and detailed in Algorithm B.T [B.] necessitates intervening on the representations throughout the optimisation. During fine-tuning, we utilise the random-subset intervention strategy, i.e. interventions are performed on subset of the concept variables by providing the ground-truth values. More concretely, interventions are performed on 50% of the concept variables chosen uniformly at random.
Fine-tuning Baselines The baseline methods described Section incorporate concept informa- tion in distinct ways. On the one hand, the multitask learning approach, FINE-TUNED, MT, utilises the entire batch of concepts at each iteration during fine-tuning. For this procedure, we a 1.0 (recall that controls the tradeoff between the target and concept prediction loss terms). On the other hand, the FINE-TUNED, A approach, which appends the concepts to the network's activations, does not use the complete concept set for each batch. In particular, before appending, concept values are randomly masked and set to 0.5 with a probability of 0.5. This practical trick is reminiscent of the dropout (Srivastava et al. 2014] and is meant to help the model remain intervenable and handle missing concept values.
Hyperparameters Below, we list key hyperparameter configurations; the remaining details are documented in our code. For the synthetic data, CBMs and black-bor classifiers are trained for 150 and 100 epochs, respectively, with a learning rate of 10-1 and batch size of 64. Across all other experiments, CBMs are trained for 350 epochs and black-box models for 300 epochs with learning rate of 10-1 halved midway through training and batch size of 64. CBMs trained using the
24

---

joint optimisation procedure (Koh et al. 2020) under a 1.0, where a controls the tradeoff between the concept and target prediction losses. All probes were trained on the validation data for 150 epochs with learning rate of 10-2 using the stochastic gradient descent (SGD) optimiser. Finally, all fine-tuning procedures were run for 150 epochs with learning rate of 10-2 and batch size of 64 using the Adam optimiser. ImageNet is an exception to the above configurations due to its large size. The black-box models this dataset were trained for 60 epochs, and the probes and fine-tuning procedures for 20 epochs. At test time, interventions were performed on batches of 512 data points.
25

---

F
Further Results
This section contains supplementary results and ablation experiments not included in the main body of the text.
F.1 Further Results on Synthetic Data
showing inter- vention curves w.r.t. AUPR under the boutleneck generative mechanism for the synthetic data with varying validation set size. The overall patterns and conclusions are similar to those observed w.r.t. AUROC (Figure
Figure supplements the intervention experiment results in Figure 3
Section [
0.75
3.1
3.70
0.0
0.7
260
8.55
0.5
60
20
:00
20
60
60
30
100
concepes Inrenwoee
concepts intenvered on
Vai (0.5%
(c) Naal = 100 (1%)
(e)
Naa1 500 (5%)
0.9
0.0
0.0
0.7
60
60
150
40
60
62
N 1000 (10%)
Nval
5000 (50%) (f) Mual 10000 (100%)
(val)
Post hoc CBM
Fine-tuned,
Fine-tuned_l
Figure F.I: Intervention results w.r.t. target AUPR on the synthetic bottleneck data. We explore the performance under varying validation set sizes (Nva)). Percentages correspond to the fractions of the original validation set. For CBMs, we report the results obtained by training on the validation performed test data across ten
(CBM val) and full training sets (CBM full). Interventions simulations. Lines correspond to medians, and confidence bands are given by interquartile ranges.
0.85
0.00
0.55
0.70
0.70
120
20
82
160
20
80
100
0.50
0.00
0.60
0.76
0.70
0.0
0.2
C15
43
116
40
100
20
learvenes
(a) Influence of A
(b) Intervention strategies
(c) Probe linearity
7. 0.2
3.2
Random
Uncertainty
Linear
Nonlinear
Figure F:2: Ablation study results w.r.t target AUROC (top) and AUPR (bottom) on the synthetic
dataset. Bold lines correspond to medians, and confidence bands are given by interquartile ranges across ten independent simulations. (a) Intervention results for the untuned black-boo model under varying values of À E (0.2, 0.4, 0.8, 1.6, 3.2} (Equation 3) Darker colours correspond to lower values. (b) Comparison between random-subset and uncertainty-based intervention strategies. (c) Comparison between linear and nonlinear probing functions.
26

---

provides ablation experiment results obtained on the synthetic tabular data under the bottleneck generative mechanism shown in Figure D.I(a). In Figure F.2ja), we plot black-box intervention results across varying values of the hyperparameter À (Equation[]) T Higher Às result in more effective interventions: this finding expected since À the weight of the term penalising the inconsistency of the concept values predicted by the probe with the given values and, in the current implementation. interventions are performed using the ground truth. Interestingly, in Figure F.2()) observe no difference between the random subset and uncertainty-bet could be explained by the fact that, in the synthetic dataset, all concepts by design are, on average, equally hard to predict and equally helpful in predicting the target variable (see Appendix D.I. Hence, the entropy- based uncertainty score should not be as informative in this dataset, and the order of intervention the concepts should have little effect. Finally, similar to the main text, Figure suggests that a nonlinear probing function improves intervention effectiveness.
Figure F.2
b). we intervention strategies. This
F.2 Effect of Interventions on Representations
In some cases (Abid et al. 2022), it may be deemed desirable that intervened representations z' (Equation]) remain plausible, i.e. their distribution should be close to that of the original represen- tations z. Figure F.3 shows the first principal components (PC) obtained from a batch of original and intervened representations from the synthetic dataset (under the bottleneck scenario) for two different values of the A-parameter. We observe that, under À 0.2 (Figure aa)), interventions affect representations, but z' mainly stay close to Z w.r.t. the two PCs. By contrast, under À = 0.4, interventions lead to a visible distribution shift, with many vectors z' lying outside of the mass of z. This behaviour is expected since À controls the tradeoff between the consistency with the given concepts e' and proximity. Thus, if the plausibility of intervened representations is desirable, the parameter À should be tuned accordingly.
70
PC1
(a) À 0.2
(b) A 0.4
Figure F.3: Principal components (PC) for a batch of data point representations before (=) and
after (z') concept-based interventions under the varying values of the parameter for (a) X 0.2 and
(b) X = 0.4.
F33 Further Results on AwA2
This section includes the ablation experiments carried out on the AwA2 dataset (Figure F.4) similar to those the synthetic shown in Figure F22 Firstly, we vary the X-parameter from Equation 3 which weighs the cross-entropy term, encouraging representation consistency with the given concept values. The results in Figure F.4 a) suggest that interventions are effective across all As. Expectedly, higher hyperparameter values yield more effective interventions, i.e. a steeper increase in AUROC and AUPR. Figure F.4 b) compares two intervention strategies: randomly selecting concept subset (random) and prioritising the most uncertain concepts (uncertainty) (Shin et al. 2023) to intervene on (Algorithms E. and
E.2 Appendix E) The intervention strategy has clear impact the
performance increase, with the uncertainty- based approach yielding a steeper improvement. Finally, Figure F.4 c) compares linear and nonlinear probes. Here, intervening via a nonlinear function leads to significantly higher performance increase.
Finally, to show the scalability of our methods to different backbone architectures, we report in Figure F55 results with the Inception (Szegedy et al. (2055) backbone. As can be seen, the intervention procedure and our fine-tuning method remain successful regardless of the backbone architectures.
27

---

1.000
1002
0.99
0393
0 999
0.997
0.977
0.96
0.93
0.995
0.905
25
40
05
50
100
40
2.90
0.90
2.36
0.36
2.44
3.92
032
0.52
61
20
40
69
80
100
20
60
05
100
(a) Influence
(b) Intervention strategies
(c) Probe linearity
(a) 0.2
=
Random
Uncertainty
Linear
Nonlinear
Figure F.:: Intervention results on the AwA2 dataset w.r.t. target AUROC (top) and AUPR (bottom) across ten independent train- validation-test splits. (a) Intervention results for the untuned black-box
model under varying values of À E {0.2, 0.4, 0.8, 1.6, 3.2} (Equation 3). Darker colours correspond
to lower values. (b) Comparison between random-subset and uncertainty-based intervention strategies. (c) Comparison between linear and nonlinear probing functions.
1.00
0.93
0.093
0.955
0.977
40
50
103
3.00
1.003
0.95
0.992
0.90
0.85
102
50 intevored
intarvenes
(a) baselines
(b) Without post CBM
Black box
CBM
Post hoc CBM
Fine-tuned, MT
Fine-tuned, A
Fine-tuned.
Figure F:5: Effectiveness of interventions w.r.t. target AUROC (top) and AUPR (bottom) on the AwA2 dataset with the Inception backbone. In the panels on the right (b), we have excluded post hoc CBM for legibility.
F,4 Results CUB
In line with the previous literature (Koh et al. 2020), we assess our approach on the CUB dataset with the results summarised in Figure [. This dataset is similar to the AwA2, the concepts shared across whole classes. Thus, concepts and classes feature a strong and simple correlation structure. Expectedly, the CBM performs very well due its inductive bias relying on the concept variables. As in the previous simpler scenarios, untuned black boxes are, in principle, intervenable. However, the proposed fine-tuning strategy considerably improves the effectiveness of interventions. On this dataset, the performance (without interventions) of the post hoc CBM and the model fine-tuned for intervenability visibly lower than that of the untuned black box. We attribute this to the poor association between the concepts and the representations learnt by the black box. Interestingly, post hoc CBMs do not perform as successfully as the models fine-tuned for intervenability. Generally, the behaviour of the models on this dataset falls in line with the findings described in the main body of text and supports our conclusions.
28

---

1.0
1.0
0.8
0.
0.7
40
20
60
100
conooppt intevvreed on
conceppts intervered on
(a) AUROC
(b) AUPR
Black box
CBM
Post hoc CBM
Fine tuned,
Fine-tuned,
Fine-luned
Figure F.6: Intervention results w.r.t. target (a) AUROC and (b) AUPR across ten initialisations on the CUB dataset.
F.5 Results on ImageNet
To further support the findings on CIFAR-I0 using CLIP-based concept annotations, we explore the intervention effectiveness our method in the large-scale ImageNet dataset. In Fi7 we show how the proposed fine-tuning method improves the intervention effectiveness when compared with the studied baselines. Note that the CBM is not computed due to the constraint of retraining the Stable Diffusion backbone from scratch for the concept bottleneck adaptation.
0.95
0.90
0.65
0.2
0
20
50
60
100
40
80
80
100
concects
(a) AUROC
(b) AUPR
Black box
Post hoc CBM
Fine-tuned, MT
Fine-tuned,
Fine-tuned,1
Figure F.7: Intervention results w.r.t. target (a) AUROC and (b) AUPR across ten initialisations on the ImageNet dataset.
F.6 Results on CheXpert
To further showcase the practicality of our approach, we present empirical findings on the CheXpert dataset, which are complementary to the MIMIC-CER results included in Section S Figure shows how untuned black-box neural networks are not intervenable but after fine-tuning for intervenability, the model's predictive performance and effectiveness of interventions improve visibly and even surpass those of the CBM. Finally, the remaining baseline including post hoc CBMs (even with residual modelling) exhibit behaviour similar the synthetic dataset with incomplete concepts: interventions have or even adverse effect on performance.
1.000
0.975
0.50
0.950
0.80
0.500
0.875
20
60
80
100
20
40
80
100
cotioepts incervereed
concepts
(a) AUROC
(b) AUPR
Black box
CBM
Post CBM
Fine-tuned, MT
Fine-tuned,
Fine-tuned
Figure F.8: Intervention results w.r.t. target (a) AUROC and (b) AUPR across ten initialisations the CheXpert dataset.
29

---

E.7
Calibration Results
The fine-tuning approach introduced leads retter-calibrated predictions (Table D. possibly, due to the regularising effect of intervenability. In this section, we further support this finding by visualising calibration curves for the binary classification tasks, namely, for the synthetic tabular data and chest radiograph datasets. Figure F.9 shows calibration curves for the fine-tuned model, untuned black box, and CBM averaged across ten seeds. We have omitted fine-tuning baselines in the interest of legibility since their predictions were comparably ill-calibrated as for the black box. The fine-tuned model consistently and considerably outperforms both the untuned black box and the CBM in all three binary classification tasks, its curve is the closest to the diagonal, which corresponds to perfect calibration.
14
14
0.8
08
0.6
250
%0
02
2.4
0.6
0.8
10
Prodiced Probab
Prodiceed Probabiityy
Predicted Probuciity
Synthetic
(b) CheXpert
(c) MIMIC-CRR
box
Fine-tuned.
Figure F99: Analysis of the probabilities predicted by the black box, fine-tuned black box, and CBM on the (a) synthetic, (b) CheXpert, and (c) MIMIC-CXR, The calibration curves, averaged across ten seeds, display for each bin the true empirical probability y = against the probability predicted by the model. The gray dashed line corresponds to perfectly calibrated predictions.
F88 Influence of the Distance Function
Throughout the experiments, we have consistently utilised the Euclidean distance as in Equa- tion D In this section, explore the influence of this design choice. In particular, fine-tune the black-box model and intervene on all models under the cosine distance given by d(xr, xx)
x'/ (11=112 ||x0|] (2)))
1.000
0.300
A:800
0.946
0.944
6.85
0.992
0.75
6.0
0.30
1.00
40
60
1.000
0.35
0.64
07
100
5.CONCOT5 stcrebsid cm
S.coecepts rnnveneed
(a) Synthetic
(b) Aw.A
(c) CheXpert
(d) MIMIC-CXR
Black box
CBM
Fine-tuned, MT
Fine-tuned, A
Fine-tuned.1
Figure F. Intervention results w.r.t. target AUROC (top) and AUPR (bottom) under the cosine distance function (Equation D on four studied datasets: (a) synthetic, (b) AwA2, (c) CheXpert, and (d) MIMIC-CXR The comparison is performed across ten seeds.
30

---

Figure F.TO shows the intervention results under the cosine distance on the four datasets considered before. Firstly, for the synthetic and AwA2 datasets, we observe that the untuned black box is visibly less intervenable than under the Fuclidean distance. In fact, for the AwA2 (Figure F.T0 top), interventions slightly reduce the test-set AUROC. These results suggest that the intervention procedure is, indeed, sensitive to the choice of the distance function, and we hypothesise that the distance should be chosen to suit the latent space of the neural network considered. Encouragingly, the proposed fine-tuning procedure is equally effective under the cosine distance. Similar to the Euclidean case, it greatly improves the model's intervenability.
Additional Baselines: Post Hoc CBMs with Residual Modelling
As an extension of post hoc CBMs, we consider residual modelling as described by Yuksekgonul et al. (202)). We refer to this baseline as POST HOC CBM-H, Lee. hybrid post hoc CBM. This variant adds a residual connection between the network's 's representations and the final output. In particular, after sequential optimisation of the post hoc CBM parameters (Section 4 a residual predictor re trained and added to the model's output: minç (( [C' (g & (gg
(x))) + re (ho(v)) v)]. Similar to Yuksekgonul et al. (023). we utilise a linear residual predictor in our experiments.
Figure shows intervention results for a selection of datasets. We specifically chose those experiments where simple post hoc CBMs exhibited poor intervenability. For legibility, we have only and post hoc CBMs, and the model fine-tuned
included the results for the original black box, for intervenability. Across all datasets, POST HOC CBM-11 shows a minor improvement in average predictive performance compared to the simple model. However, the introduction of the residual predictor, expectedly, has no significant effect on the steepness of the intervention curves. Thus, our fine-tuning approach consistently outperforms both variants of the post hoc CBM w.r.t. the effectiveness of interventions.
25975
0.55
0.35
1725
0.32
3.900
2.57
1.80
33500
0.40
0,50
40
130
1.00
1.80
0.35
0.9
5 3.20
0.00
0.44
0.8
3.5
160
60 Concupts
33
23
100
45
125
Fnoveree
(a) Synthetic, incomplete
(b) Aw.A
(c)
CheXpert
(d) MIMIC-CXR
Black box
CBM
Fine-tuned,
Post hoc CBM
Post hoc CBM-h
Figure F.11: Intervention results w.r.t. target AUROC (top) and AUPR (bottom) including the residual posthoc- -BBM baseline on four studied datasets: (a) synthetic, (b) AwA2, (c) CheXpert, and (d) MIMIC- CXR. The comparison is performed across ten seeds.
31

---

F.10 Influence of the CBM training method
Lastly, we explore Figure F.12 the intervention performance of the CBMs under the three different training methods introduced by Koh et al. (2020): independent, sequential, and joint. We show in both synthetic and MIMIC-CXL datasets that the results are comparable, and therefore, throughout the manuscript, we have focused solely the joint optimisation procedure. We chose two datasets for this ablation to span the simpler synthetic tabular scenario and the more realistic chest. X-ray classification, where the concepts and final target may have more complex dependency.
0.85
0.9
100
0.9
0.8
65
0.7
concapts Interenned
concopts riteeeeee
(a) Synthetic
MIMIC-CXE
CBM-sequental
CBM-ooint
CBM-ndependent
Figure F.12: Intervention results w.r.t. target AUROC (top) and AUPR (bottom) across ten initial- isations on the (a) synthetic and (b) MIMIC-CXR datasets for the three different CBM training procedures.
32

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: In the abstract, we claim that we introduce a new measure for intervenabil- ity and that we propose method to make black boxes more intervenable. This is later demonstrated empirically on seven datasets.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The last section of the manuscript includes the conclusion and limitations, followed by future work directions to tackle them.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
33

---

Answer: [Yes]
Justification: the methods section, we formalise our method. The appendices contain the necessary derivations and algorithms used.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all hyperparameters and architecture choices Appendix E Additionally, the code in the shared anonymised repository helps ensure total reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
34

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: provide the links and citations of all the used public datasets and the code to generate our synthetic dataset. Additionally, the code used to run the experiments and our method provided in an anonymised repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:/7
nips.cc/ htabbbic/guids////eee /Code/ubdeiieeeeeeee issi asw.ssolsolooooooooo onnolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental setup section together with Appendix E include all the necessary details to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All results reported the summary statistics across ten independent experiments (seeds). The standard deviations are provided in all tables. In the plots, all the lines correspond medians and confidence bands given by interquartile ranges.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
35

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided information on which type of GPUs used and the time of development of our method in the implementation details in Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The code of ethics strictly followed.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents work whose goal is advance the field of Machine Learning, and we do not believe there are potential societal consequences our work. Therefore, we feel that extra statement needs to be specifically highlighted here.
Guidelines:
36

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: To our best understanding, our proposed method does not have generative capabilities nor any risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external datasets used are publicly available and are accordingly cited and acknowledged, with their corresponding licences properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
37

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thooe cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide an in-depth explanation of the introduced methods as well as the algorithms used and procedures to generate data.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not include experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not include experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
38

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "0ATP5w9JU3", "reviewer": "Reviewer_W2n5", "summary": "The paper proposes a method to make any pre-trained black box model intervenable, given a small validation dataset with concept.\n\nThis is done by the following procedure:\n- Extract the output of a layer of a black-box $g_{\t\\psi} (z)$ and train a probing network $q_{\\xi}(z)$ to extract the concepts from that layer using the validation dataset.\n- For intervening i.e changing concepts $c$ to $c'$: Given $z$ find $z'$ that minimizes $L^c(q_{\\xi}(z'),c')+d(z,z')$ where $d$ is a distance measure. $z'$ is optimized using gradient descent.\n- Train the black-box network to be intervenable by finetuning, minimize $L^y(g_{\\psi}(z'),y)$ \n- To edit the representations,  replace $z'$ with $z$.\n\nThe paper demonstrated the effectiveness of their approach by comparing it to the following baselines:\n- A black-box with representation editing and no finetuning.\n- CBM\n- A finetune black-box that predicts the concepts and the output (Multitask-learning)\n- A finetune black-box by appending the black box activations with the concepts.\n- Post-hoc CBMs\n\nThe approach was evaluated on the following datasets:\n- A synthetic tabular dataset with a complete and incomplete concept dataset.\n- Vision datasets with labels: AWA2 and CUB\n- Vision datasets with no labels: Cifar10 and imageNet (Here labels are obtained from an LLM in a label-free style).\n\nResults that the proposed method outperforms others in cases where data is limited i.e the % of validation dataset is low or when the number of interventions is generally low.", "questions": "- Which layer of is used for interventions, does it matter? \n- For the multi-task baseline, Fine-Tune MT how do you intervene on test time since changing the multitask label does not affect the output of the classifier?\n- For the appending baseline Fine-Tune A do you throw away the last layer of the original network (i.e the layer after the one used for interventions) and create a new one to handle the new size?", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Originality -- Good\n\n- The method itself is original, i.e., how the representations are found and training is done is novel. While there are post-hoc CBMs that can allow intervention on the black box as well. The proposed method seems to outperform them, especially in the incomplete data regime.\n\n### Quality-- Excellent\n\n- The proposed method is simple but seems very effective and the fine-tuning procedure is well motivated by an empirical comparison to black-box model interventions without the fine-tuning.\n\n- Very good empirical evaluation in general, comparisons were done on synthetic data, and 4 different image data sets on different domains; with reasonable baselines.\n\n- Code was provided to reproduce the experiments.\n\n### Clarity -- Excellent\n- The paper is well-written, clear and easy to follow.", "weaknesses": "### Significance -- Fair\n\n- There are two main advantages of using CBMs, interpretability and intervenability. While the proposed method allows for intervenability, this method does not add interpretability to black box model.\n\n- One could argue that since you need a validation set with concept anyhow, one should just put a CBM in the original architecture giving both interpretability and intervenability. Empirical validation in the paper did show that the proposed method did better than CBM however baseline CBM used in the evaluation was not trained for intervenability as done by Mateo, et al. [1] which could dramatically improve the performance of regular CBMs. So its difficult to conclude that the proposed method is in fact superior to CBMs in any way.\n\n\n\n[1] Espinosa Zarlenga, Mateo, et al. \"Learning to receive help: Intervention-aware concept embedding models.\" Advances in Neural Information Processing Systems 36 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and questions! Below is our point-by-point response.\n\n> There are two main advantages of using CBMs, interpretability and intervenability. While the proposed method allows for intervenability, this method does not add interpretability to black box model.\n\nAs stated by the reviewer, this work focuses on intervenability and its formalisation. However, the proposed method also incorporates interpretability with respect to a regular black box model. By mapping intermediate layers to human-understandable concepts via the probe, the user gains a better interpretation of the information contained in the representations. This connection between probing classifiers and interpretability is supported and further detailed in prior work referenced in the manuscript [1,2,3].\n\n> One could argue that since you need a validation set with concept anyhow, one should just put a CBM in the original architecture giving both interpretability and intervenability. Empirical validation in the paper did show that the proposed method did better than CBM however baseline CBM used in the evaluation was not trained for intervenability as done by Mateo, et al. [1] which could dramatically improve the performance of regular CBMs. So its difficult to conclude that the proposed method is in fact superior to CBMs in any way.\n\nThe original CBM architecture requires training the concept and target predictors from scratch. This is a limitation in the settings where there is not enough data available to train a large backbone, e.g. using Stable Diffusion like in the CIFAR-10 and ImageNet experiments. This problem would be aggravated further if only a validation set with concept annotations was used. We explore this in Figure 3 and show that CBMs are not sample efficient when trained only on the validation set. \n\nWe did a preliminary exploration of the introduction of interventions at training time in the CBM, similar to the work done on the mentioned CEMs by Mateo et al., and we did not find a significant improvement with respect to the regular CBM in the presented datasets. \n\nEffectively, intervening on a vanilla CBM at training time can be thought of as a combination of independent and joint training, which, as shown in Appendix F.10, have similar results.\n\nIn the cases where the concept set is incomplete, the bottleneck layer in the CBM will hinder the overall model performance, even under interventions. However, finetuning for intervenability does not limit the expressivity of the intervened-on layer and, therefore, allows for better target prediction. \n\n>Which layer of is used for interventions, does it matter?\n\nThe reported experiments show the results of intervening two non-linear layers after the backbone. The layer you intervene on will play a role in the effectiveness of interventions. Generally, high-level information in neural networks emerges in deeper layers, which is why we recommend conducting the interventions there. Additionally, the required complexity of the probe is lower as the dimension of the layers is reduced deeper within the network. Prior works show probing at deeper layers is benefitial [2]. However, the proposed method can be applied at any step, provided the layer has extracted enough information, and we expect interventions to have a positive effect on the target prediction. \n\n>For the multi-task baseline, Fine-Tune MT how do you intervene on test time since changing the multitask label does not affect the output of the classifier?\n\nThe intervention on Fine-Tune MT is done in the same fashion as in the proposed Fine-Tune I, using the three-step solution introduced in Section 3.1. in the manuscript. This will be made clarer in the revised version of the paper. \n\n>For the appending baseline Fine-Tune A do you throw away the last layer of the original network (i.e the layer after the one used for interventions) and create a new one to handle the new size?\n\nThis is correct. We leverage the pretrained backbone until the layer where the concatenation with the concepts occurs, in this case, two non-linear layers after the extracted features. At this state, a new target predictor is created with input layer size correspondig to the representations concatenated with the concepts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qw7igYO6At", "reviewer": "Reviewer_Q4Tj", "summary": "The paper proposes a novel method for test-time concept-based intervention on black-box models. Starting from the interactive intervention procedure of concept-bottleneck models, the authors devise a novel technique for reproducing it on any pre-trained black-box model. It consists of training a concept probe on the latent space of the model and using it to modify the representation of a given concept and checking the response of the model to this intervention. Also, the authors define a measure of intervenability which can be optimized to fine-tune a model architecture, while making it more prone to concept-intervention. The proposed approach is tested over several benchmarks.", "questions": "1. Table 1 report results where the black-box model is normally not that good for the target performance (among the worst results in general). This is something that I would not have expected, since both the bottleneck and the fine-tuning pose further constraints that normally impact the model performance (as also reported by CBM and many other concept-based approaches). How do you explain it?\n2. Why is the CBM more computationally expensive in the experiments with VLM-based Concept? I think you could use a fixed CLIP to predict the concepts and simply train the task predictor on that. If I am right, I think the following sentence is unfair, and I suggest removing it “By contrast, our method allows fine-tuning the pretrained network, thus being helpful where a CBM is impractical”.\n3. In Cher X-ray classification the black-box models are not intervenable. This is a surprising result, which partly contrast with the claim that you propose a method to make “interventions on any pretrained neural network post hoc. Could you elaborate further on that?", "limitations": "The authors report mostly possible expansions of their work, rather than actual limitations and possible societal implications of their approach. Indeed, even though the proposed method improves the interpretability of the model, it may impact its security. An attacker, for instance, could misuse the proposed technique for detecting model biases towards some specific concepts and creating realistic adversarial attacks that could go undetected.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "**Novelty and impact**: although based on the idea of concept probing (step 1, already proposed in literature), the following steps of editing the representations and updating the output (i.e., of performing an intervention) of a black-box model are simple but very interesting and potentially important for a large part of the XAI community. Indeed, it brings for the first time the interactability of explainable-by-design concept-based models to post-hoc concept-based techniques.", "weaknesses": "## Major issues\n- **Presentation**: the paper is not very clear in a few passages. Also, some comments to the results in the experimental sections are excessive or not completely true, and they must be changed/softened. See minor issues for indications of the individual sentences to edit.\n- **Equation 4**: presenting Eq. 4 and then only considering the special case of $\\beta=1$ is not correct, as it completely neglects a term. This is not acceptable even if it implies the training of the probe, a reader expects to see an ablation study on that in the experiment. I solicit the authors to present it otherwise (e.g., presenting first the actual optimized equation) or provide an ablation study on that. \n\n## Minor issues\n- In the abstract, “Recently, interpretable machine learning has re-explored concept bottleneck models (CBM)”. The sentence is not very clear, it seems in this way that concept bottleneck models have already been proposed in the past. You also say it in the introduction. If it is the case, and it has been proposed earlier, you should provide a citation.\n- “While, in principle, a specialist may just override predictions, in many cases, concept and target variables are linked via nonlinear relationships potentially unknown to the user.” This sentence is not super clear and does not seem to support the paper point. I would consider rephrasing or removing it. \n- Figure 1: The concepts reported are not clear. Another example or the same example with textual concepts instead of icons would be probably easier to read.\n- In the Method, “we will assume being given a small validation set”, I would explicitly add “equipped with concept annotations”. \n- “think of a doctor trying to interact with a chest X-ray classifier $(f_\\theta)$ by annotating their findings”, which findings? Not very clear, why don’t you directly provide an example of concept for the X-ray task?\n- “Note that, herein, an essential design choice explored in our experiments is the (non)linearity of the probe”. Not clear from this whether you consider both a linear and a non-linear, or only one of the two without looking at the experiments.\n- You should probably cite also TCAV [1] in 3.1 when talking about probing. The CAV is probably the most renowned concept probe.\n- I think you should better explain and point out the fact that if you directly optimize equation 3 you could decrease the performance of the model, since you would maximize the error over the non-intervened network. Instead, you says “ Equation 3 is differentiable, a neural network can be fine-tuned by explicitly maximising”. And, in equation 4, you change the optimization problem optimizing both losses\n- Fine-tuned, A: it is not very clear the reason of including this model, nor the explanation. Please rewrite and add further motivations.\n- Commenting Figures F.4 and Figure 4 this close is confusing. If possible, position the appendix figures in a different order to avoid confusing the reader. \n- “Finally, post hoc CBMs (even with residual modelling) exhibit a behaviour similar to the synthetic dataset with incomplete concepts: interventions have no or even an adverse effect on performance.” This claim is not supported by the results presented in figure 4(d) bottom: the AUPR is increasing in a non-negligible manner. Please rephrase the sentence. A similar consideration holds also for the Discussion section.\n\n[1] Kim, Been, et al. \"Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).\" International conference on machine learning. PMLR, 2018.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed comments! Below, we respond to your concerns point by point.\n> Equation 4: presenting Eq. 4 and then only considering the special case of β=1 is not correct.\n\nWe believe the formalization of the overall optimization problem is a beneficial contribution to future lines of work. However, we agree our focus is on the simplified case and will adapt the manuscript first introducing the special case, followed by the general formulation. \n>In the abstract, “Recently, interpretable machine learning has re-explored CBMs”. The sentence is not very clear, it seems in this way that CBMs have already been proposed in the past.\n\nWe refer to the works referenced in “Introduction” and “Related Work” [4,5] and will make them more explicit. They propose concept-based approaches to classification. Although not strictly referred to as CBMs, they are very similar and the original work on CBMs [6] acknowledges them as closely related. \n> “While a specialist may just override predictions, concept and target variables are linked via nonlinear relationships potentially unknown to the user.” This sentence is not super clear and does not seem to support the paper point.\n\nWe motivated the need for concept-based interventions, especially in scenarios where a professional has the concept information but inferring the target variable is not an easy task. We understand the potential confusion and will remove it. \n> Figure 1: The concepts reported are not clear.\n\nWe will include their description in the main text and caption. Particularly, in order of appearance from left to right, and top to bottom, the meaning of the icons is: “fierce”, “timid”, “muscle”, “walks”, “otter” and “grizzly bear”. This figure is complemented by a more concrete example of model correction in Figure A.2 with a list of all intervened on concepts.  \n> “think of a doctor trying to interact with a chest X-ray classifier by annotating their findings”, which findings? \n\nThe findings are the concepts of the X-ray datasets. We will provide examples to help understand the X-ray use-case and include a table in the appendix with the full list of concepts: atelectasis, cardiomegaly, consolidation, edema, enlarged cardiomediastinum, fracture, lung lesion, lung opacity, pleural effusion, pleural other, pneumonia, pneumothorax, and support devices.\n> Not clear whether you consider both a linear and a non-linear probe, or only one of the two without looking at the experiments.\n\nWe will clarify that the experiments shown in the main body are performed under a linear probe and an ablation with a nonlinear probe is found in the appendix.\n> You should probably cite also TCAV in 3.1 when talking about probing. The CAV is probably the most renowned concept probe.\n\nWe will include it. \n> I think you should better explain the fact that if you directly optimize equation 3 you could decrease the performance of the model.\n\nWe will make it more explicit that the final loss function is a linear combination of the intervenability measure in Eq. 3 and the target prediction loss, resulting in Eq. 4. We always consider the combination of the two to better control the tradeoff. \n> Fine-tuned, A: explanation to why including this model \n\nThe motivation is: it would be natural to include the concepts in the model, either at the input or at intermediate layers. Since the considered feature spaces are high-dimensional, we append concept variables to the intermediate activations as a direct comparison to Fine-tuned I. \n> “Finally, post hoc CBMs exhibit this behaviour: interventions have no or even an adverse effect on performance.” This claim is not supported by the results presented in figure 4(d) bottom.\n\nWe will rephrase the statement to clarify that in some datasets, a small positive effect is present. \n> Table 1 report results where the black-box model is normally not that good for the target performance. \n\nIn synthetic data, the generating process favors the CBM architecture, as concepts directly relate the inputs to the target and we expect the CBM to outperform. In the remaining datasets with more complex concept-to-target dependencies, the CBM performs worse, as expected. If the concept variables are “useful” for the target prediction, we hypothesise the fine-tuning methods would improve predictive performance. \n\nSpecifically, fine-tuning enforces reliance on the concept variables, which might increase the robustness and prevent overfitting. Moreover, fine-tuned black boxes keep the original representation without introducing a bottleneck layer and can still use non-concept-based information if necessary. To finalize, our focus was on making the models more intervenable, and Table 1 suggests that most methods behave on par, i.e., fine-tuning with intervenability does not harm the performance. \n> Why is the CBM more computationally expensive in the experiments with VLM-based Concept? I think you could use a fixed CLIP to predict the concepts.\n \nThe VLM-based concept annotation does not influence the computational cost of the CBMs. We have used these annotations to explore larger backbone architectures and training these from scratch, as would be needed for the (ante hoc) CBM baseline, is not computationally feasible. It would require a large number of concept annotations and resources. Using a fixed CLIP to predict the concepts is alike the post hoc CBM, which we include as baseline. \n> In Cher X-ray classification the black-box models are not intervenable. This partly contrast with the claim that you propose a method to make “interventions on any pretrained neural network post hoc. \n\nIn some instances, activations are correlated with concepts, but the network does not necessarily rely on them for prediction, hence, only probing is not enough to intervene effectively. However, after fine-tuning, the black box learns to use the information correlated with the concepts at prediction time. Hereafter, fine-tuning may be necessary in some instances.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lv4WIGwXYG", "reviewer": "Reviewer_ajS6", "summary": "This work introduces a method for performing concept-based interventions on pretrained black-box neural networks, requiring only a small validation set with concept labels. The paper formalizes the notion of intervenability as a measure of the effectiveness of these interventions and experimentally assesses the intervenability of black-box classifiers across several benchmarks and architectures.", "questions": "### **Major**\n\n* **\"CBM val\" Baseline:** Could you clarify in detail how the baseline \"CBM val\" is trained and validated?\n\n*   **“Since interventions can be executed online using a GPU (within seconds), our approach is computationally feasible”:** This is not a formal justification for why the proposed approach is computationally tractable. The time complexity can be estimated in a more formal way.\n    \n\n### **Minor**\n\n*   **“This work focuses specifically”**: This paragraph is already quite technical. While clear, it could be moved to a \"preliminary\" section after the introduction and rephrased here to make the introduction easier for non-technical readers. Also, the notation is presented again in the methods section (which is fine as it is a technical section).\n    \n*   **Related Work:** Consider moving the related works section after the experiments. This way, you could use the experimental results to discuss the relations of your contributions with related works.\n    \n*   **Figure 2, Step 3:** The arrow in step 3 might go downwards (as in the figure in the appendix).\n    \n*   **“For a trained CBM fθ (x) = gψ (hφ (x)) = gψ (ˆ c)”**: Consider explaining again here what c^\\\\hat{c}c^ represents as this was mentioned in the introduction, but it was not mentioned in the presentation of the notation.\n    \n*   **“we define the intervenability as follows”**: Consider explaining the idea behind the equations (e.g., \"Here, the effectiveness of interventions is quantified by\" for Eq 2) before showing the equation to ease reading.\n    \n*   **“where D is the joint distribution over the covariates …”**: Consider describing the variables involved in an equation before showing the equation to avoid the reader going back and forth.\n    \n*   **“To alleviate the need for human-annotated concept values”**: This sentence might be confusing as it is not clear whether it is the proposed approach or label-free CBMs that alleviate the need for human-annotated concepts (or both). I'd rephrase it with something like \"To evaluate the effectiveness of the proposed approach in the absence of human-annotated samples ...\"\n    \n*   **Results section:** Organizing results by datasets leads to repetitions of some observations. I'd suggest organizing paragraphs by key findings (which might sometimes relate to a single dataset, but it is ok).\n    \n*   **Table 1:** Consider dropping \"0\" at the beginning of each value (e.g., \".987\") or express numbers in base 100 (e.g., 98.7) to save space. Consider highlighting the best models in bold.\n    \n*   **CIFAR 10 results in Table 1:** Why not use CIFAR-100 (where concept labels are available as a ground truth) instead of CIFAR-10?\n    \n\n* **Discussion & Conclusion:** The largest gap with respect to baselines is on AwA and MIMIC. Do you have an intuition why this is the case? The discussion could benefit from this analysis.", "limitations": "Authors adequately discuss limitations and future works in a dedicated section.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "*   **Originality:** The paper presents a novel approach that enables effective concept-based interventions on black-box models. While the methods used already existed, they were creatively applied for this purpose, leading to both methodological originality and novel experimental findings.\n    \n*   **Quality:** The research methods are appropriate and well-described. The results and conclusions are supported by the data. The proposed approach is simple and sound, making it mathematically elegant and enabling the research community to use and expand it.\n    \n*   **Clarity:** The paper is well-structured with a clear flow. The writing is clear, concise, and free of grammatical errors. Figures and tables are well-designed and effectively support the text. The abstract and title summarize the paper effectively.\n    \n*   **Significance:** In their vanilla formulation, Concept Bottleneck Models were already applied on pre-trained models (e.g., ResNets) by introducing only a few extra layers on top of the backbone to predict concepts and downstream classes. Compared to vanilla CBMs, the main innovation of the proposed approach seems to consist in using concept labels only to fine-tune the model using the validation set. The findings demonstrate the flexibility of concept-based approaches and the possibility of extending concept-based interventions to black-box architectures more effectively wrt an equivalent CBM similarly fine-tuned. In view of these results, this paper could have a significant impact on real-world applications where existing pre-trained black boxes are currently used and a small concept-annotated validation set is available.\n    \n*   **Literature:** The sources are properly cited, relevant, and up-to-date.", "weaknesses": "*   **Clarity / Quality:** I could not find the precise training/validation procedure used for the baseline \"CBM val\", which represents one of the most critical baselines to compare with. I double checked both the paper but I could not find a detailed description. I also checked the code, but I could not find the configuration file (and the rest of the training/validation flow) for this baseline. The reason why I'm curious is that CBMs can also be viewed as \"classification heads\" fine-tuned on top of pre-trained models using a small validation set (e.g., a ResNet pre-trained on ImageNet used as a backbone of a CBM trained on CUB). The main result of this paper seems to consist in making interventions in CBMs effective with a smaller validation set. That is why the way this baseline is constructed is relevant to assess the quality and impact of this work. I could imagine at least two ways to construct such a baseline using a pre-trained backbone $g: X \\to H$ (e.g., a ResNet trained on ImageNet) and a classification head $f: H \\to Y$ (e.g., trained on CUB using downstream task labels but not concept labels): \n 1. Fine-tune a single layer $f^{(c)}$ of $f$ to align the activations of this layer on ground-truth concept annotations of the validation set. Freeze all the other layers of the model $f$.\n 2. Fine-tune all parameters before the layer $f^{(c)}$, while freezing all the layers following $f^{(c)}$.\n 3. Opposite of option #2.\n 4. Train a new model $f'$ using only the ground-truth concept annotations of the validation set.\n\n I could not understand from the paper/code to which option the baseline \"CBM val\" corresponded to and whether the fine-tuning approach was used to train this baseline as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the feedback! You will find our point-by-point responses below.\n> I could not find the precise training/validation procedure used for the baseline \"CBM val\".\n\nWe have updated the public repository to include the configuration file `config_synthetic_cbm_val.yaml` and code to run the “CBM val” baseline. As mentioned, CBMs can be viewed as classification heads on top of pre-trained models. However, this would correspond to the “post hoc CBM”. We frame the presented CBM in its original “ante hoc” implementation [6], where the backbone is trained from scratch. Our work focuses on making interventions on *any pre-trained black box* more effective, not the CBMs. From the options provided by the reviewer, our “CBM val” is number *4*. This way, we show the sample efficiency of our method over the (ante hoc) CBMs when only a small portion of annotated data is available. \n>“Since interventions can be executed online using a GPU, our approach is computationally feasible”: This is not a formal justification for why the proposed approach is computationally tractable. \n\nThis statement is not meant as a formal complexity metric, but as a remark for the interested reader that the interventions are less costly than training or fine-tuning and can be run “online” with a regular GPU. As the intervention procedure is gradient-based, the computational cost varies across datasets and depends on the optimization iterations. The number required depends on several factors, e.g., the value of $\\lambda$, the number of concepts, the dimensionality of $z$, and the number of data points per batch. Given these, providing a formal description of the running-time complexity is challenging.\nFor example, on the synthetic dataset, with the convergence parameter set to $10^{-6}$ (very conservative) and for a batch of 512 samples, the intervention procedure requires approx. 500 to 2500 steps (the number varies depending on the number of given concepts and the value of $\\lambda$), which amounts to 0.5 to 2 s (for the whole batch) for an untuned black-box model. We use smaller batches and more permissive convergence criteria when fine-tuning, allowing a considerable speedup. In addition, note that the run-time of the intervention procedure is not strictly dependent on the black-box model’s architecture (except for the dimensionality of the layer intervened on). \n> “This work focuses specifically”: This paragraph is already quite technical. While clear, it could be moved to a \"preliminary\" section after the introduction.\n\nThank you for the suggestion! In order to address a broader audience, we will include a preliminary section introducing the method with the technical contributions, and reframe the introduction to be less technically oriented, removing the formal notation. \n> Figure 2, The arrow in step 3 might go downwards.\n\nWe will update the figure. \n> “For a trained CBM fθ (x) = gψ (hφ (x)) = gψ (ˆ c)”: Consider explaining again here what c^\\hat{c}c^ represents \n\nWith the new organization of the introduction and preliminary section of the methods, we will introduce the notation accordingly prior to the description of the method.\n> “we define the intervenability as follows”: Consider explaining the idea behind the equations before showing the equation to ease reading.\n\nIntuitively, the more effective an intervention is, ideally, the more it should reduce the target prediction error. Leveraging this, our equation measures the gap between the prediction error prior and post intervention, where larger gaps correspond to more effective interventions. \n> **I**. “where D is the joint distribution over the covariates …”: Consider describing the variables involved in an equation before showing the equation to avoid the reader going back and forth. **II**. “To alleviate the need for human-annotated concept values”: This sentence might be confusing as it is not clear, I'd rephrase it. **III**. Table 1: Consider dropping \"0\" at the beginning of each value or express numbers in base 100 to save space. \n\nWe will clarify and introduce the notation in **I**, rephrase **II**, and change the metrics to percentages and bold the best results in **III**. \n\n> CIFAR 10 results in Table 1: Why not use CIFAR-100 (where concept labels are available as a ground truth) instead of CIFAR-10?\n\nWe include experiments with both CIFAR-10 and ImageNet that do not originally have available ground-truth concepts to show that our method can be combined with “concept discovery” [7], where annotations are acquired through VLMs. We did not include CIFAR-100 as we have a synthetic tabular, two natural image datasets (CUB and AwA2), and two medical image datasets (MIMIC-CXR and CheXpert) that, we believe, are sufficiently representative to cover the settings with “annotated” concepts. \n> The largest gap with respect to baselines is on AwA and MIMIC. Do you have an intuition why this is the case?\n\nGenerally, our procedure has a stronger effect because we target the improvement with interventions with a stronger inductive bias, while other methods do not explicitly encourage intervention effectiveness. In some instances, activations are correlated with concepts, but the network does not rely on them for prediction, e.g., when there is a complex relation between the concepts and target, as in the case of X-rays (MIMIC), and only probing is not enough to intervene effectively. However, after fine-tuning, the black box learns to use the information correlated with the concepts at prediction time.  At the same time, our approach does not require introducing a concept bottleneck layer, and, therefore, the model may still use additional non-concept-based information from the representations. With respect to the AwA (and CIFAR-10), the experimental findings fall within our expectations: in both, concepts are useful for predicting the target variable, and our fine-tuning explicitly encourages intervention effectiveness, providing a stronger supervisory signal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We thank you for the detailed feedback; we will make sure to address the concerns and incorporate the corresponding changes in the revised manuscript upon acceptance. Below is a summary of the main responses and clarifications addressed in this rebuttal: 
- We have provided a more in-depth explanation of the baseline “CBM val”. Effectively, it consists of training an ante hoc CBM solely on the validation set to compare CBM’s sample efficiency against the proposed method. 
- In our experiments, we have compared with two baselines related to CBMs, namely, (ante hoc) CBM and post hoc CBM. We would like to emphasise that the (ante hoc) CBMs represent the original formulation of CBMs, i.e. *without* pretrained backbones. We will clarify the distinction between the (ante hoc) CBM and post hoc CBM used as baselines in the revised manuscript. 
- Our method focuses on the intervenability of the models. However, it also facilitates interpretability via representation probing. 

*References*:

[1] Belinkov, Y. (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, 48(1), 207-219. 

[2] Alain, G., & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644.

[3] Kim, Been, et al. (2018) Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). International conference on machine learning. PMLR.

[4] Lampert, C. H., Nickisch, H., & Harmeling, S. (2009). Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition. Miami, FL, USA: IEEE. 

[5] Kumar, N., Berg, A. C., Belhumeur, P. N., & Nayar, S. K. (2009). Attribute and simile classifiers for face verification. In 2009 ieee 12th international conference on computer vision (pp. 365–372). Kyoto, Japan: IEEE. 

[6] Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., & Liang, P. (2020). Concept bottleneck models. In H. D. III & A. Singh (Eds.), Proceedings of the 37th international conference on machine learning (Vol. 119, pp. 5338–5348). Virtual: PMLR. 

[7] Oikarinen, T., Das, S., Nguyen, L. M., & Weng, T.-W. (2023). Label-free concept bottleneck models. In The 11th international conference on learning representations.

Meta-review: All reviewers find the proposed approach to make a pre-trained black box model interpretable very interesting.
All reviewers clearly vote for accepting this paper.
Decision: Accept (poster)